texts: Mean across every several rows in pandas

texts:
I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a specific order. The number of repetitions of the same label in one batch is always the same. In the example below, every three rows has the same label.

I would like to get a new table where Var 1 and Var 2 are averaged across repetitions of labels. In this example, I would like to average every three rows as follows:

Is this possible to do with some functions in pandas or other Pyhton libraries?


texts:
I have data with the following structure:
created_at | customer_id | features | target
2019-01-01             2   xxxxxxxx       y
2019-01-02             3   xxxxxxxx       y
2019-01-03             3   xxxxxxxx       y
...

That is, a session timestamp, a customer id, some features, and a target. I want to build an ML model to predict this target, and I'm having issues to do cross-validation properly.
The idea is that this model is deployed and used to model new customers. For this reason, I need the cross-validation setting to satisfy the following properties:

It has to be done in a time-series way: that is, for every train-validation split in cross-validation, we need all created_at of the validation set to be higher than all created_at of the training set.
It has to split customers: that is, for every train-validation split in cross-validation, we cannot have any customer both in train and validation.

Can you think of a way of doing this? Is there an implementation in python or in the scikit-learn ecosystem?


texts: Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

texts:
To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very long transactions (the longest one is 415). At first, I kept the items just as the error messages themselves (so each transaction would be a list of strings) but since there is a certain amount of possible error messages, I created a dictionary where each possible message is encoded as an int. Now, my transactions are in the form of a list of ints (so all the transactions are represented as a list of lists of ints).

I have tried the apriori algorithm and the fp-growth algorithm. Both take a very long time to run (over a day for apriori, and less for fp-growth) and then terminate with "Killed: 9" being the only error message. There is nothing else printed in the console, no other information. From investigating online,my best guess as to why this happened is that the process was using too much memory, so it received the SIGKILL signal. I've tried the apyori package, pyfpgrowth, and this one I found on github. I am guessing that the reason why this happens is because my transactions have a lot of items, as that's the only difference from the example data the documentation provides. I have tried running it on a small amount of my data, only 6 transactions, and the same thing happens. From my understanding of how the algorithms work, it goes into a recursive loop.

I have discussed pruning some of the error lines from my dataset with the engineers, however the purpose of this project is to find patterns within these error logs that we haven't considered, and it's difficult to discern if something is actually insignificant enough to leave out. Is there a more efficient algorithm for long transaction lists (and if so please indicate the implementation) or anything I can do to find frequent patterns successfully?

Additionally, I'm worried that these algorithms may not be taking into account the fact that the order of items within the transaction matters. Each line has a timestamp, so they are organized in a sequence, which is pretty significant. Finding errors that tend to occur together in a thread ID is helpful, but not as much as ones that will appear in sequence. In my research, frequent pattern mining and sequential pattern mining appeared to overlap and at times be used interchangeably, so if I am going about this wrong and there is a better approach, I would appreciate it.


texts:
Given a time series with job-submission counts, how can I predict which certain features about the jobs?

I need to predict how many jobs and which jobs arrived in some system. Using pandas.groupby, I've sliced the data into 15 minutes intervals.

I can predict how many jobs arrived but also need to predict which type of job will arrive and some other features.


texts: Artificial neural network on ternary content addressable memory

texts: One of the applications of Ternary content addressable memory(TCAM) is artificial neural network. How does TCAM do prediction or recognition? How TCAM is related to artificial neural network?


texts: GradientChecking, can I blame float precision?

texts:
I am trying to GradientCheck my c++ LSTM. The structure is as follows:

  output vector (5D)

  Dense Layer with softmax (5 neurons)

  LSTM layer (5 neurons)

  input vector (5D)

My gradient check uses this formula

$$d = \frac{\vert \vert (g-n) \vert \vert _2 }{
\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$

The Dense Layer returns descrepancy = 2.843e-05  with an epsilon of 5e-3

With 5e-5 or lower the network doesn't see any changes in Cost at all, anything greater than 5e-2 results in low precision.

This is only for top layer, and I am using 'float' 32 bit values for everything.

For the LSTM layer, I am using 5e-3 as well, but descrepancy is 0.95

The LSTM seems to converge fairly quickly on any task

0.95 bothered me, so I manually compared NumericalGradient array against the BackpropGradient array, side by side. All of the gradients match sign, and only differ in size of each entry.

For example:

numerical:    backpropped:

-0.015223     -0.000385
 0.000000      0.000000
-0.058794     -0.001509
-0.000381     -9.238e-06
 9.537e-05     2.473e-06
 0.000215      6.266e-0.6
-0.015223     -0.000385
 ...

As you can see, the signs do indeed match, and the numerical gradient is always larger than the back propped gradient

Would you say it is acceptable and I can blame float precision?

Edit:

Somewhat solved, - I simply forgot to turn-off "average the initial gradients by the number of timesteps" during backprop of my LSTM.

That's why my gradients were always smaller in the "backpropped" column.

I am now getting descrepancy of 0.00025 for LSTM

Edit: setting epsilon to 0.02 (lol) seems like a sweet-spot, as it results in descrepancy of 6.5e-05. Anything larger or smaller makes it deviate from 6.5e-05, so it seems like a numerical issue ...Only 2 layers deep though, weird af

Someone had this precision before?


texts:
I have a dataset of 4 classes with the following number of instances:

Class 0:  13175
Class 1:  82
Class 2:  75
Class 3:  121

Have have applied several subsampling and oversampling methods from the Python imbalance-learn API but none of them had a good performance for all classes. I have applied:

Undersampling: CondensedNearestNeighbour, EditedNearestNeighbours, NeighbourhoodCleaningRule, RandomUnderSampler.
Oversampling: SMOTE, ADASYN
class_weight:['balanced'] parameter option in my grid search
costcla library, but it does not work with more than two classes.

And I was not successful. Could you suggest a solution for this problem?


texts: Best object detection model where inference speed is not a factor?

texts:
As in the question I’m wondering what is the current best model for object detection in images where inference time doesn’t matter?
Thank you in advance.


texts:
I want to compare two corpora (two different collections of texts) using Topic Modeling. I trained the model separately on the two collections and manually matched similar topics based on their frequent words.

I was wondering if there is a systematic way of comparing the topics across two corpora and measuring their similarity.


texts:
I am working on a time series prediction problem. I am using keras models for machine learning.

For this prediction, weather variables are used as input. They can be of two types: forecasted and actual. I have acquired both these types of data of a considerable amount of time and I want to train and test my model on the data. My question is:

1) Should I use forecasted weather variables OR actual weather variables in input while training? (consider that only forecasted weather variables will be available at model inferencing time).

2) Same question 1 for testing.

Is there a rule or general practice regarding the above questions? If yes, I would like to know that.


texts: How do i get to make colab read my images from my google drive?

texts: I havee about 2000 images, I have uploaded them to my Google drive. How do I make colab see and read these images. All the images are in one folder.


texts: ML algorithm for determining CSV file header names based on content

texts:
I have a large amount of CSV files, an example of which (for job titles) is listed below. The data is noisy (there are misspellings, difference in capitalisation, missing values, and they are not well-formed (some files have headers, some have not, and if headers are present, they don't always agree on name). I have gazetteers available.

...,IT Manager,...
...,Senior IT Manager,...
...,it manager,...
...,IT managre,...
...,junior IT managre,...
...,NULL,...
...,Business Consultant,...
...,Business consultent,...

I have a finite set of entity types (First Name, Last Name, Location etc.), and the task is: Given the content of this column, which entity type, if any, does it represent? (It is the first step in a processing pipeline.)

It is a form of named entity recognition [supervised learning classification] task, but all the papers I have read about named entity recognition uses conditional random fields or maximum entropy for natural language tasks, and I don't think this count as natural language.

What would be the appropriate approach/ML algorithm for such a task? How should my training data be formatted?


texts:
It is my impression that a single layer LSTM architecture consists of $t$ LSTM cells that are identical duplicates, where $t$ is the number of time steps. Then there are gates within the LSTM cell. I have struggled to find a rigorous explanation of what each “gate” actually consists of. Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?

Thanks


texts: Neural Network: how to utilize weakly-/unsupervised data to improve supervised network?

texts:
Let's consider one has built a fully-supervised neural network for some task, e.g. localizing an object in various scenes. As you can imagine, it is quite time-consuming to label data: one has to manually localize the object in an image and draw a bounding-box around it - one at a time.

Say we have a normal convolutional neural network (CNN) for the fully-supervised localization, thus something like this:

          2D input image
                |
       convolutional-layer 1
                |
               ...
                |
       convolutional-layer N
                |
      ---flattened output---
                |
    fully-connected classifier
                |
           output layer

How can we adapt this architecture to utilize weakly-labelled data, too?

No matter if we have a sliding-window approach, something like OverFeat or probability distributions for actually localizing objects: we always need fully-labelled training data. This is a problem, because labelling data fully-supervised is really time consuming. Hence, fully-labelled data sets are pretty rare.

In contrast to this, typically large amounts of weakly-labelled or unlabeled data exists. In my opinion there is a large potential in weakly-labelled / unlabelled data, due to its vast availability. The problem is using this potential without having to manually label every sample by hand.

That said, my question is: how can weakly-labelled data sets (i.e. "object" vs. "no object") or completely unlabelled data sets be utilized to improve the robustness of fully-supervised architectures, like the mentioned CNN?

Is it generally possible to mix a supervised approach with some unsupervised approach? Like enabling a fully-supervised architecture to somehow utilize weakly-labelled data for training?


texts: I Can't find my RMSProp implementation bug?

texts:
I'm trying to implement RMSProp in my own Neural Network library so I can undertand the 'under-the-hood' operations, but this specific implementation is not working / converging, and I can't figure out why. I'm pretty sure I followed the formula (RMSProp + Momentum), and here is my code :

//Step 1 - Compute a hidden neuron (this == ClNeuron) error gradient (TanH)
double tmpBuffer = 0.00;
for (std::size_t i = 0; i<this->m_output_connections.size(); i++)
{
    ClNeuron* target_neuron = (ClNeuron*)m_output_connections[i]->m_target_neuron;
    tmpBuffer += (target_neuron->m_error_gradient * this->m_output_connections[i]->m_weight);
}

//Get the TanH derivative of this neuron's last output
this->m_error_gradient = tmpBuffer * this->TanHDerivative(this->m_result_buffer);

//Step 2 - For each of this neuron's input weights, compute the new weight's change
for(std::size_t i=0;i<this->m_input_connections.size();i++)
{
    double new_weight_delta = this->m_input_connections[i]->m_learning_rate * this->m_error_gradient * this->m_input_connections[i]->m_data;
    this->m_input_connections[i]->m_last_error_gradient_mean_square = 0.9 * this->m_input_connections[i]->m_last_error_gradient_mean_square + (1 - 0.9) * (new_weight_delta * new_weight_delta);
    this->m_input_connections[i]->m_momentum = this->m_input_connections[i]->m_momentum_constant * this->m_input_connections[i]->m_momentum + this->m_input_connections[i]->m_learning_rate * new_weight_delta / std::sqrt(this->m_input_connections[i]->m_last_error_gradient_mean_square)+0.000000001;

    //Make the actual weight update
    this->m_input_connections[i]->m_weight -= this->m_input_connections[i]->m_momentum;
}

Would anyone be kind enough to point me in the right direction ?
Many thanks !!


texts: How do I send the results of a convolutional layer and non-deep-learning features into a dense layer in Keras?

texts:
I understand that I can set up a convolutional network for 1-dimensional sequence/time series.

model = Sequential()
model.add(Conv1D())
model.add(GlobalMaxPooling1D())
model.add(Dense())

Let's say I'd like to use "regular" (non-deep-learning) features too in my model, how should I best combine the two at a dense layer?

Concretely, let's assume that, for each row of my dataset, there are 1k points in the time series, along with 100 "regular" features.

To generalize my question, let's say there are now two kinds of time series plus regular features for each row in my dataset. If I would like to have a separate convolutional block for each time series, how do I combine all three?


texts: Am I justified in dropping this independent variable?

texts:
I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and 136 females who do. So roughly 14% of both genders churn and 86% don't. From what I can see, this variable does nothing to explain churn. Am I justified in dropping this variable before I put it into a predictive model? (I have conducted stepwise logistic regression and this variable isn't included in the final call)

I'm sorry if this seems like a very basic or obvious question. I've just started with ML and didn't find a question like this elsewhere.

EDIT: Gender is highly uncorrelated with other independent variables


texts:
I have to build a neural network for text classification that should tell the category and subcategory it belongs to. For example, I have to classify an input(text) to {'Poultry','Wild'} and if it is poultry(First Stage Classification), then we have to classify whether it is a cow or duck or any other element in the subset mentioned here (subset = {‘Cows’,‘Ducks’, ‘Geese’, ‘Goats’, ‘Hens’}).
So, how to implement this in a Neural Network ? How to map the output of the first level of classification to the input of second neural network ?


texts: How to prove Separable Convolution layer is theoretically identical to traditional Convolution?

texts:
I have seen the saying that Separable Convolution layer is theoretically identical to traditional Convolution for so many times,
but yet no one has pointed out where the proof is.
God, I have google this for several hours,
but all those proofs I found are only in the case of 2D sobel filter which is spatial separable instead of depwise separable, and the situations would are totally different with 3D input in convolution layers (and actually the "convolutioin" here is actuall "cross-correlation......").
Is Separable Convolution layer is really identical to traditional Convolution mathmatically?
Where I can find the proof?
I would be very apperiated if you can point it out.


texts: question about splitting data set by attributes and labels

texts:
Why the dataset need to use X as attributes and y as label?
WHy not use a X to have it all?


texts: ExtraTreeClassifier does not handle missing values

texts:
I am using sklearn.tree.ExtraTreeClassifier. It does not handle missing value in training data. All tree-based algorithms handle missing value internally. So, is there anything exceptional ExtraTreeClassifier algorithm or it is just that sklearn has not implemented handling missing values in  ExtraTreeClassifier ?

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')


texts:
I wrote a Neural Networks prediction model in Python.

My data has a few inputs and two outputs. In order to make it work, I have to normalise every column on data for good prediction results.

However, I had an issue. I run several times the prediction model with the same inputs to get the average and standard deviation of it. But obviously, those are also normalised to 0-1. On the test data I knew the min and max, so I could denormalised them. On the prediction values, I cannot the min and max real value.

How do you solve this kind of problem and if you cannot, are there any other decent prediction techniques without the need of normalisation?


texts: I have a dataset of amazon Alexa reviews and want to group negative and positive reviews in separate groups. Is k-means a good approach to it? The dataset is unlabeled so how will my model know which review is for the negative and which is for the positive cluster? Is there any other way you would like to suggest in which I can do the above task?


texts: Assigning a new document to a cluster based on keywords extracted and tf-idf

texts:
I have about 40 clusters of documents defined by a combination of k-means clustering algorithm and hand curation. For example, some of the clusters given by k-means are too noisy so they have been further subdivided.

Now I want to assign new documents to these clusters.

I found that it is possible to extract keywords using tf-idf based methods as mentioned here.

My approach is to extract key terms from each of these clusters using tf-idf based method and I can extract the keywords from the new document using the same method.

My question is, how do I assign the new document to the cluster that has the most similarity?

Edit: I do not have enough reputation to comment on Marks answer: the input to kmeans are document vectors (from doc2vec) of all documents -- and I get the centroids of the initial clusters i.e. centroids = kmeans_model.cluster_centers_. But I have split many of these clusters manually into sub clusters. For example, original cluster 3 is now two clusters -- 3_1 and 3_2. How do I generate a representative vectors (like a centroid) for the documents in these sub clusters?


texts:
I have not GPU support so it often happens that my model takes hours to train. Can I train my model in batches , for example if I want to have 100 epochs for my model,but due to power cut my training stops(at 50th epoch) but when I retrain my model I want to train it from where it was left (from 50th epoch).

It would be much appreciated if anyone can explain it by some example.


texts: Best practical algorithm for sentence similarity

texts:
I have two sentences, S1 and S2, both which have a word count (usually) below 15.

What are the most practically useful and successful (machine learning) algorithms, which are possibly easy to implement (neural network is ok, unless the architecture is as complicated as Google Inception etc.).

I am looking for an algorithm that will work fine without putting too much time into it. Are there any algorithms you've found successful and easy to use?

This can, but does not have to fall into the category of clustering. My background is from machine learning, so any suggestions are welcome :)


texts: Percentage of smaller dataset with respect to bigger dataset

texts:
I have two datasets, which are lists of multidimensional real-valued vectors. One dataset (call it $A=\{x_1, x_2, x_3, ..., x_n\}$ is of a big size, the other (call it $B=\{x_1, x_2, x_3, ..., x_m\}$). Furthermore, the other is far smaller and is a subset of the bigger one ($B \subset A$). The smaller one $B$, comes from some sampling process and what I want to do is to calculate, what fraction of the smaller (obtained from sampling) is in the bigger.
Additionally, since those are real-valued vectors, I can't compare them directly one by one, so a clustering algorithm may be employed. Also the size of one dataset is bigger than the other $|A| >> |B|$.


texts:
AttributeError: module 'torch.distributed' has no attribute 'init_process_group'

Still getting this error after installing latest pytorch-nightly and other stuff, when I tried to run the imagenet example script from pytorch/examples.

Steps to reproduce:

Go to https://github.com/pytorch/examples/tree/master/imagenet
Prepare ImageNet dataset according to the instructions
Run main.py with your suitable arguments along with multiple gpus distributed training


texts: How to do spelling correction for a language but also correct some words in other language

texts:
I want to do spell correction for the portuguese language, specifically for restaurant bots. The problem is that food names aren't always in portuguese as well, and for that reason are the most likely to need correction as the user would not always know how to spell it correctly.

I though of a few things. For example:

Training the model with portugese words + food words spelled wrong,
but it'd be very difficult to find these food words
Training the model for several different languages (but I guess would
make it confused and actually correct a lot of things wrong).
Training the model with portuguese words and for the food words, use
something that picks words that the user wrote and try to approximate
them to the list of food words. (woudln't it be slow?)

What would be a good solution to this problem that can be fast to be used with lots of requests?


texts: Convolutional AE always overfitting time series - what’s wrong?

texts:
I've build a CAE for anomaly detection in time series, but it is always overfitting. I've tried data augmentation, short/long inputvector, dropout rates... I don't know what I'm doing wrong, may be you can give me a hint?
This is my architecture
class CAE(nn.Module):
    def __init__(self):
        super(CAE, self).__init__()

        self.encoder = Encoder()
        self.decoder = Decoder()

        self.encoder.to(get_device())
        self.decoder.to(get_device())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()

        self.conv_enc = nn.Sequential(
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=16,
                out_channels=8,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU()
        )

        self.conv_enc.apply(self.init_weights)

    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_enc(x)
        return x

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.conv_dec = nn.Sequential(
             nn.ConvTranspose1d(
                in_channels=8,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            #nn.AlphaDropout(p=0.5),
            nn.ConvTranspose1d(
                in_channels=16,
                out_channels=1,
                kernel_size=6,
                stride=2,
                padding=0
            )
        )

        self.conv_dec.apply(self.init_weights)

    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_dec(x)
        return x

I'm using following setup

optimizer - AdamW with lr=0.0001
scheduler - OneCyleLR
Normalized Data with MinMaxScaler()
loss - L1Loss
batchsize - 250
trainingsize - 12000 sample
inputsize - 800


texts: Constant Learning Rate for Gradient Decent

texts:
Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step
of the gradient descent process.  What would be the impact of using a constant value for $\alpha_n$ in gradient descent?


texts: Magnifying or reducing the size of input groups into a neural network

texts:
Say you've got two inputs (X1 and X2) that you want to use to predict Y.

You're not sure how important X1 and X2 are for predicting Y, but you assume about even.

One-hot encoding is a good strategy for X1 and it yields a vector of size 10,000.

X2 is an unsigned int between 0 and 1, so you can just pass it as-is.

So your network will look something like

10,001 -> {some hls} -> Out-layer

In theory it could "learn" to assign a lot of importance to X2 compared to X1, but in practice I assume this is hard when the difference in dimension varies so much. At least based on some datasets I tested this was certainly true.

The one "simple" solution I can think of to this issue is instead having a network shapes more like:

10,000 -> {some hls} -> 1,000 --
                               |
                               |-> {some hls} -> Out-layer
                               |
1 -> {some hls} ------> 1,000 --

So basically have some encoders/backbones/whatever-you-want-to-call-them, that increase/reduce the size of certain groups of inputs and train them at the same time as the normal network.

My question here is:

a) Is the problem I identified here a "real" one or would it not come up in practice ?
b) Does it have a name and are there already established solutions to it ?
c) Is the solution I proposed here a "good one" ?
d) Do you have any examples of networks where this method is actually being used ? Preferably implemented in pytroch but it doesn't matter really, the implementation per-say seems easy enough.
f) Are there potential pitfalls in terms of performance and or implementation to this solution that might not be obvious ?


texts: DataFrame.dtypes error when training model using xgboost

texts:
I am getting a DataFrame.dtypes error while following the last steps of this tutorial.

Here is my code:

import xgboost as xgb
regr = xgb.XGBRegressor(colsample_bytree=0.2,
                       gamma=0.0,
                       learning_rate=0.05,
                       max_depth=6,
                       min_child_weight=1.5,
                       n_estimators=7200,
                       reg_alpha=0.9,
                       reg_lambda=0.6,
                       subsample=0.2,
                       seed=42,
                       silent=1)

regr.fit(train_new, label_df)

And this is the error I am receiving:

ValueErrorTraceback (most recent call last)
<ipython-input-64-010296d611e9> in <module>()
     12                        silent=1)
     13
---> 14 regr.fit(train_new, label_df)

D:\anacodna\lib\site-packages\xgboost\sklearn.pyc in fit(self, X, y, sample_weight, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, callbacks)
    358                                    missing=self.missing, nthread=self.n_jobs)
    359         else:
--> 360             trainDmatrix = DMatrix(X, label=y, missing=self.missing, nthread=self.n_jobs)
    361
    362         evals_result = {}

D:\anacodna\lib\site-packages\xgboost\core.pyc in __init__(self, data, label, missing, weight, silent, feature_names, feature_types, nthread)
    378         data, feature_names, feature_types = _maybe_pandas_data(data,
    379                                                                 feature_names,
--> 380                                                                 feature_types)
    381
    382         data, feature_names, feature_types = _maybe_dt_data(data,

D:\anacodna\lib\site-packages\xgboost\core.pyc in _maybe_pandas_data(data, feature_names, feature_types)
    237         msg = """DataFrame.dtypes for data must be int, float or bool.
    238                 Did not expect the data types in fields """
--> 239         raise ValueError(msg + ', '.join(bad_fields))
    240
    241     if feature_names is None:

ValueError: DataFrame.dtypes for data must be int, float or bool.
                Did not expect the data types in fields Alley, Condition2, Electrical, GarageType, GarageYrBlt, Heating, LandContour, LandSlope, LotShape, MiscFeature, PavedDrive, RoofMatl, Street, Utilities


texts:
I am executing this code which works perfectly for me:
(I only have 'positive' and 'negative' sentiments):
from sklearn import metrics
print('Accuracy:',metrics.accuracy_score(test_sentiments, predicted_sentiments))
print('Precision:',metrics.precision_score(test_sentiments, predicted_sentiments, pos_label='positive'))

My question is: how come accuracy_score recognizes the positive label and precision_score does not?
ps: if I execute:
print('Precision:',metrics.precision_score(test_sentiments, predicted_sentiments))

or
print('Accuracy:',metrics.accuracy_score(test_sentiments, predicted_sentiments, pos_label='positive'))

They both fail.


texts: How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

texts:
In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and Attention() layers, implementing Bahdanau and Luong's attentions, respectively. (docs here and here.)

These new type of layers require query, value and key inputs (the latest is optional though). However, Query, Value, Key vectors are something I've always read referred to Transformer architectures.

What do these vectors represent, when it comes to Bahdanau and Luong attention? For example, if I want to train an RNN model for a common task (let's say time series forecast), what would these inputs represent?

EDIT:
I'm thinking about a seq2seq to make forecasts. The input would be a series of given length, and a series of external variables. The output would be the series shifted forward of n steps.


texts: There are 2 different ImageNet datasets: ImageNet 1k usually referred to in papers as just ImageNet and the full ImageNet dataset also called ImageNet 22k. Is the ImageNet 1k a subset of the 22k? And if so are the 1k categories represented the same way in the 22k version or maybe images of a certain class in the 1k version are aggregated from several different classes in the 22k version (like different dog breeds are merged into the "dog" class)? Can I somehow cut the 1k version from the 22k version and say the rest of the 22k version except the 1k version is a completely different dataset or even an out of distribution dataset for the 1k version? Any links to some trusted sources would be cool. Thanks


texts: Impact of varying sequence length in ensemble GRU model

texts:
I am using ensemble gru for my project and keeping different cell sizes for different models !For example, first gru model is of size 16 and the second is of 8 and 4 for the third model.
The model is running well but I don't see any difference in the results in keeping same unit size or different.
can anyone explain the impact of varying unit size for ensemble gru. ?
it would be great if answer is given with theoretical proof.


texts: Effect of Skewness and data range in machine learning

texts: I have a feature for machine learning as follow that skew to the left, and only have number in certain number range (here 0-2000). Will skewness and range of number affect the learning? If yes what should I do?


texts:
I am trying to work with Classification model. I am planning to train and test my model with a large dataset (Training with 80% and testing with 20% volume - no under/oversampling).
What I understand about classification models is that it focuses on a binary TRUE/FALSE or YES/NO type of prediction. However, I should also be using some evaluation metrics e.g. Accuracy/Precision/Recall etc. Ideally, there would be a threshold which I need to elevate/lower to meet my expectations.
My question - if I am to use any bespoke software solution (e.g. some Python/R library) for this, what am I expecting to see from my model to output? Is it as simple as YES/NO output or with some form of accuracy/precision number?


texts: Using an unmerged branch from StatsModels for Anaconda

texts:
I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive.

See this blog, which as some point mentions, almost casually: ``We can estimate a Two-Step Heckman Model in Python using an unmerged branch from StatsModels (this replicates the Stata two-step results).''

Yeah right.

My question: how to accomplish this?

StatsModels packages reside in three locations on my drive, see below the folders for linear_model.py as an example.

~/anaconda3/lib/python3.7/site-packages/statsmodels/regression/linear_model.py
~/anaconda3/pkgs/statsmodels-0.10.0-py37hdd07704_0/lib/python3.7/site-packages/statsmodels/regression/linear_model.py
~/anaconda3/pkgs/statsmodels-0.10.1-py37hdd07704_0/lib/python3.7/site-packages/statsmodels/regression/linear_model.py

I dread to just copy the heckman.py file in any of them and risk breaking Anaconda.


texts: Discrimination vs Calibration - Machine Learning Models

texts:
I came across a new term called Calibration while reading about prediction models.

Can you please help me understand how different it is from Discrimination.

We build ML models to discriminate two/more classes from one another

But what does calibration mean and what does it mean to say that "The model has good discriminative power but poorly calibrated/calibrative power`?"

I thought we usually only look for separation only between 2 classes.

Can help me with this with a simple example please?


texts: Faster grid search with small dataset to derive best params instead of full dataset?

texts:
I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to implement gpu running for the different algorithms (xgb,lightgbm..). Also the models which are from scikit-learn like random-forest dont run on gpu.
The idea is now, instead of 300 000 rows, I will create a small dataset with max. 500 rows, which need less time than the full dataset.

Could the minimal required sample size calculation help?

How big should the sample be? to get a good data distribution of the big dataset?


texts: Azure Cloud SQL - Querying large number of rows with Python

texts:
I have a Python Flask application that connects to an Azure Cloud SQL Database, and uses the Pandas read_sql method with SQLAlchemy to perform a select operation on a table and load it into a dataframe.

recordsdf = pd.read_sql(recordstable.select(), connection)

The recordstable has around 5000 records, and the function is taking around 10 seconds to execute (I have to pull all records every time). However, the exact same operation with the same data takes around 0.5 seconds when I'm selecting from a local SQL Server database.

What can I do to reduce the time it takes to load data from Azure to a dataframe? Would moving the entire Python application to Azure serverless help? Thanks

Additional Information

Azure database is on Standard tier with 20 DTUs
Database region has been configured to be close to my location
Ideally looking for the operation to take under 2 seconds


texts:
The choice of batch size is in some sense the measure of stochasticity :

On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact GD on the whole data, but allows for more exploration and performs in some sense a Bayesian inference.
Larger batch sizes approximate the exact gradient better, but in this way one is more likely to overfit the data or get stuck in the local optimum. Processing larger batch sizes also speed-ups calculations on paraller architectures, but increases the demand of RAM or GPU RAM.

Seems like a sensible strategy would be starting from smaller batch sizes to have a lot of exploration in the initial stages, and then increase gradually the batch size to fine-tune the model.
However, I have not seen implementing this strategy in practice? Did it turn out to be inefficient? Or the appropriate choice of learning rate  scheduler with the Dropout does this well enough?


texts:
I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation in change in property values. What are some possible ways of selecting the closest type of distribution?

I plan to use python, scipy and pandas to do this, although my question is more about what are the correct statistical tests to do to determine whether one distribution is a better fit to the data than another.


texts:
Let's say I have a dataset with multiple types of multiple ingredients ($salt_1$,$ salt_2$, etc).
Each $n\text{-th}$ variation of each ingredient vs flavor may be represented by an $n \times k$ matrix that where an ingredient corresponds with a particular value of "flavor".

A recipe consists of a $1\times n$ vector (where $n$ is the number of ingredients) where each value corresponds to the quantity of ingredient in the recipe.

A particular combination of ingredients, with particular weights, with some transformation, would result in a particular $1 \times k$ "flavor" profile, in this simple model.

One approach could be to formulate this as a  Probabilistic Matrix Factorization problem (I think), with $k$ being the number of flavor parameters. And combining the recipe vector with the flavor matrix might do the trick.

But the problem is, the flavor value of each ingredient (and each variation of the ingredient) in the ingredient-flavor matrix would be very very limited. The recipe flavor profile might have a corresponding flavor vector,  that too would be limited, and would not be available, at the beginning. So in order to capture the relationship between the ingredients and the flavor, the system would be dependent on user-submitted data on recipe/ingredient flavors.

Is there a way I could create clusters of recipes based on user flavor ratings and extrapolate these to the constituent ingredients or vice versa? Could this be done via some unsupervised learning algorithm?

I am quite new to this, I would appreciate some help or some pointers to which mathematical approaches I should be looking at to model this problem.


texts: Need help on Deep learning model on sentiment analysis

texts:
Before I present my problem, please note that I am a newbie in deep learning and I am trying things for the first time. Most of my code/logic were adopted from various references in the internet.
Goal : Build a LSTM/CNN model to classify the IMDB reviews available in tensorflow datasets
Approach 1 : 1) LSTM based - train data - 45000 (10% validation split),test data - 5000 , accuracy > 95% , validation_accuracy > 85% , glove embeddings of size 100 was used
Approach 2 : 1) CNN model - a) train data - 45000 , test_data - 5000
b) train data - 50% , test_data - 50%
accuracy > 95% , validation_accuracy > 85%
Code :  https://github.com/shankartmv/Deep-Learning-Work/blob/main/IMDB_Sentiment_reviews_using_tensorflow_dataset.ipynb
Problem : Test_data accuracy doesn't go beyond 52% with both the approaches.Most of the code/references available out there use test_data during training.  test_data wasn't part of my training.
Methodologies tried to increase test accuracy :

Movie reviews length (padded) and max number of words in the vocabulary
dropout
epochs
train test split ratio
embeddings trainable=true/false
with and without GLove word embeddings

My guess is there isn't enough training data. I need help on how to increase the test data accuracy.


texts: How to implement Access control for Headers in r?

texts:
When i call service from client side then give that error.
No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘null’ is therefore not allowed access. The response had HTTP status code 404.

I was trying to set request header-

cors <- function(res) {
res$setHeader(“Access-Control-Allow-Origin”, “*”)
plumber::forward()
}

But it’s not working

Please tell me how to set request header in plumber.

Again, I cannot install httr and curl packages, is these packages needed in this case.


texts:
I have customer data with the products they purchased and the purchase date.

I want to extract a result that shows each customer and the first two fruits they purchased.

My actual set has 90000 rows with 9000 unique customers. I have tried groupby and summarise functions but I would like to be able to use summarise with condition like we use select with a where clause.
Thanks for your suggestions


texts:
So, I have 9k of 1 second wav files of a person speaking. These are labeled by whether the person speaking is wearing a face mask or not. I am supposed to come up with a machine learning model to classify these criteria.

So far I tried using KNN on the mfcc feature of the audio. This gets around 56% accuracy on test data.

I also tried to convert the wav files to jpg spectograms and apply a CNN. This one gets 60% accuracy on test data.

But I don't have that much experience, I am not sure which features of audio would best help to this problem.

Also, if you could recommend a machine learning model for this particular problem.


texts:
I am a new user in StackExchange and a new learner of Data Science. I am working on better understanding how to estimate the results collected, specifically fake users extracted from a dataset running some analysis.
Using a specific algorithm, I found some users
User_Alg

user1
user2
user3
user28
user76
user67

and I would like to estimate accuracy of my algorithm comparing with the dataset which contains all the fake users manually labelled:
User_Dat
user1
user5
user28
user76
user67
user2
user29

As you can see, there are some users, in my extracted list (User_Alg), who are missing, i.e. not included in the list manually labelled (all the fake users in the dataset; User_Dat).
I have thought to use a confusion matrix to check the accuracy, but I would like to know from people with more experience in statistics and machine learning than me, if such method can be ok and how it looks like, or if you recommend another approach.
Thanks for your attention and your time.


texts: How word2vec can be used to identify unseen words and relate them to already trained data

texts:
I was working on word2vec gensim model and found it really interesting.
I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the trained model.

Is this possible?
Can word2vec be tweaked for this?
Or the training corpus needs to have all the words of which i want to find similarity.


texts: Should I standardize first or generate polynomials first?

texts:
Recently I am dealing a classification problem with some algorithms, say logistic regression.

When I preprocess my data, I standardize all my features and then generate polynomial features based on them.

from sklearn.preprocessing import PolynomialFeatures, StandardScaler

and I do

# features is my entire features dataset, labels excluded
features = poly.fit_transform(features)
features = std.fit_transform(features)

After finishing training my model, the accuracy is, say about 80%. Then I invert the two line of preprocessing code to

features = std.fit_transform(features)
features = poly.fit_transform(features)

I have read this post but it seems the answers is not strong enough to help me figure it out.

Should I standardize my data first or generate polynomials from original features first?


texts: How to predict product price range using machine learning algorithms

texts:
I would like to run thru regression algorithms(linear, SVM, Random forest, Xgboost) thru historical data to predict the price range of a product.

To get the price range am going to use top predict value as upper bound and next best value as lower bound).

Ex: Product a price range preicttion - $1000 to $5000.

$5000 is from top predicted price.
$1000 is from next best-predicted price.

Is my approach correct?

Yet to receive data, so do not have much data sample to provide.

Please suggest.


texts: Reinforcement algorithm for binary classification

texts:
I am new to machine learning, but I've read a lot about Reinforcement Learning in the past 2 days. I have an application that fetches a list of projects (e.g. from Upwork). There is a moderator that manually accepts or rejects a project (based on some parameters explained below). If a project is accepted, I want to send a project proposal and if it is rejected, I'll ignore it. I am looking to replace that moderator with AI (among other reasons) so I want to know which Reinforcement Algorithm should I use for this.

Parameters:
Some of the parameters that should decide whether the agent accepts or rejects the project are listed below. Assuming I only want to accept projects related to web development (specifically backend/server-side) here is how the parameters should influence the agent.

Sector: If the project is in related to IT sector it should have more chances of being accepted.
Category: If the project is in the Web Development category it should have more chances of being accepted.
Employer Rating: Employers having a rating of over 4 (out of 5) should have more chances of being accepted.

I thought Q-Learning or SARSA would be able to help me out but most of the examples that I saw were related to Cliff Walking problem where the states are dependent on each other which is not applicable in my case since each project is different from the previous one.

Note: I want the agent to be self-learning so that if in the future I start rewarding it for front-end projects too, it should learn that behavior. Therefore, suggesting a "pure" supervised learning algorithm won't work.

Edit 1: I would like to add that I have data (sector, category, title, employer rating etc.) of 3000 projects along with whether that project was accepted or rejected by my moderator.


texts: Sequences of time series data with only 1 output classification

texts:
So I'm facing a problem where I have a sequence (30h of data with 10sec intervals) and which is labeled to a class (3 different classes) for the whole sequence. I'm used to work with time series who have an output for each time sample in the sequence and now I'm block with this.

So far I have thought of performing a statistical study of the sequence and create a dataset where each row contains the results of that study from a sequence and the target. This way I'll have a dataset with a Y for each X (but I'll lose a lot of information from the sequences)

I come here to see if any of you have faced a similar problem before or if you have any ideas of how to approach this.


texts: scoring argument in scikit-learn LassoCV, LassoLarsCV, ElasticNetCV

texts: I was looking at the arguments in the linear regularization methods with cross validation within scikit-learn. RidgeCV has an argument scoring which is None by default but one can use a custom scorer (e.g. RMSE). I noticed that no such option is given for the rest of the classes (LassoCV, LassoLarsCV, ElasticNetCV). Why is this? Is MSE used by default in all of them?


texts: Creating dummy variables to match fitted model at inference

texts:
I have built a machine learning classifier using Sklearn and pandas as my main tools. Now, one of the input features to the model is country (to letter country code such as US). I have fit a model using the pd.get_dummies function.

Now I want to run inference on the data but a few of the countries haven't appeared in my dataset over the past months, so the pd.get_dummies function is misaligned with the fitted model.

How can this dealt with?


texts:
I have a sample with 10 independent variables (X1, X2, X3 ....), and multiple output labels (y1, y2, y3).

Here y1 will depend on X1, X2

y2 will depend on X3, X4 and so on.

y1, y2, y3 might or might not be correlated.

Can you please suggest pros and cons of clubbing this into single model, or should i go with multiple models with single output

X1,X2,y1

X3,X4,y2

.....

Thanks


texts:
Pr. Hinton in his popular course on Coursera refers to the following fact:

Rprop doesn’t really work when we have very large datasets and need to perform mini-batch weights updates. Why it doesn’t work with mini-batches? Well, people have tried it, but found it hard to make it work. The reason it doesn’t work is that it violates the central idea behind stochastic gradient descent, which is when we have small enough learning rate, it averages the gradients over successive mini-batches. Consider the weight, that gets the gradient 0.1 on nine mini-batches, and the gradient of -0.9 on tenths mini-batch. What we’d like is to those gradients to roughly cancel each other out, so that the stay approximately the same. But it’s not what happens with rprop. With rprop, we increment the weight 9 times and decrement only once, so the weight grows much larger.

As you can see, the central idea behind SGD is that the successive gradients in mini-batches should be averaged. Does any one have any valid formal source for this? is there any justification? I've not encountered to any proof till now.


texts: Ordered and unordered categorical features – terminology

texts:
In the famous book "The Elements of Statistical Learning" by Hastie et al., the authors denoted unordered categorical variables as qualitative variables / nominal variables / factors.

I wonder, do other statisticians strictly follow this or some authors can use these terms (qualitative variables / nominal variables / factors) not only for unordered categorical variables but also for ordered categorical variables?


texts: Cleaning the univariate dataset with high noise

texts:
At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension.

For example:

[1]: [10, 12, 13, 15, 16] indicates that the sensor [1] will have some operating duration like [10, 12, 13, 15, 16].

I want to see the range of operating duration for each sensor, by measuring the mean and standard deviation for each sensor. But my problem is in my dataset, each sensor has many noises. For example:

[1]: [1, 1, 1, 1, 1, 2, 2, 2, 10, 12, 13, 15, 16, 200, 400, 500].

You could see that sensor [1] has many noises like 1, 2, 200, 400 and 500. In my dataset, there are many cases like this.

Without removing the noise, the standard deviation is always smaller than the mean. This makes my duration analysis not meaningful.

So my question is: I want to ask if there is any method for removing the noises like that in my dataset.

Thank you.


texts: I am trying to write a batch generator function so that I can train my Keras model using it but I don't know from where to start. Does anyone have some examples or tutorials?


texts: Train classifier on synthetic images to recognise real images

texts:
I am trying to train a classifier (let's say to classify an object X or not X). But I don't have too much real images of object X which I want to classify. So I made some synthetic images of my own using some photo editing tools, but the problem is after feeding synthetic data my classifier doesn't recognise the real object. And since I had like 2-3 images of real object so I can't feed it more real data.
Is there any solution on how to approach such problem with synthetic data to recognise real ones?


texts: Cost function for support vector regression

texts:
The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):

minimise:
\begin{align*}
C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2}
\end{align*}

subject to the constraints:

\begin{align*}
& y_{i} - <w,x_{i}> - b \leq \epsilon + \xi_{i} \\
& <w,x_{i}> + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\
& \xi_{i}, \xi^{*}_{i} \geq 0
\end{align*}

I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.

The paper says that the "goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this."

But I am not sure what "flat" means.

Does someone know?


texts: creating a dataframe out of nested loop data

texts:
I have three parameter arrays ,Parameter1 = [6,7,8],Parameter2 = [11,12] and Parameter3 which can be calculated using Parameter1 and Parameter2. There is a mapping between parameter 1 and 2.

I want to create a dataframe in pandas which looks like :

There are multiple parameters and multiple rows so can not do manually. Can we do this using nested for loops


texts: Transfer learning between Language Model and classification

texts:
Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem.

What is exactly taken from the Language Model training? Is it just the word embeddings? Or is it also the weights of the LSTM cell? The architecture of the neural net should be quite different - where in a LM you would output a prediction after every sequence-step, in a classification problem you would only care about the output of the final sequence step.

(I would happy to know what is the general practice, and also if anyone knows how fast.ai does it)


texts:
Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.
Problem
I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example:

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline).  Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.
Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks.


texts: Sentiment analysis of the target in articles

texts:
The goal is as follows: I have a big article and I want to define the sentiment of the particular word. For example, the article describes pros and cons of bikes and cars and I want to find the sentiment of the word car.
In such an example I cannot use document-level SA as the article itself can be positive while the car was mentioned in a negative way.
So, I studied papers related to aspect-based sentiment analysis, but my constait is absence of data for training NNs. Hence, I concentrated on the approaches that basically do not involve training process. One of my attemts was to build sentiment analysis tool using word2vec and K-Means so that each cluster corresponds to one of three sentiments (pos, neg and neu).
It actually worked great but I found that for some reason one word can be at two clusters at the same time. Plus it generally goes not give sentiment for specific keyword but for all aspects found in the text.
Another problem is that basically cannot test the correctness of the output if only not to read the text by myself and check whether the keyword belonged to the correct cluster or not.
So I came to the decision to make summarization of the article first and then applying sentiment analysis (like sentiwordnet or similar).
Question 1
Are there ways to improve word2vec+KMeans approach? Is it even worse improving?
Question 2
Is it a good idea to go through text summarization before sentiment analysis?
Question 2
Is there better way to find sentiment of the particular word without training process (due to no training data and small amount of unlabeled data)?


texts: MLP classifier Gridsearch CV parameters to tune?

texts: I'm looking to tune the parameters for sklearn's MLP classifier but don't know which to tune/how many  options to give them? Example is learning rate. should i give it[.0001,.001,.01,.1,.2,.3]? or is that too many, too little etc.. i have no basis to know what is a good range for any of the parameters. Processing power is limited so i can't just test the full range. If anyone has a general guide of which are the most important to tune and a general range that would help me a lot moving forward. Thanks!


texts: How do I pivot a table while keeping the other columns intact

texts:
I have looked around and I have not been able to exactly get the result I am looking for.

I have the following Dataframe:

My goal, is to have a dataframe that for every year-state combination, has columns white, black, hispanic, asian, with values in Score. I would expect the dataframe to get a lot shorter.

I have tried to do the following:

(raceexpanded is the df provided)

racedf = raceexpanded.pivot(values='Score', index=['Year','State'], columns='Race')

but I get the error ValueError: Length of passed values is 1908, index implies 2


texts: How to calculate the fold number (k-fold) in cross validation?

texts: I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?


texts: What is the possible range of SVR parameters range?

texts: I'm working on a regression problem. While tunning the Parameters of SVR I got the following values c=100, gamma= 10 and epsilon =100. For which I got 95 percent r-square. My question is what is the theoretical range of these parameters values.?


texts: LSTM's for timeseries with additional regressors

texts:
I have a dataset consisting of the weekly sales of 3,000 stores over the past 5 years, and have constructed a LSTM to forecast the next year of sales, given the previous year of sales. At each timestep $t$, the features are the previous 52 observations.
$$X_{t-52:t} \rightarrow LSTM \rightarrow \hat{X}_{t:t+52}$$
The LSTM is trained using a rolling window over the timeseries, starting with the input of $X_{0:52}$ and output of $X_{52:104}$ and ending with the input of $X_{T-52:T}$ and output of $\hat{X}_{T:T+52}$, so that the cell state is built up over the whole series to eventually forecast $\hat{X}_{T:T+52}$.
However, I also have a dataset $E$ of events for which I know the dates both past and future. These are events which may affect sales such as Christmas or Easter. It is encoded as a binary vector of length n_events=30 for each week $t$. I'm wondering what the best way would be to include this data as my forecast. Ideally, I want the NN to be able to learn the uplift or decrease associated with each event and use this information in its forecast. It is more complex than seasonality because some events such as Easter move around in a way that cannot be learned by the NN.
However, the events for the next year at any timestep $t$ is a tensor of shape [n_events, forecasting_horizon] = [36, 52] with 1972 individual "features" and hence is too large to realistically include in the input to the LSTM. What other methods could I use to feed these features into the model in a way that would inform its forecast over the next 52 weeks?


texts: How can I calculate support/confidence/lift on a dataset in order to find frequent itemsets and determine association rules, in python? What would be the most effective method for predicting and offering recommendations on a test set of incomplete "shopping carts"? I am limited to the Anaconda distribution so I cant use packages such as orange3, etc.


texts: Methods of de-emphasizing some dimensions in a cluster analysis

texts:
I'm trying to understand how "weightings" on different dimensions in a cluster analysis might relate to the range of values along a given dimension in the dataset.
DATA SET
List of 1,000 to 10,000 objects with 6 dimensions:

X position
Y Position
Transparency (either 0% or 100% in this data set)
Color (3 dimensional: RGB, HSL)

GOAL:
Break the objects into an unknown number of clusters.
In identifying clusters, I'd like to de-prioritize the color dimension.  It's less important than position, but it would be needed to "break ties" on the borders of position clusters.  There might be some similarly positioned objects that have different colors, and they should be placed in separate clusters
I'm using python's scikit-learn, although I would be open to using any other python tool. I see that there are weighting options for samples, but not for dimensions.
One approach I could use would be to limit the range of values on color.  In other words, map H, S, and L to a range between [0,0.333].  Since the other values are on a range of [0,1] then the values for widely different color values would be closer together than the position vector, so they should impact the cluster determination less.  (I would naively call this a 1/3 weighting for the color dimensions)
I'm not familiar enough with the clustering algorithms to know the implications of doing this.  With the internet search terms I've been using, I haven't been able to hit upon a good discussion of this topic of "weighting dimensions" in cluster analyses.
My questions are:  Is this method of "range shortening" an appropriate means to accomplish my goal?  Does scikit-learn or any other python library allow for an explicit dimension weight parameter in cluster analysis?  How do others accomplish this goal?  How do experts describe this sort of problem so that I can find further resources on it?


texts: How to apply a trained Random Forest model to a new data set in R?

texts:
So I have a data set that is essentially football players statistics in 2017 and 2018. I have trained my model to use the 2017 data to predict the 2018 number of touchdowns. My code is below:

set.seed(1)
data.rf <- randomForest(2018_td ~ ., data = data, proximity = TRUE)

In my  data set, I had the actual # of touchdowns in 2018, and trained a random forest algorithm to predict that value. Now, I want to apply the trained random forest on the same 2018 data set, but to predict 2019 # of TD, that I don't have.

I'm not sure if I'm missing something or if I have a fundamentally wrong understanding of how RF works. How would I go about predicting those 2019 values from my data.rf model?


texts: How to run a saved TensorFlow Model? (Video Prediction Model)

texts:
I was reading this paper. The code is available in GitHub. In the README.md file, they've mentioned how to train the model

python prediction_train.py

with optional parameters.
Can anyone please explain how do I use this model to predict a video sequence? I'm new to deeplearning and tensorflow. So, I'm not able to understand the code properly. My current task is to just run the code and see the output (i.e. the videos predicted by this model).

All I could understand was, it uses tensirflow saver to save the checkpoints. I'm guessing these checkpoints are the intermediate trained model after few epochs (2000 in this case). How to use these models to predict the next frames of a video?

Any help is greatly appreciated :)


texts: What is the meaning of semi-handcrafted features?

texts: I know handcrafted features are features which are being created by a human made algorithm. But what are semi-handcrafted features?


texts: Suggestions on using model in production 1 test at a time

texts:
I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious:

  ParentPath                                  ParentExe
0   C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe
1   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs
2   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs
3   C:\Windows\System32                         svchost.exe
4   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs

ChildPath                                   ChildExe    Suspicious
C:\Windows\System32                         conhost.exe  0
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe   0
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe   0
C:\Program Files\Common Files               OfficeC2RClient.exe  0
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe  1
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe  0

I have used sklearn for label encoding and one hot encoding on the data:

#Import the dataset
X = DBF2.iloc[:, 0:4].values
#X = DBF2[['ParentProcess', 'ChildProcess']]
y = DBF2.iloc[:, 4].values#.ravel()

#Encoding categorical data
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
#Label Encode Parent Path
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Label Encode Parent Exe
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Label Encode Child Path
labelencoder_X_3 = LabelEncoder()
X[:, 2] = labelencoder_X_3.fit_transform(X[:, 2])
#Label Encode Child Exe
labelencoder_X_4 = LabelEncoder()
X[:, 3] = labelencoder_X_4.fit_transform(X[:, 3])

#Create dummy variables
onehotencoder = OneHotEncoder(categorical_features = [0,1,2,3])
X = onehotencoder.fit_transform(X)

I have split the data into a training and test set and run it on my gpu box with a nvidia 1080.  I have tuned the hyperparameters and am now ready to use the model that is trained in a production environment with one test sample being tested at a time.  Lets say I just want to test one sample:

   ParentPath            ParentExe     ChildPath           ChildExe
0  C:\Windows\Malicious  badscipt.exe  C:\Windows\System   cmd.exe

The issue that I am running into is the training set has seen the ChildPath "C:\Windows\System" and the ChildExe "cmd.exe"  which are normal, but the training set has not seen the ParentPath "C:\Windows\Malicous" or ParentExe "badscipt.exe" so these have not been label or one hot encoded.  My big question is how do I handle one test feature where part of it has not been trained?

I have seen examples using feature hashing but im not sure how to apply that or if that would even solve this problem.  Any help or pointers would be greatly appreciated.


texts:
I am approaching the use of python, i was wondering if it is correct to install  Spyder in the 'root' environment and then point to other environments when coding on a project.
Is it correct or it is better to install a Spyder for each environment?
Thanks


texts:
Below is the code I am trying to execute.

x = np.random.normal(size=1000)
fig, ax = plt.subplots()
H = ax.hist(x, bins=50, alpha=0.5, histtype='stepfilled')

Can anybody elaborate what fig, ax = plt.subplots() doing here ?

Thank you.


texts: Conditional entropy calculation in python, H(Y|X)

texts:
Input
X: A numpy array whose size gives the number of instances. X contains each instance's attribute value.
Y: A numpy array which contains each instance's corresponding target label.

Output : Conditional Entropy

Can you please help me code the conditional entropy calculation dynamically which will further be subracted from total entropy of the given population to find the information gain.

I tried something like the below code example. But the only input data I have are the two numpy arrays. can you please help me correct this ?                              [code]

def gain(data, attr, target_attr):

    val_freq = {}
    subset_entropy = 0.0
for record in data:
        if (val_freq.has_key(record[attr])):
            val_freq[record[attr]] += 1.0
        else:
            val_freq[record[attr]]  = 1.0
for val in val_freq.keys():
        val_prob = val_freq[val] / sum(val_freq.values())
        data_subset = [record for record in data if record[attr] == val]
        conditional_entropy += val_prob * entropy(data_subset, target_attr)


texts: Extract key phrases from a single document

texts:
I need to extract relevant key phrases from a single document. Since I don't have a lot of documents, TF-IDF doesn't really work.

Currently I'm using TextRank. It produces okay-ish result - some really good phrases along with a lot of garbage.

Is there a better algorithm to use for this? Can anyone give me a rundown of available options?

Real-world use case: I'm developing a help desk app that comes with Knowledge Base (a bunch of articles, think of it as FAQ). When a user writes a new support ticket I want to extract key phrases and find the most relevant KB articles. Overall there is not enough data to train a model. I need to compare sets of key phrases I think.


texts: "Results do not have equal lengths" using ldply in R package plyr

texts:
I've found a few similar questions, but I am new to R and can't figure out how it applies to my specific problem. Here is my code:

library(rvest)
library(plyr)
library(stringr)

#function passes in letter and extracts bold text from each page
fetch_current_players<-function(letter){
  url<-paste0("http://www.baseball-reference.com/players/", letter, "/")
  urlHTML<-read_html(url)
  playerData<-html_nodes(urlHTML, "b a")
  player<-html_text(playerData)
  player
}

#list of letters to pass into function
atoz<-c("a","b","c","d","e","f","g","h",
        "i","j","k","l","m","n","o","p","q","r",
        "s","t","u","v","w","x","y","z")
player_list<-ldply(atoz, fetch_current_players, .progress="text")

So what this code is trying to do is use the URL structure of this website to pass a list of the letter A through Z into my function to produce a list of names that are in bold. I think the problem is that each list of players it returns is of different lengths and that is producing an error as when I manually type in each letter into the function the function appears to work.

Any help is appreciated, thanks!


texts: How do I turn images dataset into a numpy array?

texts:
I have a directory for a dataset of images, I I want to transorm it to a numpy array in order to be able to fit an image generator to it. What I have tried to do is the following:

trainingset_temp = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

import cv2
import glob

trainingset = []
files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Haze') # your
image path
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Sunny')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Snowy')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Rainy')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

trainingset = np.array(trainingset,dtype='float32')

but it gives me the following error message:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-44-2a679dbff06e> in <module>()
     54     batch_size=batch_size,
     55     class_mode="categorical",
---> 56     shuffle=True
     57 )
     58

1 frames
/usr/local/lib/python3.6/dist-
packages/keras_preprocessing/image/directory_iterator.py in __init__(self,
directory, image_data_generator, target_size, color_mode, classes,
class_mode, batch_size, shuffle, seed, data_format, save_to_dir,
save_prefix, save_format, follow_links, subset, interpolation, dtype)
    104         if not classes:
    105             classes = []
--> 106             for subdir in sorted(os.listdir(directory)):
    107                 if os.path.isdir(os.path.join(directory, subdir)):
    108                     classes.append(subdir)

FileNotFoundError: [Errno 2] No such file or directory: array([],
dtype=float32)

Can someone help me? Thanks in advance.


texts: Is it effective to use one-hot encoding when its dimension is as large as thousands

texts: Here I try to construct a classifier using DNN(deep neural network) with its inputs being many portfolios. In essence, each portfolio contains several stocks which are labeled by there inner-code, for example "1430" or "5560", etc. Since the inner-code is discrete digital number, I prefer to use on-hot encoding to represent each portfolio. However, there are as many as a few thousand different inner-code which means that the dimension of on-hot code may also be that large. I wonder whether the dimension is too large for real training and is there any way to solve or ease this problem, such as using PCA afterward.


texts:
Could you please provide me an example of how I can compute the accuracy for a kmeans clustering?
I split my dataset into train and test sets and computed the predicted clusters for the train set. However, I do not know how to compiute its level of accuracy and see if it is good not also for the test.

An example with a sample dataset of your choice it would be extremely useful.


texts:
It is quite hard to name the title properly as I just started to learn ML, will try to explain here. I want to practice ML by creating Movie suggestion algorithm. I came up with the following list of dimensions/features:

Rating
Number of votes
Genre
Actors
Directors
Writers
Year
Combination of Actors/Directors/Writers working in the team
Combination of Year and Actor or Director or Writer
Actor or Director or Writer working in some genre

There is no problem for me for numeric fields, but in case of actors I have multiple values. How to create feature for the actors?


texts: Similarity score: Can Sklearn SVR predict values greater than 1 and less than 0?

texts:
I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve a similarity problem. Using GridSearchCV, I am finding the best hyper parameters using the scoring as "R2". The best hyperparameters are C=1, cache_size=200, coef0=0.0, degree=3, epsilon=0.001, gamma=0.005, kernel='rbf', shrinking=True, tol=0.001.

I fit the training data and training label as model.fit(X_train, Y_train)

Now I use test data in the same model as: prediction = model.predict(X_test)

The reason I am using SVM-Regression is to find similarity between two inputs. However, for some of the test data, the prediction contains value in negative (less than 0) and for all the same vs same comparison it returns the value as 1.09469178. I am expecting the value to be between 0 and 1. Is this normal or am I doing something wrong?


texts: Linear Regression in R Mapreduce(RHadoop)

texts:
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs

Error:

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  :
   hadoop streaming failed with error code 1

Code :

Sys.setenv(HADOOP_HOME="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop")
Sys.setenv(HADOOP_CMD="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/bin/hadoop")

Sys.setenv(HADOOP_STREAMING="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.7.0.jar")
library(rmr2)
library(rhdfs)
hdfs.init()
day_file = hdfs.file("/hdfs/bikes_LR/day.csv","r")
day_read = hdfs.read(day_file)
c = rawToChar(day_read)

XtX =
  values(from.dfs(
    mapreduce(
      input = "/hdfs/bikes_LR/day.csv",
      map=
        function(.,Xi){
         yi =c[Xi[,1],]
         Xi = Xi[,-1]
         keyval(1,list(t(Xi)%*%Xi))
       },
  reduce = function(k,v )
  {
    vals =as.numeric(v)
    keyval(k,sum(vals))
  } ,
  combine = TRUE)))[[1]]

XtY =
 values(from.dfs(
    mapreduce(
     input = "/hdfs/bikes_LR/day.csv",
     map=
       function(.,Xi){
         yi =c[Xi[,1],]
         Xi = Xi[,-1]
        keyval(1,list(t(Xi)%*%yi))
       },
     reduce = TRUE ,
     combine = TRUE)))[[1]]
solve(XtX,XtY)

Input:
------------

instant,dteday,season,yr,mnth,holiday,weekday,workingday,weathersit,temp,atemp,hum,windspeed,casual,registered,cnt
1,2011-01-01,1,0,1,0,6,0,2,0.344167,0.363625,0.805833,0.160446,331,654,985
2,2011-01-02,1,0,1,0,0,0,2,0.363478,0.353739,0.696087,0.248539,131,670,801
3,2011-01-03,1,0,1,0,1,1,1,0.196364,0.189405,0.437273,0.248309,120,1229,1349
4,2011-01-04,1,0,1,0,2,1,1,0.2,0.212122,0.590435,0.160296,108,1454,1562
5,2011-01-05,1,0,1,0,3,1,1,0.226957,0.22927,0.436957,0.1869,82,1518,1600
6,2011-01-06,1,0,1,0,4,1,1,0.204348,0.233209,0.518261,0.0895652,88,1518,1606
7,2011-01-07,1,0,1,0,5,1,2,0.196522,0.208839,0.498696,0.168726,148,1362,1510
8,2011-01-08,1,0,1,0,6,0,2,0.165,0.162254,0.535833,0.266804,68,891,959
9,2011-01-09,1,0,1,0,0,0,1,0.138333,0.116175,0.434167,0.36195,54,768,822
10,2011-01-10,1,0,1,0,1,1,1,0.150833,0.150888,0.482917,0.223267,41,1280,1321

 Please Suggest me any mistakes.


texts: Pandas Data Frame: Calculating custom moving average

texts:
I have a time series containing stock price data.
I would like to calculate the Money Flow Index (MFI) for each row.
Given that the MFI, uses the previous approx. 14 rows to calculate the current MFI, what would be the best approach to doing this?
The below calculates the current MFI for a given DataFrame, but I would like to implement this for each row in the data frame
# Typical Price =(high price + low price + closing price) / 3
tp=(hst['High']+hst['Low']+hst['Close'])/3
tp=tp.to_frame(name='Price')

# Raw money flow = typical price x volume
tp['Raw']=tp['Price'] * hst['Volume']

# Identify flow (upwards or downwards)
tp['Direction'] = np.where(np.nan_to_num(tp['Raw'].shift(1))>tp['Raw'], 'up', 'down')
mfr=0
# Money flow ratio = (14-day Positive Money Flow) / (14-day Negative Money Flow)
if tp.loc[tp['Direction']=='down']['Price'].sum(axis=0) != 0:
    mfr=tp.loc[tp['Direction']=='up']['Price'].sum(axis=0) / tp.loc[tp['Direction']=='down']['Price'].sum(axis=0)

# MFI = 100 - 100 / (1 + money flow ratio)
100-(100/(1+ mfr))

I managed to work this using the 'apply' function, but this function only allows for 1 argument, whilst I would need to pass more than 1 argument to compute the above.


texts: Are there any tree-based models that use a genetic algorithm to generate the trees?

texts:
I have a large dataset (195 features x 20m samples) that I have trained using XGBoost. I would like to see if a genetic algorithm can beat XGBoost since the data has so much noise it is prone to overfitting.
I would like to use a  tree-based model so I don't have to standardize the data, and the features do have some interrelationships.
Are there any python packages that have this all done? Ie.that can create trees through a genetic optimization process?


texts: Opening a 20GB file for analysis with pandas

texts:
I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I keep getting memory errors.

From your experience is it possible? If not do you know of a better way to go around this? (hive table? increase the size of my RAM to 64? create a database and access it from python)


texts: What is a genetic algorithm, and what are its practical advantages over other algorithms? Is it similar to any commonly used machine learning algorithm like linear/logistic regression, neural networks, or tree-based methods like gradient boosting and random forests? I've heard it's based on "mutated" combinations of other models. Does this make it more like an ensemble?


texts: Evaluate clustering by using decision tree unsupervised learning

texts:
I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion and obviously if someone is aware of this method it would be great if he/she could explain to me the whole idea.
Clusters have been made to the data set (sample of 250000 rows and 5 features out of 500000 rows) by using k-prototypes as one of the features is categorical.
All the combinations of k= 2:10 and lambda = c(0.3,0.5,0.6,1,2,4,6.693558,10) have been made and 3 methods to figure out the best combination have been use.

Elbow method (pick the number of clusters and lambda with the min WSS)

Silhouette method pick the number of clusters and lambda with the max silhouette)

Decision tree

They build a decision tree for the data and after that they calculated for every different clustering combination the following value:
(inverse leaf size weighted within cluster purity)* cluster size/ total obs
and the picked the combination which had the max value. (k=10 and lambda=4)
So my question is:
Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?
Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is?
And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?
I would really appreciate if someone can help me with that.
Thanks in advance!


texts: What are available methods for modeling startup survival rates?

texts:
I am interested in modeling startup companies failure and success rates to describe what is the representative startup.

I have 40 companies in a dataset. Each company is represented as a list of all the investment financing rounds it has gone through. For example:

Company 1: Seed Round, Series A Round
Company 2: Seed Round, Series A Round, Series B Round
Company 3: Seed Round, Series A Round, Series B Round
Company 4: Seed Round, Series A Round, Series B Round, Series C Round
Company 5: Seed Round
Company 6: Series A Round, Series B Round
Company 6: Series A Round

And so on.

I can see that each round can be represented as a state in a Markov Chain, and transitions are only allowed from earlier stages to later stages. I can go from Seed to Series A, and from Series A to Series B, but not from Series B to Series A.

So we have N-order markov chains (production data has N <= 4).

The output I'm looking for is a binary tree chart showing each stage as a node and each node can either transition to the next node or to a final state meaning the company has failed.

This problem can also be seen as a real options model...

Any ideas on how to implement this model?

I can code in Python or Ruby (but I am no expert).


texts: Can I classify a healthy population into clusters found in an injured population?

texts:
Just a bit of context: We are applying Machine Learning algorithms in the field of Human Biomechanics. In a previous project, my colleagues identified three different subgroups in an injured population performing a movement utilising hierarchical clustering. In order to be able to classify new subjects, we developed a classification algorithm (using the same dataset) that was able to allocate each case to the predefined cluster.

We currently are working in another study that involves new comparing injured and healthy subjects performing the same task in which the clusters were found.

My questions are, is there any way to know to what extent clusters found in injured athletes apply to healthy athletes? If I used the same algorithm and features, what could be the effects/consequences? can I justify it? What are the limitations?

Any answer or directions to literature will be greatly appreciated as I am struggling to find any similar situation in the literature/internet.

Thanks everyone.

Cheers!


texts: Unable to Use The K-Fold Validation Sklearn Python

texts:
I have an dataset.
I am unable to use the K-Fold Validation. I am getting the error raised:

ValueError("{0} is not supported".format(y_type))
ValueError: continuous is not supported .

I do not want to do encoding to int, since it may affect the data, and also I want to understand why K-fold is not working.
Below is my python code.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor
from sklearn import cross_validation, metrics
from sklearn.cross_validation import train_test_split
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB

from sklearn import svm
from sklearn import preprocessing

 - `List item`

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00242/ENB2012_data.xlsx"
names=['Relative Compactness','Surface Area','Wall Area','Roof Area','Overall Height','Orientation','Glazing Area','Glazing Area Distribution','Heating Load','Cooling Load']
df = pd.read_excel(url,names=names)

#Feature selection
train=df.sample(frac=0.8,random_state=150)
test=df.drop(train.index)

#save the original values in a dataframe so we can compare later
test_loads=test[["Cooling Load"]]

#Create 2 lists of response values to train our model
Y1=np.array(train['Heating Load'])
Y2=np.array(train['Cooling Load'])

#Select the features
train_corr=train[['Overall Height','Relative Compactness','Roof Area','Surface Area']]
test_corr=test[['Overall Height','Relative Compactness','Roof Area','Surface Area']]
seed = 7
scoring = 'accuracy'
X_train,X_test,y_train,y_test=cross_validation.train_test_split(train_corr,Y1,test_size=0.2)
kfold = model_selection.KFold(n_splits=10, random_state=seed)
cv_results = model_selection.cross_val_score(RandomForestRegressor(), X_train, y_train, cv=kfold, scoring=scoring)
print (cv_results.mean())


texts:
I have a dataset where each sample is a list of ordered items, lets say grocery list ,  and a label from 6 categories . each list can have up to 120 items but the mean items is 12 items in a list.
i would like to embed each sample in a similar way to word2vec and maybe average the vectors for each list to get 1 vector per sample.
You think that word2vec is the right approach for it ? since the "texts" are very short and the location of each item is not contextual but more like the number purchased


texts: How to perform SMOTE-N when there is no majority vote?

texts:
In the SMOTE paper, the authors present the logic of creating synthetic examples when all features are nominal (section 6.2, SMOTE-N):

  To generate new minority class feature vectors, we can create new set
  feature values by taking the majority vote of the feature vector in
  consideration and its k nearest neighbors

Along with this example:

  Let F1 = A B C D E be the feature vector under consideration and let
  its 2 nearest neighbors be

  F2 = A F C G N

  F3 = H B C D N

  The
  application of SMOTE-N would create the following feature vector: FS =
  A B C D N

How would FS be chosen in the case that F3 = H B C I N? How does Value Difference Metric by Cost and Salzberg described in the paper assist in this case?


texts: Binary classification with time-series features

texts:
I have the following time-series features: Diastolic Blood Pressure, Systolic Blood Pressure, Heart Rate, RR variability and Arterial Blood Pressure. Each of these clinical parameters was measured for 900 seconds during a surgical procedure and after the surgery, the patient was assessed for acute kidney injury: 1(yes) or 0(no).

My training data kind of looks like this: (see below for screenshot)

Patient 1 Time(s) Features AKI

Patient 2 Time(s) Features AKI

and so on.

What approach would I take to utilize this data for the binary classification task?


texts: What is the lightest browser to launch a jupyter notebook?

texts: Currently I am using Firefox as default browser, I have no issue to naviguate with it, but I have a little limited memory (8GB, 6.8 available) and Firefox takes a lot of RAM for nothing big, the more I use it in the time, the more it "eats" the RAM. I want to know which is the lightest browser to use jupyter notebook to save my RAM for computations on my dataset for instance.


texts: Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit

texts:
Softmax cross entropy with logits can be defined as:

$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$

$l={\sum_{\forall i}}y_ilog(a_i)$

Where $l$ is the actual loss.

But when you look deep inside into C++ Tensorflow implementation of SoftmaxCrossEntropyWithLogits operation, the exact formula which they use is descibed as:

$l={\sum_{\forall j}}y_j ((z_j-max(z))-log({\sum_{\forall i}}e^{z_i-max(z)}))$

The part: $z-max(z)$ - is perfectly understood - it is just normalization which helps to avoid under/overflow.

BUT:

Where is the actual Softmax in their implementation?
Why from each $z_j$ they subtract $log({\sum_{\forall i}}e^{z_i-max(z)})$ before multiply it by $y_j$?

Note: One may argue that the code I provide is just Tensorflow's implementation of CrossEntropyWithLogits operation, but the actual SoftmaxCrossEntropyWithLogits operation - additionaly checks only dimentions and do not perform any more computation.


texts: When I would use a specific similarity coefficient over another?

texts:
Like using Jaccards over Dice.
I want real examples, of when I would prefer to use Jaccards, Dice, Cosine or any other similarity coefficient.


texts: Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution

texts:
I am working on a binary classification problem with unbalanced data (17% for positive class).
The problem is as following:
My three individual models when predicting on the test set (for which I don't have the labels) gives quite similar distribution as for Train set.
But ensemling these models, while giving slighltly better result (F1-score), it drastically changes the distribution on Test set going from ~20% to 5%.
My question is :
I am confused between choosing the best individual model which maintains almost same distribution but lose some efficiency
Or
The ensembled one who gives really different distribution
And I have no Idea about the Test set distribution.
Thanks for any help


texts: How can I get probabilities of next word with ELMO?

texts: ELMO is a language model, build to to compute the probability of a word, given some prior history of words seen. How can I get this probability from pretained ELMO model?


texts: PCA and k-means for categorical variables?

texts:
I have a clustering task at hand. The data that I have contains only categorical variables. So, k-modes seemed like the best option. But I am not sure what are the data pre processing steps required for the same ?
What I am doing right now is the following:

label encoding features which have ordinal values.

one hot encoding the others.

and that's all I am doing as part of data preprocessing steps. My feature space gets to 50 from original 4 after doing above mentioned  steps. I am getting 17 clusters as best number of clusters for silhouette score 0.60.
Also, I think doing Principal Component Analysis (PCA) to reduce dimensions and feature scaling doesn't make sense here as if I do this, I might as well use K-means. Would it be a good decision to run PCA and then use k-means for categorical variables ?


texts: Probabilistic gold standard vs Deterministic gold standard

texts:
I understand that we say something as a gold standard when it involves human intervention/judgement/review.

But can someone help me understand what's the difference between probabilistic gold standard and deterministic gold standard.

For ex: Patient has cancer or not - binary response - Deterministic gold standard which can be provided by humans.

Whereas Patient has 60% chance of being a cancer and 40% chance of not being a cancer. Am I right to understand that this is called as probabilistic gold standard but this can't be produced by Humans right?

Can any human/doctor for example, say this patient has 60% chance of being a cancer and 40 % chance of not  being a cancer?


texts:
I have following pipeline:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

Originally, the data looks like

After applying the estimator/StandardScaler()

X_train = prepare_data.fit_transform(X_train_raw)

Why are the values mostly/always negative now?

When I have a look at another variable in X_train it looks as it should (I guess):

Before

after


texts: Sentiment Analysis to Predict Stock Movement

texts:
I am using news sentiment analysis and machine learning to predict stock market movement.
I have two datasets: dataset 1 is historical stock price; dataset 2 is news headlines.
After I use VADER to get polarity score for dataset 2, I use panda.concat to combine with the first data set. Is it feasible that I use the output of this as training data for my model (both of the dataset output are already numbers, which are polarity score and historical prices) or do I need to do any additional task for the document representation part (e.g TF-IDF to vectorized both for my datasets)


texts: Deploying the prediction model under missing values for test data

texts:
I have successfully built a logistic regression prediction model based on data set that is complete and clean, i.e., there is no missing values and the data is consistent.
Now, for deploying the model and testing it for online use, there is missing values in the inputs, i.e., not all inputs are available to predict the target value.

Is there a standard way to deal with this?


texts: Calibrate the predicted class probability to make it represent a true probability?

texts:
Let's say that we have a simple binary classification model (a neural network -- NN) for classifying input images as "dog" ($y=1$) or "not dog" ($y=0$). Let's assume that the NN has one "sigmoid output" $\hat{y}$. After training, we feed an image of a dog and the output of the NN is e.g. $\hat{y} = 0.8$. As I understand it, this value of "0.8" is the probability ("confidence") that the particular image is a "dog", according to the model, and the prediction is that "it's a dog" (because 0.8 > 0.5).

But is the $\hat{y}$ really a (true) probability? What I mean is: if we feed the NN with many different images, and say for 100 of them the model gives the same $\hat{y} = 0.8$. For this $\hat{y}$ to be a true probability, I would expect 80 of these images to actually be dogs and 20 not dogs. To check this we would look at the $y$ (assuming that they exist). In the case when all the 100 images are actually dogs, our model made correct prediction (since it correctly labeled all of them as dogs, because 0.8 > 0.5 for all of the 100 images), but the $\hat{y}$ is not a true probability prediction, because it should be 1. This is actually the accuracy, $100/100 = 1$. On the other hand, if out of 100 images 80 are actually dogs and 20 are not dogs, then the model prediction has accuracy $80/100 = 0.8$, because it incorrectly classified all 100 images as dogs, again, but only 80 are actually dogs. However now the $\hat{y} = 0.8$ represents the true probability. In other cases, the accuracy can be e.g. 0.9, meaning that the predicted $\hat{y} = 0.8$ is not a correct probability, but is closer to it than in the first case.

So, first of all, is what I am saying correct? Second, does the "calibration" fix this issue, i.e. making $\hat{y}$ = accuracy allows the $\hat{y}$ to be interpreted as true probability?


texts: Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

texts:
Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as

xx = tf.layers.dense(inputs=xx, units=1, name='Prediction')

Should we use a sigmoid or tanh activation function after xx respectively to enforce the output of the 1 unit dense layer to be between [0,1] or [-1,1]?

To mention that most of the regression examples that I have seen they do not add an activation function after xx, and leave it to be open and take whatever value it ends up taking.