texts: Advice on imputing temperature data with StatsModels MICE

texts:
This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for five neighboring stations (LOC1:LOC5) over many years and I'd like to impute the missing data for any given site. Following the examples I have:

imp = mice.MICEData(dfLocal)
fml = 'LOC1 ~ LOC2 + LOC3 + LOC4 + LOC5'
mice = mice.MICE(fml, sm.OLS, imp)
results = mice.fit(10, 10)
print(results.summary())

dfLocal.dropna(axis=0, how='all', inplace=True)
imp.data = imp.data.set_index(dfLocal.index)

# In this case I only want to fill one specific set of missing data
# hence gap_start and gap_end
dfLocal.loc[gapStart:gapEnd, 'LOC1'] = imp.data[fillSite]

My understanding of MICE is broadly that missing values are imputed multiple times and then combined to find the best value from the many. The only way I've found to actually get any numbers out of the above code is with imp.data but I'm afraid that might just be one of the individual imputations before they're combined? All I can seem to get from fitting the model (results), though, is the summary?

I'm far from a statistician (and not much of a programmer either) so I've been reading through the code for mice.MICE and other resources on general MICE applications, but I'd appreciate any guidance on this as I can't find much about using statsmodels' MICE online. Normally I'd post some data on Gist but the full set is a bit large. That said, I'll upload it if ya'll think it would help.

Thanks!


texts: Machine Learning for hedging/ portfolio optimization?

texts:
With increasingly sophisticated methods that work on large scale datasets, financial applications are obvious. I am aware of machine learning being employed on financial services to detect fraud and flag fraudulent activities but I have a lesser understanding of how it helps to predict the price of the stock the next day and how many stocks of a particular company to buy.

Do the hedge funds still employ portfolio optimization techniques that are right out of the mathematical finance literature or have they started to use machine learning to hedge their bets? More importantly, what are the features that are used by these hedge funds and what is a representative problem set up?


texts: Agglomerative Clustering Stopping Criteria

texts:
I am trying to implement section 3.4 of paper Predicting Important Objects for Egocentric Video Summarization where they have created a distance matrix of frame histograms.

In short, let say Ω is mean of distances between all frames,DVis distance matrix.

I didn't understand what is meant by this:

  We next perform complete-link agglomerative clustering
  with DV , grouping frames until the smallest maximum interframe
  distance is larger than two standard deviations beyond
  Ω

Can this be achieved by setting the cutoff value to 2Ω in Matlab's clusterdata function ?


texts: Why do we calculate partial derivative of Error w.r.t output of Neural Network during backpropagation?

texts: As seen in this image we calculate partial derivative of Error w.r.t output of the output neuron. Shouldn't it be normal derivative? Does not that particular error is determined by only that output?


texts: decision rules for each feature (binary classification)

texts:
I have a collection of 10 features (all numerical) and a single binary outcome variable. I need to train a binary classification model, find the best features and compute thresholds for each feature.

To find the best subset of features is easy, I use logistic regression with L1 penalisation and it works nicely. However, the next step is to find thresholds for each subselected feature: if the values of a feature i, j,...k are above/below than some numerical values, than chances to be in the class A (rather than in B) are higher.

FOR EXAMPLE

Consider, we found that out of 10 original features, there are L1 leaves only 3: F1, F4,F7. And if you take three unseen (new) data with some values {F1_i, F4_i, F7_i}, where

F1 > 1.23
F4 < 7.15
F7 > 2.74

Then the new data point {F1_i, F4_i, F7_i} belongs to class A. Here {1.23, 7.15, 2.74} are thresholds.

I tried to explain the problem as clear as possible, but if it isn't, please let me know.

QUESTION What will be the best approach to solve this problem? How to compute thresholds?


texts:
What is the best option or rather options to predict how much order a customer will place in the future, say next 3 months on a monthly basis. Also will a customer place an order.

I used ARIMA to predict the future order but the MAPE is quite high. Basically what i have done is predict the probability of a customer placing an order using survival analysis and then multiplied that with the predicted order quantity.

Does that make sense or is there something fundamentally wrong with the approach?


texts: Confusion-matrix clarification from Python

texts:
confusion_matrix(y_test1, pred)
That is my codes

Confusion matrix, without normalization
True     [[724258    438]
value    [ 25396    302]]
          Predicted value

I understand that is how this is the confusion matrix. But I do not know the order of classification result.

0   [[724258    438]                1 [[724258    438]
1   [ 25396    302]]                0 [ 25396    302]]
       0         1                        1        0

Which order generally? I check the documentation, it does not tell me the specific result.


texts: Training and test split for time series analysis

texts:
I need help with below code:

dataset.tail(4)

                Date    Close
    1274    1/24/2019   4871.96
    1275    1/25/2019   4925.82
    1276    1/28/2019   4888.58
    1277    1/29/2019   4928.18

I want to split my dataset to training and test using:

training_data=dataset[dataset.Date<datetime(2018, 1, 1)]

But I get the following error:

TypeError
Traceback (most recent call last)
<ipython-input-74-904bd8b45617> in <module>()
----> 1 training_data=dataset[dataset.Date<datetime(2018, 1, 1)]
      2 training_data.tail(4)

TypeError: 'module' object is not callable


texts: Scikit learn train test split without mixing participants in trails

texts:
I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.

I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.

The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?


texts: The minimum number of groups for any class cannot be less than 2 error

texts:
cv_split = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=0)

# Performing cross-validation to estimate the performance of the models
for idx, est in enumerate(estimator):

    cv_results = cross_validate(est[1](), x_value, y_value, cv=cv_split)

I got following error from this code

  ValueError: The least populated class in y has only 1 member, which is
  too few. The minimum number of groups for any class cannot be less
  than 2.

and my y_value looks like
y_value = array([221900, 180000, 510000, ..., 360000, 400000, 325000])

I used this method before and y_value has the same shape as before.


texts: How to using elmo embedding for other language?

texts:
I am using the language model ELMo represent my text data as a numerical vector. This vector will be used as training data for a named entity recognition with BilSTM-CRF. My text data is not English, so I use https://github.com/HIT-SCIR/ELMoForManyLangs
I've got the vector representation, but I don't know how to use this elmo vector to combine it with the 'bilstm-crf' model. It doesn't seem the same as pretrained elmo from tensorflow. Has anyone ever tried it?


texts: Error 'Expected 2D array, got 1D array instead:'

texts:
While performing a simple fitting operation on the Titanic dataset. The following is my code:

data = pd.read_csv(r'.\Desktop\DS\Titanic\train.csv')
sex_train = data['Sex'].map({'male':0,'female':1})
survived_train = data['Survived']
sex_survivor_tree = GaussianNB()
sex_survivor_tree.fit(sex_train,survived_train)


texts: Handling too much range in texts' length

texts:
I have a classification problem where inputs are news article and target are topics. However, articles length are very varying from 100 words to 800! I'm using TfIdf and I think that it can puzzle the model.
What are techniques to deal with such imbalances?


texts:
I am working on a ML model to be deployed in a product operating in many countries.
The issue that I am having is the following: should I

train one model and serve it for all countries?
train a model per country and serve each model in its country?

I've faced this problem several times, and to me, there's a trade-off in the learning: in the first case, the model has more data to learn, and it'll be more robust (also, the solution is simpler). In the second case, I'll have a more tailored model to each country, and will be able to see effects that are specific to that country.
I'm very interested in knowing if there's an intermediate solution - a general model with some country-specific fine-tuning that can see all the data but also specialize in each specific country. If I were to use Neural Networks, this fine-tuning is natural - you train some epochs with all the data, then the last epochs with each specific country. I am wondering if something similar can be done in Linear regression models and Xgboost, which are the models I generally use.
Is there any literature on this? I think it is kind of a generic topic and there should be some.


texts: Unbalanced multi-label multi-class classification

texts:
What are common approaches in order to deal with unbalanced multi-label multi-class classification problems in deep learning? Furthermore there is correlation between the labels. I tried two approaches.

Using weights for the loss function. I used a sperate weight for each class and label. The weight was indirectly proportional to the frequency of the target class given a label.
Training $q$ classifiers for $q$ labels. There I sampled the target label with an equally distributed class distributions. I used weights as weighting between different labels. The target label got a weight higher compared to other labels.

In the beginning I thought that these two approaches should result in similar results. But Approach $2$ performed much better. It would be nice to see either it was just a random result in my experiment or there is more systematic in this kind of problems.

Is there any literature for this problem or does anybody have some experience?


texts: When training an image classifier, is it best practice to equally distribute the number of images in each category?

texts:
When training the model, I understand

If I supply too many on a certain category, it may become overfitted and treat almost all predictions as the overfitted category. This can lead to false positives in the overfitted directory.
If instead, I supply the least amount of photos to equally distribute, is the probability of of a rarer category appearing is now equal to a category that would be common to find? This can lead to false positives in the rare category but may turn my false positives in #1 to be true negatives.
If instead I skip training the rare category, I get false positives in other categories.

How do we account for the natural distribution of our target classifications (for example, types of common lesion vs rare lesions)?

Is it best practice to equally distribute? If so, what should we do if we have so little of the rare category, the sample images are orders of magnitude smaller?


texts: anomaly detection in vehicle sensor data

texts:
I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors.

It seems like there is no proper book or article that goes deeper into this topic.

Hoping to get help on where and how I could find anything regarding this topic. Maybe some of you had to deal with something like this and have acquired some knowledge on where to find some sources.

would really appreciate some help


texts: Create recommendation system to recommend products to a customer on any e-commerce website

texts:
The recommendations should be based on the products consumer has searched on other sites like Google.

This basically means, that recommendations have to be made to the user based on his/her search history. No other information is available for a user.

Has this sort of thing already been implemented? (preferably in Python)

The approach I have thought of is removing stop words and extracting keywords from a search query, based on some criteria. I am falling short at the implementation.


texts:
I am trying to solve a sparsity-promoting optimization problem. It is well known that the L1 norm is a good surrogate to the L0 norm, and it is studied in (Candes et al, 2008: Enhancing sparsity by reweighted L1 minimization https://arxiv.org/abs/0711.1612) that a better approximation can be obtained by successively reweighting the L1 norm penalization term, where the result of a step gives the new weights for the next iteration.
So far I am implementing this directly in my code but I wonder if there is any software/optimization package that does this automatically.


texts: Rephrase Neural Network : Find the Fittest Word for Given Meaning/Explanation

texts:
I am a sophomore student who's interested in deep learning and its method layering up some linear/non-linear operations and constructs up the complex function through the network.

I'd like to comprise a network that finds the word upon the given explanations :

e.g.1 input: a spherically shaped fruit which tastes sour and sweet and gets red when matured well output: apple

e.g.2, input: to make up the boundary between multiple options for the base of one's behavior or decision output: determine.

So basically it's a classification problem but the class might be more than 20,000.

I'd like to name this task "rephrase neural network".

Any good reference or starting point, or, any network recommendation for comprising this neural network?


texts:
I am experimenting with BagginClassifier, but I fail to get the expected functionality.

Basically, the BagginClassifier should draw (bootstrapping) a new data set with replacement. For example: the following code should generate a new bootstrapped sample of the same size as the original data set:

import sklearn.datasets as ds
import numpy as np

X, y = ds.load_iris().data, ds.load_iris().target

bag = BaggingClassifier(base_estimator=LogisticRegression(),
                        n_estimators=100,
                        max_samples=1.0,
                        bootstrap=True,
                        n_jobs=1)
bag.fit(X, y)

print X[bag.estimators_samples_[0]].shape
>> 95

(or any other number close to 95).

Naively, I would expect to get the bootstrapped sample of the same size as the original one (150), but with some random repetition of rows. However, I get a smaller sample size with unique rows. That's strange.

What's wrong here?


texts: How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

texts:
As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and still don't) have the hardware to fully train models from scratch.

Lately, I've been dealing with more and more images, and the need to feed data on the fly during training (fine-tuning) has become important. My current setup is essentially the legacy train.py script for training and feed_dict method for inference.

I've been reading a lot on Estimators API and their usual pipeline, but I just cannot find tutorials or help on how to use pretrained models like I have with that pipeline; all examples work on image classification and do not mix dataset creation and model training, which is a real puzzle.

*Tensorflow r1.14

Step 1: Dataset creation

So, given my data in the following format:

data = [
  [img_path_1, boxes_1],
  [img_path_2, boxes_2],
  ...,
  [img_path_N, boxes_N]
]

Where each boxes_i is in the format:

[
  [label_A, xa, ya, xb, xb],
  [label_B, xa, ya, xb, xb],
  [label_A, xa, ya, xb, xb],
  ...,
  [label_D, xa, ya, xb, xb],
]

According to some tutorial and tweaked for object detection instead of classification, it seems like I would need to process images like this (further preprocessing and batching functions left out for simplicity):

dataset = tf.data.Dataset.from_tensor_slices(data)

def load_image_and_annotations(path_and_boxes):
    path, boxes = path_and_boxes
    image_string = tf.read_file(path)

    image = tf.image.decode_jpeg(image_string, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    return (image, boxes)

dataset = dataset.map(load_image_and_annotations, num_parallel_calls=...)

and then make use of the tf.data.Iterator object to traverse the dataset as needed.

Step 2: Training

Here is mostly where I'm unsure of how to proceed. The pre-made estimators and custom estimators tutorials seem inappropriate for my task since I already have a frozen model that I wish to convert to an estimator (instead of fully building one).

How should I proceed to transition from my old setup to this new pipeline ?


texts: Establish relationship between two sets of data

texts:
I have two data sets - Product to Features and Products to Parts

A = { (P1, F1, F2, F3), (P2, F2, F4, F6), (P3, F1, F6, F8)...}
B = { (P1, M1, M2, M3), (P2, M4, M7), (P3, M1, M5, M7, M9, M10)..}

where:

P1, P2, P3... are products
F1, F2, F3... are features
M1, M2, M3... are parts used in building products.

Is it possible to come up with relationship amongst features and parts?

e.g. (F1, M1, M2), (F2, M1, M4, M6)... and so on?


texts: How to (better) discretize continuous data in decision trees?

texts:
Standard decision tree algorithms, such as ID3 and C4.5, have a brute force approach for choosing the cut point in a continuous feature. Every single value is tested as a possible cut point. (By tested I mean that e.g. the Information gain is calculated at every possible value.)

With many continuous features and a lot of data (hence many values for each feature) this apporach seems very inefficient!

I'm assuming finding a better way to do this is a hot topic in Machine Learning. In fact my Google Scholar search revealed some alternative approaches. Such as discretizing with k-means. Then there seem to be a lot of papers that tackle specific problems in specific domains.

But is there a recent review paper, blog post or book that gives an overview on common apporaches for discretization? I couldn't find one...

Or else, maybe one of you is an expert on the topic and willing to write up a small overview. That would be tremendously helpful!


texts:
Question: Can a classifier be trained with reinforcement learning without access to single classification results?

I want to train a classifier using reinforcement learning. However, there is one big restriction: the program does not have access to the score regularly, not even after every classification. Only after many classifications were completed (e.g. around 40-200 classifications, let's call them a batch) the final score of that batch is available. One batch can be executed rather quickly: it takes just around one second. Therefore, thousands of batches can be executed, each of them returning a score for its classifications. Every time a batch is executed, the current ML Model is given as input for the batch to use.

Other than that, of course, the feature vector is known (contains around 60 features) and the labels are known (around 6 labels).

I have never applied Reinforcement Learning before, therefore, I can not tell whether this can work. In theory, I think, it should: all data is available. The algorithm can choose some parameter values for the model, try them out, and get a score. Then try out different values and get the score again. This way it should be able to improve step by step.

Additional Notes: Although the text above should be enough to understand the problem and provide an answer (which can be general and not specific to a concrete use case), my personal use case and details about it are explained here. This might be useful to understand the problem in more detail.

Edit: Before, I used "Random Forest" as an example of a possible ML model that one could use. As a Random Forest seems to require supervised learning contrary to reinforcement learning, I have removed it from the text (not counting some special use cases, such as this one).


texts: How to deal with highly skewed (on counts) dependent variables?

texts:
I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in the last 3 months.

The problem is, these features are highly skewed. This is how the distribution for the above variable looks like:

0.0     98.175855
1.0      1.275902
2.0      0.348707
3.0      0.199535

This is due to the nature of the event being evaluated during the construction of the feature. For example, the majority of the population may not have defaulted hence the value is 0 for 98% of them.

There are several such variables and they are measuring important events. Therefore I cannot remove these variables. However, I am afraid the model would not be learning anything from these features as there is very less information in them.

Questions:

Am I right in assuming these features will not be useful to the model in the current state?
How can these features be handled?


texts: significance test and sample size estimation for classifiers

texts:
What is the test to tell if e.g. an F1 score of 0.69 for classifier A and 0.72 for classifier B is truly different and not just by chance? (for mean-values one would use a "t-test" and obtain a "p-value"). I have access to the underlying data and not only to the F1 scores.
... and how can one estimate the sample size needed in the test-set in order not to miss a true difference between the F1 scores? (as in the example above) (for mean-values one would use a "power analysis"). Or in other words, if I want to know which classifier (A or B) is truly better (to a certain significance level): how many test cases do I need?
Google just returns some research papers but I would need some type of established standard methods for the sample size and significance test (ideally implemented as a python package).
--- EDIT ---
Thanks for pointing out this post in the comments - it points in the right direction but unfortunately does not solve my two related problems as stated above.


texts: Writing a piecewise linear function as a sum of ReLU functions

texts:
Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions of the form:
$$  \phi_{j}(x)=\left\{\begin{array}{ll}\frac{x-x_{j-1}}{h}, & \text { if } x_{j-1} \leq x \leq x_{j} \\ \frac{x_{j+1}-x}{h}, & \text { if } x_{j} \leq x \leq x_{j+1} \\ 0, & \text { otherwise }\end{array}\right.$$

Rough illustration:

Now, suppose I am working with the ReLU activation function $\sigma(x)=max(0,x)$ and I am trying to find the coefficients $\alpha,W$ and biases $b$ such that the function $u_\theta(x) = \sum^n_{i=1} \alpha_i \sigma(w_ix+b_i)$ is exactly our function $f(x)$.

In other words, for a given piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, what are the coefficients $\alpha,W,b$ such that $\sum^n_{i=1}a_i\phi_i(x)=\sum^n_{i=1} \alpha_i \sigma(w_ix+b_i)$?


texts: Get most likely topic per document in pandas dataframe using gensim

texts:
I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call model.get_document_topics(model_corpus) to get a list of list of tuples showing the topic distribution for each document. For example, when I am working with 20 topics, I might get the following for the first three documents in my data frame:
[(5, 0.11253482), (7, 0.75876033)]
[(19, 0.96343607)]
[(0, 0.010002977),
 (1, 0.010002977),
 (2, 0.010002977),
 (3, 0.010002979),
 (4, 0.8099435),
 (5, 0.010002977),
 (6, 0.010002977),
 (7, 0.010002977),
 (8, 0.010002977),
 (9, 0.010002977),
 (10, 0.010002977),
 (11, 0.010002977),
 (12, 0.010002977),
 (13, 0.010002977),
 (14, 0.010002977),
 (15, 0.010002977),
 (16, 0.010002977),
 (17, 0.010002977),
 (18, 0.010002977),
 (19, 0.010002977)]

This means that the most likely topic for document_1 is 7, for document_2 is 19, and for document_3 is 4. The primary output that I would like to see is simply this most likely topic for each document. The way I'm doing this now is using a loop:
import numpy as np
import pandas as pd

def get_max(doc):
        idx,l = zip(*doc)
        return idx[np.argmax(l)]

data['doc_topic'] = [get_max(doc) for doc in model.get_document_topics(model_corpus)]

I have around 80k documents in my data frame, so this code takes about 45 seconds to execute. But since gensim has already done all the computations, I keep thinking that that 45 seconds of computational time is simply spent on reorganizing data, so there must be a more efficient way of doing this.
If possible, a secondary output that would be nice to have is the document-topic matrix, such that each row corresponds to a document in my data frame, and each column represents the probability (or similarity) of the document to the topic. So this would yield a DxT matrix, where D is the number of documents, and T is the number of topics.


texts:
I am looking to design a custom loss function for Keras model. The model itself is neural network that accepts a set of images and is supposed to run a regression to get an output, which is a value. Due to the physical conditions of the problem, I need to add a regularization term which would calculate the $cos(y_{pred})*f(X_i)$, where $y_{pred}$ is the output of the neural network, $X_i$ is the training example used to calculate $y_{pred}$, $f$ is some function which would calculate a value based on the image.

My problem is how to get the $X_i$ from the model? Loss function is supposed to accept just two inputs: $y_{pred}$ and $y_{true}$ which are tensors.


texts:
I read about the validations curves, and the following plot is similar to overfit, but in this case, the validation curve doesn't' growth again.
So is this overfit? why?
Thanks


texts: How to determine a shape of input and output nodes in graph. (Tensorflow C++)

texts:
At first, I tried to implement the following pipeline:

1a) I converted the model from keras (.hdf5) to tensorflow (.pb) and saved using the tf.saved_model.simple_save function in Python.
2a) I loaded the model using tensorflow::LoadSavedModel
3a) I determined a shape using the following code:

const string export_dir = "/home/user/MyBuild/build_tftest2/models";
 SavedModelBundle bundle; RunOptions run_options;
status = LoadSavedModel(sopt, run_options, export_dir,
               {kSavedModelTagServe}, &bundle);
if (status.ok())
{
    GraphDef graph = bundle.meta_graph_def.graph_def();
    auto shape = graph.node().Get(0).attr().at("shape").shape();
    for (int i = 0; i < shape.dim_size(); i++) {
        std::cout << shape.dim(i).size()<<std::endl;
    }
}

Later I was wishing to use optimal model, therefore I used another pipeline:

1b) I converted the model from keras (.hdf5) to tensorflow (.pb) using

frozen_graph = convert_variables_to_constants(session, input_graph_def,
output_names, freeze_var_names)

and saved in Python using:

tf.train.write_graph(frozen_graph, "model", "tf_model.pb", as_text=False)

2b) I optimized the model using the transform_graph utility from tensorflow package.
3b) I loaded my graph using the following code (C++):

status = ReadBinaryProto(Env::Default(), "/home/user/MySoftware/foreign code/netology_JN/Diplom/Models/optimized/optim_model.pb", &graph_def);

4b) I tried to determine a shape using 3a) and an exception was thrown:

[libprotobuf FATAL /usr/local/include/google/protobuf/map.h:1064]
CHECK failed: it != end(): key not found: shape terminate called after
throwing an instance of 'google::protobuf::FatalException' what():
CHECK failed: it != end(): key not found: shape 14:21:36: The program
has unexpectedly finished.


texts:
I have built a machine learning model to predict whether or not a team will cover the spread. In sports handicapping covering the spread means

The Point Spread: When betting on basketball, the team you bet on must
"cover the spread." This means the team must win or not lose by a
predetermined margin of points.

However I am limited to a number of features for this model. Those features are the spread (float both positive and negative), spread percentage (number of bets place on one team or another - float), spread money (percentage of money placed on one team - float), spread move (how much the line has moved from the original line - float), spread direction (whether the line moved in favor of the team or opponent - 1 or 0), spread bets (number of bets placed on one team or another - int). Out of all these features none are strongly correlation with the target variable which is COVER (represented by a 1 or 0).
Is there a better way to determine what features are relevant or do I just choose the highest correlated features out of the group?
Here is the code for the heatmap:
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns
training_data = pd.read_csv(
    "/Users/aus10/NFL/Data/Betting_Data/NBA/Training_Data_Betting_NBA.csv")
df_model = training_data.copy()
df_model = df_model.dropna()
df_model = df_model.drop(["Money_Line", 'Money_Line_Percentage', 'Money_Line_Money', 'Money_Line_Move', 'Money_Line_Direction', 'Money_Line_Bets', 'Team_Total',
                          'Total_Move', 'Total', 'Total_Direction', 'Total_Over_Bets', 'Total_Under_Bets', 'Over_Percentage', 'Under_Percentage', 'Under_Money', 'Win', 'Over_Under', 'Over_Money'], axis=1)
corrmat = df_model.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(20, 20))
g = sns.heatmap(df_model[top_corr_features].corr(),
                annot=True, cmap='coolwarm')
plt.yticks(rotation=0)
plt.xticks(rotation=90)
plt.show()

here is a link the spreadsheet
https://docs.google.com/spreadsheets/d/1yZJ-HNHpYh9pJmYoyJxVQ1W4CyIcazN7ues79Os_UIw/edit?usp=sharing


texts:
Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in the row-oriented approach, parquet is more efficient in terms of storage and performance.

What is the advantage of a parquet file?
Are parquet files compressed?


texts:
when training a VAE, typically one samples from the latent distribution using the reparametrization trick using a fairly large minibatch size (>100) in the decoder/generator half of the VAE. I'm assuming this minibatch size allows the network to 'smooth' out the error and allows us to avoid having to repeatedly sample from the latent space.

However, I'm interested in online scenarios where you are training the VAE on streaming data as it arrives, so the batch size would be 1. In this case, it can take the VAE a long time to converge because the error is highly volatile.

Is there any way to avoid this issue in practice? I am unsure what will happen if I have to repeatedly sample from the latent distribution and then take the mean of those samples (or something) - aside from obvious performance concerns. The other alternative is to wait for enough samples to arrive that I can train them in a larger batch, but even in this case I wouldn't be able to wait for 100+ samples to arrive.


texts:
I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim.

What I have managed for now is to divide every text into sentence, put into one flat array and give every sentence an i (its order, basically) tag.

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(texts_flat)]

For the reason of possible improvement I want to tag every sentence not only with its order but with the name/order of text it is from. For that, I have made a list of lists, where every text is a list and every text contain list of sentences. i.e

texts = [text1 = [sent1, sent2, ...], text2, text3 ...]

How to iterate over this kind of document?
I came up with smth like

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i) + '.'+ str(j)]) for i, j, _d in enumerate(texts)]

but i get ValueError

  ValueError: not enough values to unpack (expected 3, got 2)

Is it even going to do anything?


texts: Random forest model gives same result for all test data, Next step?

texts:
I am teaching myself some data science and have started a Kaggle project.
I have fitted a random forest classification model (using sci-kit learn) with a few millions rows of data. There are two possible outcomes for each row (0 or 1). When I run it against the test data, I get 0 for every row. This is a practical impossibility, but I am at a loss as to how to diagnose my model and how to move forward.
Is this an example of extreme overfitting ? Is it more likely a problem with my training data (or test data) that i am not seeing ?

Should I simply increase the # of estimators? Is it possible that I have misformatted the input files in a subtle way that doesn't cause an error ? I am at a loss as to how to move forward.


texts: clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure

texts:
I've got a set of approx. 50 000 2-dimensional euclidean vectors which are connected with 20 groups, i.e. each group has approx. 2500 2-dimensional euclidean vectors. My data includes endpoints coordinates, i.e. $x_0, y_0, x_1, y_1$. Now I would like to cluster the vectors within these groups, probably using k-means/k-medoids clutering (or other clustering algorithm with pre-defined no. of clusters). What is also important, my main focus is on vector's direction, length is the minor concern (but at best, still should be taken into conideration). What I'm struggling with is a choice of dissimilarity measure that would be suited to my problem. So here are my question:

Does it matter how the data is specified? Alternatively, I could calculate an angle and length of vector and specify the data as $x_0, y_0, angle, length$. My intuition is that if angle is explicitely present, the euclidean distance should do a better job capturing the vector's direction. What is more I could maybe use some weighting, modify a euclidean distance and calculate distance between two observations as for example:

$\sqrt{(x^1_0 - x^2_0)^2 + (y^1_0 - y^2_0)^2 + (angle^1-angle^2)^2 + \frac{1}{n}(length^1-length^2)^2}$

where $n$ is some constant.

I also considered angular distance as a dissimilarity measure. From what I know this is equivalent to clustering the standarised data points and therefore doesn't capture size (lengths in my case). But I'm not sure if k-means clustering can be done with cosine distance. If so, is there any package in R that allows that?
Is is a good and statistically valid idea to perform clustering twice: firstly, to cluster starting points and secondly, within those clusters perform clustering for angles and lengths?
Do you guys know any papers regarding similar problem, i.e. clustering the 2-dimensional data points? Any example would be very handy.


texts: Should I report loss on the last batch or the entire training dataset?

texts:
In the training loop below, you can see that train_loss represents the loss on the most recent batch.
Whereas the eval_loss is computed on the entire training dataset.
When I print it out, should I be recalculating the train_loss against the entire training split instead of the last batch?
for epoch in range(epochs):
    # Loop over batches.
    for i, batch in enumerate(samples_train['features']):

        train_preds = model(samples_train['features'][i])
        train_loss = loss_fn(train_preds, samples_train['labels'][i])

        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()

    # Only the training data is batched.
    eval_preds = model(samples_evaluate['features'])
    eval_loss = loss_fn(eval_preds, samples_evaluate['labels'])

    print(f"Epoch: {epoch}, Train_Loss: {float(train_loss)},  Eval_Loss: {float(eval_loss)}")


texts: I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, which fold should I use? Is averaging all 5 of the folds the best choice? Secondly, why is the data-set split ratio (70/30) different in 5 fold Cross validation and 10 fold cross validation? Shouldn't it be the same?


texts:
I'm using KMeans of scikit-learn for clustering of forums.

The attribute inertia_ of model gives the within class inertia but how can i get the between class inertia? In other word how can I verify the quality of my clustering?

Thanks


texts:
In this example on tensorflow website, I was surprised to see a neural network's predicted values graph as a straight line. Isn't the purpose of neural networks to approximate non-linearities of data into it?
Please let me know what I am missing here.


texts: I analyzed my time series using Breusch Pagan test and observed the presence of heteroscedasticity in it. After box-cox transformation, I again tested the time series using Breusch Pagan test. The time series still shows heteroscedasticity. However, I do not know how to test if the heteroscedasticity has been rectified even partially or not. Can anyone please guide on how to measure the degree of heteroscedasticity because the test I am using in R, gives Pvalue and I am only able to figure out if heteroscedasticity is present or not.


texts: Naive Bayes / SVM classifiation - min. number of records (Python)

texts:
I am doing text classification with Python. I have around 120 records with 2 columns:

text
class

I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the alghoritms using sklearn and divide it to training and test sets each time I run the script the Accuracy Score is so different each time for both alghoritms. Sometimes I get around 70%, sometimes 40%. Is it because of the number of records (120) or not necessarily? if it is about number of records how much of them should I have?


texts: How can I plot data after cluster it into two cluster?

texts:
I have some unlabeled data with 15 attributes and 30161 instances. I took 70% data for clustering and cluster those data into 2 cluster. I also added one column after clustering named 'cluster' and the column '<=50k' was label column. Now I need to plot those data except label column, How can I plot it ?
Here's the sample of my code and data.

Data

cols_x=train.columns[0:14]
cols_y = train.columns[14]
X = train[cols_x]
y = train[cols_y]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

kmeans = KMeans(n_clusters=2)
train["cluster"] = kmeans.fit_predict(train[X_train.columns[0:14]])
train.head(10)


texts: Predict a tree structure out of nodes with different features

texts:
The Problem

Suppose we have a representation of a document's text layout as in the image below:

Here, each rectangle represents a chunk of text (one word, an expression or even part of a sentence).

Each rectangle has a set of features and connections to adjacent rectangles (the image doesn't show all connections, but they are present):

top/bot/lft/rgh sibling
top/bot/lft/rgh distance to sibling (in case of sibling presence)
margins alignment (for rectangles aligned in columns)
font size (continuous variable type)
font type (categorical)
all_caps (binary)
and some other

What I would like to achieve is to train an algorithm (ML? DNN? Graphs? something other?) that would structure these rectangles, would create a tree structure out of these individual nodes. For instance the above example would be structured as follows:

          ROOT
 __________|___
|          |   |
2          10  12
|_         |   |_________
| |        |   |         |
3 9        11  14        20...
|________      |__       |__
| | | | |      |  |      |  |
4 5 6 7 8      13 15...  19 21...
                            |  \
                            22 23...

My results so far

I spend some time (~8h already) searching for a solution, but couldn't find anything applicable to this case. I'm relatively new to ML and don't have much experience, however I think it can be solved using a supervised learning algorithm by feeding a training set consisting of documents with all these nodes with their features and annotated trees that should be generated as a result out of these nodes.

Some ideas I found while searching that might help (probably one of these is the solution, but I need some guidance on how to adjust in properly):

Using some sort of DNN. It would figure out the relationship between distances and other features of the nodes from the training set so that it would be able to predict a structure given only the nodes and their features. I didn't work with DNN and I don't know how to make it generate the structure instead of categorize which digit is shown in the image (I did some basic tutorials)
Creating/Training a PCFG (Probabilistic Context Free Grammar). This approach seems to be valid, as it used in generating parsed trees in NLP (those that structure a sentence in form of a tree) and I found some resources on it, however I don't understand how to use all the features that are available on my nodes in learning such a grammar? Also, I don't see how this approach could make use of the positioning data in 2D space and the distances between siblings, as it is one of the key features in structuring the nodes.
Probably some Graph or Tree related algorithms would solve my problem, but I don't know how to formulate the query that would help me find the solution.

A very similar question was asked on Computer Science SE site, but there are no answers and the OP doesn't respond.

I would really appreciate if you would provide a solution, or even a hint in what direction to move in order to solve this problem.


texts: I'm trying to compare two NER tools on an annotated corpus and I'm not sure which is the best metric to use, as I haven't worked with NER models before. To be more specific, I'm interested in one class only, so I want to evaluate them on that particular class.


texts: Error opening Iris.tab file in Orange load data widget

texts:
I am attempting to load in the Iris.tab data set into the Orange load data widget. However, when I do this, I get the following error message (shown above).
This keeps popping up when I start up Orange Data Mining and try to load the iris.tab data file. It used to work before, but now it is showing this error.
Does anyone have any idea what why this might be happening and what might be causing this error?
Thanks!


texts: From my experiences the R Decision tree returns more accurate results than the python decision tree. Can anymore confirm this assumption and maybe knows the reason?


texts: How to handle addresses of the restaurants to feed the data-set in the ML model?

texts:
I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the crucial data which I need to predict the food delivery timing, now my problem is the address is in string format so how do I handle this address to feed the data in the ML model. I have a total of 35 unique addresses if I do one-hot encoding my dataset will be very large and it will take too much time to train, is there anyway except one-hot encoding to handle the addresses of restaurants.

Data Sample-


texts:
If we treated NER as a classification/prediction problem, how would we handle name entities that weren't in training corpus?

For example, "James was born in England." James was labeled as a PERSON and England as a LOCATION. But we type another completely strange sentence like "Fyonair is from Fuabalada land." We as human can understand Fyonair is a person (or maybe princess from a fairy tale) and Fuabalada is the land where she comes from.

How would our model identify it if it wasn't included in billions of corpus and tokens? Can unsupervised learning achieve this task?


texts: Ignoring unlabeled data for a single class

texts:
I have a data set of transactions with a binary flag labeling each as fraud or not fraud. However, it can take up to 90 days for a transaction to reveal itself as fraudulent. Sometimes it happens in a few days.

I can safely use any data older than 90 days as we are 100% confident the labels are accurate. However, for the 90 day window, anything that has been classified as fruad we are 100% certain is right, but anything labeled not fraud we cannot say with confidence as not enough time has passed for fraud to come through the payment system.

What are the dangers of adding any of the known fraud from that 90-day window to the older known and completely labeled data set for training a classifier, while ignoring any "unknown" transactions within that 90-day window? Obviously, this seems to be a great task for semi-supervised learning, just wondering what big errors and bias this would cause.

Fraud of Txs Over Time:

0000100001100000100000001010010000 || 90 days ago || 1????11??1?1????1
Which turns into:
0000100001100000100000001010010000 || 90 days ago || 111111


texts:
When i run my code with 5 epochs, code gets stuck at first epoch and run continuesly. I tried applying various parameters but couldn't make it.

here is my code...

import tensorflow as tf
import numpy as np
import os
from google.colab import drive
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow import keras
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D

training_path = 'drive/My Drive/tesla'
validation_path = 'drive/My Drive/validation'

training_generator = ImageDataGenerator(rescale=1.0/255)
validation_generator = ImageDataGenerator(rescale=1.0/255)

training_set = training_generator.flow_from_directory(batch_size=2,
                                                      directory=training_path,
                                                      target_size=(150,150),
                                                      class_mode='binary',
                                                      shuffle=True)

validation_set = validation_generator.flow_from_directory(batch_size=2,
                                                          directory=validation_path,
                                                          target_size=(150,150),

                                                          class_mode='binary')

model = keras.models.Sequential([
                                 Conv2D(8, 3, padding='same', activation='relu', input_shape=(150,150,3)),
                                 MaxPooling2D(),
                                 Conv2D(16, 3, padding='same', activation='relu'),
                                 MaxPooling2D(),
                                 Conv2D(16, 3, padding='same', activation='relu'),
                                 MaxPooling2D(),
                                 Conv2D(4, 3, activation='relu'),
                                 Flatten(),
                                 Dense(4, activation='relu'),
                                 Dense(1)

])

model.compile(optimizer='adam',
              loss=tf.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.fit(training_set,
          validation_data=validation_set,
          epochs=5,
          verbose=1,
          validation_steps=20/2)

but when i run my code, it shows this and run non stop at first epoch.

Epoch 1/5
  14781/Unknown - 630s 43ms/step - loss: 4.0421e-06 - accuracy: 1.0000

Although my training sample size is only 40 (20 cats and 20 dogs) and validation sample size is 20 (10 cats and 10 dogs). I am coding in google colab.

Thanks in advance.


texts: what is the difference between Bias and overfilling and under fitting?

texts: I am confused about Bias and overfitting as I see it sounds similar but not able to explain differene?


texts: preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

texts:
I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) which are quite similar in trend and value ranges. The dataset contains 10 variables, 1 to be predicted (green one below) and 9 independent ones (only 3 shown here).

The independent variables are already chosen by a domain guy(I’m the domain guy), after careful consideration from over 160 variables.  I have tried LSTM when treating data as time series and boosted regressions when treating the data as cross-sectional. After months of work I’ve realized that I don’t have realization on how to look at the data from a machine learning point of view. The problem now is, what is the definition of my data? In my head the conceptual title of my dataset is multiple, multivariate, multi-trend, timeseries data - using which I’m trying to predict a variable cross-sectionally. There are questions and tutorials all over about dealing with each of the above individually; but how should I simplify this problem?

I stacked together all the data from all the systems (one on top of the other). The results of both models can capture the trend very well, but there are issues in scaling as seen below (red is predicted): The scaling difference can vary in magnitude and direction (+ve/ -ve). I normalized the stacked dataset globally; and also tried normalizing each system dataset individually and then stacking, the issue stayed.

Then I tried to split one-system data 70:30, train a model on 70% of one system data and test on remaining 30% using LSTM. The results were very good. I concluded that the scaling issue is being caused by stacking the data together. But if can’t stack the data together, how else do I use all that data?
Now I’m trying to decrypt the multi-trend in time (because I think that is also causing a scaling issue). The trend is supposed to be handled with the help of two categorical variables in the dataset. But now I’m thinking putting these separate trends in time to separate neurons in the network, consequently building separate activation functions for separate trends. To summarize:

What category of ML or time-series theory does my problem fall in?
Should I try convert non-stationary parameters to stationary before feeding it to LSTM?
Should the NN have separate neurons to treat each trend differently, even though there are categorical variables for that (there may be an issue of oversampling with one or two categories)
I’ve tried clustering based on averaged parameters and building separate models for each cluster, the issue was still there. Should I try a different time-series clustering approach like Euclidian distance or iSAX?


texts: Shouldn't an autoencoder with #(neurons in hidden layer) = #(neurons in input layer) be "perfect"?

texts:
I'm exploring autoencoders for the first time. I'm using the Matlab neural networks toolbox. I have created a synthetic dataset consisting of points in 2D space plus some noise. My idea was to visualize the features of the hidden layer of the autoencoder to get the feeling about how it works. After training the autoencoder, I apply it to the same training data set to check the output.

If I consider one neuron in the hidden layer, then I get the output data all clustered on a straight line. Which makes sense to me, as with one degree of freedom (plus bias), there is not much else the autoencoder can do other than to find the line that minimizes the distance to the data points.

However, if I have two neurons in the hidden layer, I would expect the autoencoder to be perfect, since we have two degrees of freedom to match all the points. Hence, with enough epochs, it should be able to reduce the error to almost zero, and reproduce the input data perfectly. To my surprise, this doesn't happen: the training quickly stalls and we see some imperfections when trying to reproduce the input data.

So where's the catch ? Shouldn't an autoencoder be able to trivially replicate the input to the output when it doesn't have to compress the input information ?

PS: in the picture below, "# hidden layers" actually means "# of neurons in the hidden layers"


texts: How to feed a table per timestamp to LSTM neural network?

texts:
I have a time-series dataframe like this

         feat1  feat2  target
date id
0    1      12     16     192
     2      15      6      90
     3       2      9      18
1    1       0      3       0
     2       0      9       0
     3      56      9     504
2    1       5      9      45
     2       6      9      54
     3       5      8      40

and my problem is for regression.

What I know about the LSTM sequences, is that normally, the row_id is the date, so you build sequences of n rows.

In my data, as you can see, it's different. In each row that refers to the date, I have 3 more rows representing a product.

What I have thinked for make the sequence to my problem,  my sequence will be of 2 days:

sequence = [day0,day1],[day1,day2]

For every date, I have

date0 = [id1,id2,id3].

For every  id, I have:

id = [feat1,feat2].

the sequence[0], and it would be something like

[
    [
        [12,16],
        [15,6],
        [2,9]
    ],[
        [0,3],
        [0.9],
        [56,9]
    ]
]

Is this valid?

Will the LSTM layer understand this? Or do I have to do some kind of extra transformation?


texts: Tuning a classifier for high precision, with no regard for recall

texts:
I understand this falls under the decision making aspect, rather than the probabilistic, but for the purposes of some work I am doing, I need the classifier to have very high precision, as I can't afford a false positive. I do not care about false negatives, and consequently, do not care about recall. Since it is currently a binary classifier, some might say to play with the decision probability threshold from its current 0.5 value, but I will eventually need to add a third class, and will therefore need to switch to 3 outputs with softmax. I am unaware of traditional methods for shifting my pipeline towards a high precision outcome, and am looking for ways to achieve this.

If it is any help, the problem is classification of 256x256 grayscale images in a domain that is very difficult to classify, according to current whitepapers in the computer vision research area.


texts: What are the ways to perform constrained optimization of weights in deep model? I am working on deep similarity learning, and many methods are based on constrained optimization; the vast majority of deep learning methods, however, only rely on unconstrained optimization. Are there any was to implement constraints in SGD?


texts:
We need to implement a time series problem with the LSTM model.

But, while implementing the same, the main challenge I am facing is the feature selection issue. Because our data-set contains 2300 observations and 600 features. And we already know there are so many features which are not at all relevant. But, no one has the domain expertise to confirm which feature is relevant.

I have tried with…

All 600 features, [shape of X become, 2280 X 20 X 600 (if time step is 20)]. The accuracy is very poor 53-55%.
Considered a single column as a predictor (the feature which we want to predict) [shape of X become, 2280 X 20 X 1 (if the time step is 20)]. The accuracy is also poor.
Also, tried with PCA so that all 600 features are reduced to 20 features [approx 98% data was retained], but not seen any improvement.
Finally, I have gone through a very lengthy process.

a. Out of 600 features, each of them is tried out individually with the LSTM algorithm. The feature which was giving the best result is selected.

b. The selected feature is paired with the rest of 599 features individually. Then, I select the best pair that gives the best output.

c. The best pair is further tried with rest 598 features individually and this process continues...

After doing so, as of now, we get 6 features, the accuracy gets improved to 65%.

My question is, is it the only approach that I can do feature selection in time series problems?

In case of any non-time series problem there are so many ways to do the feature selection. What will be the best way to do feature selection in the time series issue?

Let's give another example...

Let's assume we developed the LSTM model with 10 predictors/features. The model is working very fine and we get 80% accuracy.

Now, somehow 100 more features get added which are not relevant for deriving target y.

So, the accuracy will be dropped as the model will focus on 120 predictors/features, where 100 are not relevant.

Then, how will we eliminate the 100 unwanted features?


texts: Algorithm to create graphics from scribble

texts:
I want to implement a system that takes as input scribbles (i.e. drawings with writings on paper, input to the algorithm is a photo of this) and outputs the forms / writings e.g. in Powerpoint.

What it the best algorithm to detect forms (e.g. this is a circle, this is a box, this is handwritten text and it writes xyz)? There will be multiple forms and writings within a single input scribble / photo.

Thanks for any hints!

Update: Here is an example of a typical scribble. In this case conversion target would be powerpoint


texts: Is the first Linear layer in Pytorch considered as the input layer?

texts:
For example:
nn.Linear(100,50)
You have 100 input features and 50 outputs. Would this be considered as the input layer or already as a kind of second layer?
The problem: I don't want to apply dropout to the input layer so I am not sure how should I look at this.


texts:
I have an mxnet model for facerecognition that needs to be loaded in keras?
Any idea how to load the JSON and params file in keras?

Ref. Insightface model for facerecognition.


texts: analogy for pearson r statistics for binary classification task

texts:
I am trying to get idea how variables of my data correspond to target variable (binary class).

In regression, Pearson r statistic is quite good to get sense of variable relationship. Also I can use it for classification, treating classes 0 and 1 as real values but it's a risky trick.

My question: is there any equivalent statistic saying which variable is good for classification task ? Thanks


texts:
Are there any machine learning libraries for Ruby that are relatively complete (including a wide variety of algorithms for supervised and unsupervised learning), robustly tested, and well-documented? I love Python's scikit-learn for its incredible documentation, but a client would prefer to write the code in Ruby since that's what they're familiar with.

Ideally I am looking for a library or set of libraries which, like scikit and numpy, can implement a wide variety of data structures like sparse matrices, as well as learners.

Some examples of things we'll need to do are binary classification using SVMs, and  implementing bag of words models which we hope to concatenate with arbitrary numeric data, as described in this StackOverflow post.


texts: How to make a dataframe with lists or vectors as its elements

texts:
This is something I have been wondering for ages but I am never able to get an answer.

I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector or a matrix.

For example, lets say we have a regular vector V with elements being real numbers.

Then to acess any number we would have

V[3] which would give the third element of said vector.

Now I want to know how to do this with say a dataframe D,

where each element of the dataframe is itself a vector or matrix, so that say

D[3] is not a real number, but a vector.

How can this be done in R?

Thanks all


texts: Activity recognition with binary sensors

texts:
I have a bunch of streams coming from a set of 28 binary sensors around a SmartHome like this:

Where:

OBJECT: indicates the name of a binary sensor
STATE: is obviously the state of the sensor at that precise instant (you can think of it like Movement/Pressure/Open=1 and NoMovement/NoPressure/Close=0)
ACTIVITY: is the Label I would like to predict and indicates the activity the Human was doing
TIMESTAMP/HOUR: you know what they mean...

The activities to be recognised are 24 and are a bit complex. For example: making breakfast, making lunch, making dinner, having breakfast, having lunch, having dinner, going to the toilet, teeth brushing, dressing, watch TV, and so on...

Can you suggest a technique to use this dataset for AR?


texts: What is the bleu score of professional human translators?

texts:
Machine translation models are usually evaluated using bleu score. I want to get some intuition for this score. What is the bleu score of professional human translator?

I know it depends on the languages, the translator ect. I just want to get the scale.

edit: I want to make it clear - I talk about the expected bleu. It's not a theoretical question, it is an experimental one.


texts: How to implement global contrast normalization in python?

texts:
I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast normalization we use this equation:
$$\mathsf{X}^{\prime}_{i,j,k}=s\frac{\mathsf{X}_{i,j,k}-\overline{\mathsf{X}}}{max\left\lbrace \epsilon, \sqrt{\lambda+\frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3}(\mathsf{X}_{i,j,k}-\overline{\mathsf{X}})^2}\right\rbrace }$$ where $\mathsf{X}_{i,j,k}$ is a tensor of the image and $\mathsf{X}^{\prime}_{i,j,k}$ is a tensor of the normalized image, and $\overline{\mathsf{X}} = \frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3} \mathsf{X}_{i,j,k}$ is the average value of the pixels of the original image. $\epsilon$ and $\lambda$ are some constants, usually with $\lambda=10$ and $\epsilon$ set to be a very small number, and here is my implementation:

import Image
import numpy as np
import math
def global_contrast_normalization(filename, s, lmda, epsilon):
    X = np.array(Image.open(filename))

    X_prime=X
    r,c,u=X.shape
    contrast =0
    su=0
    sum_x=0

    for i in range(r):
        for j in range(c):
            for k in range(u):

                sum_x=sum_x+X[i][j][k]
    X_average=float(sum_x)/(r*c*u)

    for i in range(r):
        for j in range(c):
            for k in range(u):

                su=su+((X[i][j][k])-X_average)**2
    contrast=np.sqrt(lmda+(float(su)/(r*c*u)))

    for i in range(r):
        for j in range(c):
            for k in range(u):

                X_prime[i][j][k] = s * (X[i][j][k] - X_average) / max(epsilon, contrast)
    Image.fromarray(X_prime).save("result.jpg")
global_contrast_normalization("cat.jpg", 1, 10, 0.000000001)

original image:

result image:

I got an unexpected result. What is wrong with my implementation?


texts: Difference between sklearn’s “log_loss” and “LogisticRegression”?

texts:
I am a newbie currently learning data science from scratch and I have a rather stupid question to ask. I’m currently learning about binary classification, and I understand that the logistic function is a useful tool for this. I looked up the documentation and noticed that there are two logistic related functions I can import, i.e. sklearn.metric.log_loss and sklearn.linear_model.LogisticRegression. When and where should I use them, and what’s the difference?

On a broader note, what’s the difference between a metric and a model, and why is the log loss function a metric? Apologies if this question sounds completely nonsensical, but this is a genuine source of confusion for me!


texts: Performance Metric Over Expected Value Functions that are Changing

texts:
I am trying to find a way to calculate a profitability metric for a model of mine.

I currently have data labels corresponding to the value earned during each certain move made in my training set. My goal is to analyze how close my model is to these values by comparing these to the models expected value predictions for each possible move to be made. The problem here is that the value will almost never equal the exact Expected Value, but I want to get some sort of evaluation metric of how close these predicted EV's are to the labels on average.

I know that with Expected Values, given the same formula v * p where v (values) and p (probabilities) are vectors of size n, over a large number of simulations the average of these values will converge to the expected value. But for my problem each EV formula is different for every training point which poses problems.

The result I came up with includes doing the same exact thing over a large number of iterations and take the average of the differences between Actual Value - Expected Value. My goal was to evaluate a perfect model which has the correct expected value equation and use this as the max limit for what my model can achieve. I just need some help interpreting my result, or hopefully finding a better way where I can get a value that can be directly represented as an expected value of profitability.

The code below is my current solution. When num_options is very large the np.mean(differences) converges to 0. But for small num_options like 3 (which represents my problem) it always converges to a certain number. I need help figuring out what this number is so I can interpret the result. Does anyone have any idea? I'm assuming it has to due with the spread of data within the values vector, because if you have more num_options you are more likely to pick an EV thats closer the Value.

Code:

import numpy as np
from bisect import bisect_right
import random

enumerations = 100000
num_options = 2

#Initialize Random probability vectors and random value vectors of size num_options
probability_vectors = [np.random.random(num_options) for _ in range(enumerations)]
value_vectors = [np.random.random(num_options) for _ in range(enumerations)]

differences = []

for i in range(enumerations):

    #Normalize for values and probabilities
    probabilities = [j / sum(probability_vectors[i]) for j in probability_vectors[i]]
    values = [j / sum(value_vectors[i]) for j in value_vectors[i]]

    #Calculate EV
    EV = np.dot(values,probabilities)

    #Choose random value
    running_sum = np.cumsum(probabilities)
    random_number = np.random.random(1)[0]
    chosen_outcome = bisect_right(running_sum,random_number)
    value = values[chosen_outcome]

    differences.append(abs(value - EV))

print(np.mean(differences))
print(np.std(differences))

With large enumerations:

num_options = 2 : 0.1490140542519784
num_options = 3 : 0.1275503821941484
num_options = 4 : 0.1045104868184497
num_options = 5 : 0.08740796919164531
num_options = 6 : 0.07480764714073941
num_options = 7 : 0.06507271751946012
num_options = 8 : 0.05784730185234172
num_options = 9 : 0.051894809981252676
num_options = 10 : 0.04697499914796892
num_options = 11 : 0.04300556153302974
num_options = 12 : 0.039613242732981065
num_options = 13 : 0.036683873410078144
num_options = 14 : 0.034154073132997134
num_options = 15 : 0.0320761992188373
num_options = 16 : 0.030082494700577653
num_options = 17 : 0.028371757269382177
num_options = 18 : 0.02695502418214529
num_options = 19 : 0.025488676908302983

....
num_options = +infinity = 0

So Questions:

What is this offset from 0?
Is there a better approach to calculate a long term profitability metric over Expected Value equations that keep changing?


texts: How to make smaller categories with factor character variables

texts:
I have this data set with consist of ISO3166 Alpha-2 codes for countries.
Example: DE, AD, AE etc
They are coded as factor variables in R and there are about 173 observations.

Now because there are too many and this would just overwhelm a boxplot, I want to make a contingency table with other variables by condensing the codes and create shorter categories (also coded as factors) with the codes, for example, having

DE, RE, ED, FR->  Europe

CA, US->          North America

VF, HG, HY, TY->  South America

HG, TY, UT,FT ->    Africa

How can I do that because I tried a few things that did not work.

Thank you!

P.S  These codes are made up codes but it is just to illustrate.


texts: Decision Tree Bayes rules / Maximax / Maximin

texts:
I need to draw a decision tree about this subject :

  The research and development manager in an old oil company, which is
  considering making some changes, lists the following courses of action
  for the company:

  (i) Adopt a process developed by another oil company. This would cost
  €7 million in royalties and yield a net €20 million profit (before
  paying the royalty).

  (ii) Carry out one or two (not simultaneously) alternative research
  projects :

  (R1) the more expensive one has a 0.8 chance of success; net profit
  €16 million and a further €6 million in royalties. If it fails there
  will be a net loss of €10 million.

  (R2) the alternative research programme is less expensive but only has
  a 0.7 chance of success with a net profit of €15 million and a further
  €5 million in royalties. If it fails a net loss of €6 million will be
  incurred.

  (iii) Make no changes. After meeting current operating costs the
  company expects to make a net €15 million profit from its existing
  process. Failure of one research program would still leave open all
  remaining courses of action (including the other research programme).

I need also to indicate the different payoffs. This is what I've done so far :

I would like to be sure that I'm going in the right direction since I'm a beginner with decision trees. And then I need to decide the best course of action using Bayes, Maximax and Maximin rules.


texts: How to get Euclidean Distance between feature maps

texts:
I'm trying to find "keyframes" within a video using this paper however I'm a little new to machine learning and I'm stuck on the distance vector step. The goal is to calculate the distance vector between consecutive frames to detect a big context change indicating a keyframe. here's what I've done:
Using Googlenet pre-trained model I extracted a (1024,7,7) feature map, so a vector of 7x7 matrices for every other frame in the video.
Now I want to calculate the euclidean distance between two consecutive frames but I'm not sure how that would be.
My intuition is that the distance vector would also be (1024,7,7).
The last step would be to apply a "convolution of 4-window of last distance values with vector [0.1, 0.1, 0.1, 0.99]" on the distance vector?? I don't understand this step either.
Any help or guidance would be appreciated!


texts: How to find the relationship between categorical variables?

texts:
I have a dataset where I have the Outcome of a case and the Country under review, as well as whether a judge (among a panel of judges) reviewing the case was also from the country under review, Country_Judge, categorised as TRUE or FALSE.

How can I measure the relationship between the Country_Judge and the Outcome of a case? I want to know whether a judge's nationality has an impact on the outcome of the cases.


texts: Autoencoder train and test accuracy shooting to 99% on few epochs

texts:
I am trying to train an autoencoder for dimensionality reduction and hopefully for anomaly detection. My data specifications are as follows.

Unlabeled
1 million data points
9 features

I am trying to reduce it to 2 compressed features so I can have better visualization for clustering.
My autoencoder is as follows where latent_dim = 2 and input_dim = 9
class Autoencoder(tf.keras.Model):
  def __init__(self,latent_dim,input_dim):
        super(Autoencoder32x, self).__init__()
        self.latent_dim = latent_dim
        self.input_dim = input_dim
        self.dropout_factor = 0.5
        self.encoder = Sequential([
                                  # Dense(16, activation='relu', input_shape=(self.input_dim,)),
                                   #Dropout(self.dropout_factor),
                                   Dense(8, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   Dense(4, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   Dense(self.latent_dim, activation='relu')
                                   ])
        self.decoder = Sequential([
                                   Dense(4, activation='relu', input_shape=(self.latent_dim,)),
                                   Dropout(self.dropout_factor),
                                   Dense(8, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   #Dense(16, activation='relu'),
                                   #Dropout(self.dropout_factor),
                                   Dense(self.input_dim, activation=None)
                                   ])

  def call(self, inputs):
    encoder_out = self.encoder(inputs)
    return self.decoder(encoder_out)

Model compilation
ae_train_x, ae_test_x, ae_train_y, ae_test_y = train_test_split(scaled_df[COLUNMS_FOR_AUTOENCODER], scaled_df[COLUNMS_FOR_AUTOENCODER], test_size=0.33)
autoencoder = Autoencoder(latent_dim=2,input_dim=9)
autoencoder.compile(loss='mse', optimizer='adam',metrics=['accuracy'])

Finally training
ae_history = autoencoder_10_32x.fit(ae_train_x, ae_train_y, validation_data=(ae_test_x, ae_test_y), epochs=50)

Output of training
Epoch 1/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.3330 - accuracy: 0.9646 - val_loss: 0.2816 - val_accuracy: 0.9999
Epoch 2/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.2664 - accuracy: 0.9999 - val_loss: 0.2818 - val_accuracy: 0.9999
Epoch 3/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.2649 - accuracy: 0.9999 - val_loss: 0.2845 - val_accuracy: 0.9999

What could be the problem? I think the network is learning to just pass the values. But that should not be possible with the bottleneck and dropout layers. I have also decreased layers but still the result is same. How can I fix it?


texts: Combining image and scalar inputs into a neural network

texts:
I'm looking at the best way of combining CNN with image input and a scalar value.

I know that one of the ways is to concatenate flatten layer with this scalar value. But flatten layer consist for example 2048 such scalar values with different magnitude than a single input value. And what if in a real task this scalar value has more influence than image.
Also one of the examples is a combination of a text and image and then some fusion on top of that, but I still think it is a little different task because you get pretty the same vectors from the text model and CNN network.
Another one solution is to apply some ml algorithms, like Xgboost on top of flatten layer from CNN and this scalar value. But in that case, we need to train CNN networks separately, which is not good.

Can someone tell what is the best way to combine image input with scalar value so that I can train CNN network together with scalar input and that network will "decide" which input more important?


texts: Seaborn Violin plot error No loop for unfunc add

texts:
I'm trying to plot a violin plot with a split based on Sex ( like in the fourth example in the doccumentation but with Sex)

I can produce a categorical scatter plot and split it by Sex. However, when i attempt the same but as a violin plot; it throws an error.

Traceback (most recent call last):

  File "<ipython-input-868-0599b976fd6c>", line 1, in <module>
    sns.catplot(x="Batch", y="Age", hue = 'Sex', data = ages,  kind='violin')

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 3755, in catplot
    g.map_dataframe(plot_func, x, y, hue, **plot_kws)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py", line 820, in map_dataframe
    self._facet_plot(func, ax, args, kwargs)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py", line 838, in _facet_plot
    func(*plot_args, **plot_kwargs)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 2387, in violinplot
    color, palette, saturation)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 564, in __init__
    self.estimate_densities(bw, cut, scale, scale_hue, gridsize)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 679, in estimate_densities
    kde, bw_used = self.fit_kde(kde_data, bw)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 719, in fit_kde
    kde = stats.gaussian_kde(x)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 208, in __init__
    self.set_bandwidth(bw_method=bw_method)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 540, in set_bandwidth
    self._compute_covariance()

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 551, in _compute_covariance
    aweights=self.weights))

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 2427, in cov
    avg, w_sum = average(X, axis=1, weights=w, returned=True)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 419, in average
    scl = wgt.sum(axis=axis, dtype=result_dtype)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py", line 36, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial)

TypeError: No loop matching the specified signature and casting
was found for ufunc add

My code is:

>>> print(ages.head())
            Age     Sex  Batch
PassengerId
852          74    male  Train
86           33  female  Train
161          44    male  Train
812          39    male  Train
837          21    male  Train

>>> sns.catplot(x="Batch", y="Age", hue = 'Sex', data = ages,  kind='violin')

Removing the kind argument produces the following scatterplot:

How do I get rid of the error to display the data as a violin plot?

Thanks in advance

edit:
Seaborn version: 0.9.0
Numpy version: 1.16.2
Python version: 3.7.3


texts: Predicting next adverse event based on previous events

texts:
I have a clinical trial dataset where the patient details are recorded at certain time intervals(visits) over period of time. Each visit will contain data recordings of all the adverse events experienced prior to the visit, any medicines taken and the dosage of experimental drug in no particular order. for example a visit could have recordings of 5 adverse events (nausea, headache, hypertension etc.) experienced in the week prior to the visit, dosage of experimental drug and some medicines given to avoid the adverse events.

Now i need to predict the occurrence of adverse events for a patient before his next visit and also need to know which medicines cause some of these adverse effects.

I hope I've clearly explained the problem. What Machine learning/Statistical methods would best solve this?


texts: Testing trained model on the image from the test set

texts:
I trained my EfficientNet (CNN) and got accuracy=0.73. The question is how to check it on one concrete image from the testing set? How to write a code in python for it?
I described the testing set this way:
testing_data = np.load("testing_data.npy", allow_pickle = True)
test_X = torch.Tensor([i[0] for i in testing_data]).view(-1, 3, 224, 224)
test_y = torch.Tensor([i[1] for i in testing_data])

This one (https://pytorch.org/docs/stable/data.html ch5) doesn't work for me because I didn't use torch.utils.data.DataLoader in the previous code. My model is called net. I only tried something like this:
net(test_X[0])

but got an error:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 3, 3], but got 3-dimensional input of size [3, 224, 224] instead


texts: Difference between interpolate() and fillna() in pandas

texts: Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone explain me in layman terms. I already visited through the official documentation and wanted to know the difference


texts:
I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering:

Can I use model selection in clustering using the silhouette score? - Since my dataset does not have any ground truth labels, I read on the sklearn documentation that you can use Silhouette score to evaluate the performance of the model. Can I use different clustering techniques (like K Means, DBSCAN, Mean shift, etc.) and select the model with the highest silhouette score? The idea is sort of similar to how we do model selection in supervised learning except in the latter we use cross validation.

How do I detect overfitting in clustering? Since the dataset has no labels, I cannot think of a way to identify if the model is overfitting the data.

How do I plot the final clusters when my dataset has more than 2 dimensions? I have seen a lot of visualizations around clustering (like the one below):

Should I use PCA to reduce the features to 2 and then plot the clusters? or is there another way to do this?


texts: Can finite state machines be encoded as input/output for a neural network?

texts:
I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task.

Are there any ways in the literature for doing this?

I've already found some algorithms being able to extract a DFA from a recurrent neural network, but nothing about DFAs either as input or output of ANN.


texts: Deep learning model performance for different datasets?

texts: I have a question that my single deep neural network model gives above 90% accuracy for one data set and the same model gives an accuracy between 70-80% for the other data set. I want to know why this variation in accuracy happens in spite of the fact that i am using the same deep neural network model.


texts: What is the azure solution for storing and querying Netcdf data on the fly?

texts:
We currently run a live web application which utilises standard azure mounted file storage to store 2tb worth of metocean data in netcdf format. On here we run python queries directly on the data e.g. Xarray. Extract 1 month of wave height data for a single location (this takes about 10 seconds).
We want to scale this storage and looking at different solutions such as azure datalake. Is a datalake mountable and accessible like a normal storage drive? Is this suitable for more immediate queries or more suited to long term data intensive calculations? Any ideas on a better solution?


texts: Does scikit-learn have forward selection/stepwise regression algorithm?

texts:
I'm working on the problem with too many features and training my models takes way too long. I implemented forward selection algorithm to choose features.

However, I was wondering does scikit-learn have forward selection/stepwise regression algorithm?


texts: ADAM algorithm for multilayer neural network

texts:
I’m trying to touch neural networks without using “in box” algorithms. And so I found out that nowhere is written how to calculate square of gradient for hidden layers in ADAM optimizer.
I took the description from original article and the problem is that Vt is some kind of normalizing coefficient, so it should be scalar. But for hidden layers gradient is not really a vector byt a matrix. So how should we calculate gt^2 in this case?


texts: Train a classifier for a game with feedback on chosen move instead of true labels

texts:
I'm having some trouble describing in one line what I want, which is probably why I haven't had much luck with Google.

Say I have a game like 2048 where the possible actions each step are fixed (and more than two). I want to train a neural network that chooses a move, so I have 4 neurons in output layer and I make the move with the highest prediction. The output vector is normalized (softmax layer).

However, the training data I have is just the state, the move that was made, and whether that had good or bad results. If the chosen move is bad, I don't know which of the other ones was better (if any).

How should I train this? My current thought is like this:

Good move? -> chosen action gets positive error (so prediction goes up)
Bad move? -> chosen step gets negative error (so prediction goes down)

But I haven't found literature supporting this guess. There are alternatives:

Maybe I should also update the options that weren't chosen (in the opposite direction)?
Is it a good idea to set error directly instead of using goal predictions?
The error for correct and incorrect could be different, maybe to preserve normalization?
...

(I'm doing 2048 and using neural networks, but I think this is not limited to this game or this method.)


texts: Using deep-learning on graph data for binary classification

texts:
The data:

I have certain data that I decided to represent it as a graph (I thought it would suit).
So I have the weighted graph data that includes a numeric attribute for each node. (networkx graphs).
Each graph represents a session.
Each session label is either good (1) or bad (0).

The mission:

I need to predict given an unlabeled graph, whether it's good (1) or bad (0).

What did I do so far:

I've made the ML method that calculates features (using networkx excellent algorithms) over those graphs.
For example I took the networkx algorithms for calculating betweenesss, degree_centrality, closeness_centrality, etc

I've received better results than the currently available results (which didn't use a graph for representing the data): F1-Score ~ 65%, ROC_AUC ~ 90%.

My intuition:

Maybe I shouldn't randomly choose networkx function. What if I could do something smarter using deep-learning. The model should understand how bad graph looks like, a good graph looks like, and make the classification

Problems:

I'm not sure if my intuition is correct. Maybe feeding the graph as is wouldn't be enough in order for the model to learn. I feel that I need advice regarding this approach, and especially previous similar works if they exist.

Relevant previous work

http://openaccess.thecvf.com/content_cvpr_2017/papers/Monti_Geometric_Deep_Learning_CVPR_2017_paper.pdf
http://proceedings.mlr.press/v48/niepert16.pdf
https://arxiv.org/pdf/1803.03324.pdf
https://arxiv.org/pdf/1709.05584.pdf

Questions:

Is anyone familiar code implementations of those previous work / other previous works?
Is someone familiar for methods that do it with a weighted graph that includes node attributes?
Do you agree/disagree with my intuition?
Do you suggest alternate ways with handling this problem? (DL or not DL)

Thank you :)


texts: I am from CS background and moving towards Data Sciences, I have came to know ML is highly influenced by Statistical Inference/signal processing. The X we use in data science is called input, feature set or independent variables and Y is called target, class, label. What do you call it? Are they different in market and field among practitioners?


texts:
This dataset is collected by a drug making company trying to sell its drug to doctors of different specializations.
The drug company has made promotional activity for its brand
The promotional activity has included calls made, emails sent and faxes sent. The dataset shows how many calls have been made ('Calls Made' column), how many calls were successfully completed ('Calls Successfully Completed' column), how many emails were sent ('Emails Sent' column), how many emails were opened ('Emails Opened' column), how many faxes were sent ('Faxes Sent' column). Brand 1 is the brand of the drug company. Brand 2 is the brand of its competitor.
The dataset shows how many sales have been made for various brands. Brand 1 Sales (Company's brand) - number of sales of the drug company's brand Brand 2 Sales (Competitor brand)- number of sales of the competitor's brand Total Branded Market Sales - number of sales of the all brands (from all companies selling branded drugs) in the market Total Market (Branded + Unbranded) sales - number of sales of all branded and unbranded drugs
My task is to develop a ML system using this dataset that will recommend which of the three channels to use (calls/emails/fax) for future promotional activity. How do I develop such a system? My understanding is that this is a classification problem with three classes (calls/emails/fax). For a prospective customer, I have to classify him/her into one of the three classes that would indicate what channel to use.
What kind of feature engineering should I do? What model can I use?


texts: How to detect when the "bibliography" of a paper has began?

texts:
I'm working on project that involves reading papers from ArXiV to look for particular patterns (It gets complicated but it basically has to do with common phrases and expressions) in their text.

My process can be abstractly described as:

$$ PDF \rightarrow "\text{PDF as a string}" \rightarrow \text{Pattern_Finding_ Method}("\text{PDF as a string}") $$

Now unfortunately, the "bibliography" or references section of many papers cause the Pattern_Finding_Method to break, because they just happen to carry a lot of the behavior we are looking for (but they are not of interest).

As a human it's quite easy for me just manually read and see where the "references" begin and then I can trim that part of the pdf string but at scale this is not practical. Moreover paper writers don't have consistent ways of declaring when their "references" "bibliography" or "acknowledgements" begin.

So it seems natural to view this as an ML/AI problem, where I have a string, I have a loose notion of what constitute the "references" of the string which I can provide training data for (I have the pdf as a string and I can list a character index on the string the references begin)

Now given training data I need to come up with some kind of model that can effectively learn how to detect references on its own.

This is where I get stuck. The data problem i'm dealing with is a highly semantic problem (it's particular organizations of words and their underlying meaning and patterns to how those words are displayed that is giving me cues as to when the reference have began), but my knowledge of learning algorithms is limited to mostly geometric data (SVMs), or at least highly continuous data (Neural Network Models), and then in the case of NLP, my understanding is at best a recipe book of very goal specific algorithms: (ex: TF-IDF for document classification).

I don't know how to bridge the gap from my understanding to creating a specialized model for the problem at hand; that I have intuitive reason to believe will work.

Formal Problem Statement:

Given a collection of large strings, each attached with an integer $i$ indicating where the references begin, determine a model that can reliably detect when references begin on new text.


texts: Getting different visualization results for jupiter and datacamp existing code shell. How to solve this?

texts:
The left one image is in jupiter notebook and the right one is from datacamp exercises. Can anyone please let me know why I am getting different results in Jupiter? Used hacker statistics to calculate the chances of winning a bet. Used random number generators, loops, and Matplotlib to gain a competitive edge!
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(123)
# Simulate random walk 500 times
all_walks = []
for i in range(500) :
   random_walk = [0]
   for x in range(100) :
       step = random_walk[-1]
       dice = np.random.randint(1,7)
       if dice <= 2:
        step = max(0, step - 1)
       elif dice <= 5:
        step = step + 1
       else:
        step = step + np.random.randint(1,7)
       if np.random.rand() <= 0.001 :
        step = 0
       random_walk.append(step)
  all_walks.append(random_walk)

Create and plot np_aw_t
np_aw_t = np.transpose(np.array(all_walks))
Select last row from np_aw_t: ends
ends = np_aw_t[-1, :]
Plot histogram of ends, display plot
plt.hist(ends)
plt.show()`


texts: Can we use a neural network to perform arithmetic operation between 2 numbers?

texts: How to develop a neural network which can perform subtraction?


texts: How to use GAN for unsupervised feature extraction from images?

texts:
I have understood how GAN works while two networks (generative and discriminative) compete with each other. I have built a DCGAN (GAN with convolutional discriminator and de-convolutional generator) which now successfully generates handwritten digits similar to those in MNIST dataset.

I have read a lot about GAN's applications for extracting features from images. How can use my trained GAN model (on MNIST dataset) to extract feature from MNIST handwritten digist images?


texts: Why is a perceptron initialized with a random line?

texts:
I'm setting up a single perceptron for doing linear classification.

Why is the perceptron initialized with random weights and a random bias instead of just having all of the weights set to zero initially? Wouldn't the loss optimization happening in the back propagation work more effectively on average if these values were all set to zero, as (I think, intuitively) this would be the average of all values that these parameters could take, so starting there is essentially like starting at the "middle" of the space of all perceptron solutions? Why is random a better choice than zero?


texts: Risk analysis on the uptime of web applications (classification? survival analysis?)

texts:
I've conducted a couple studies examining how "risky" several web applications are based on whether the respective support teams raise a major incident ticket. The data we have includes the historical times of major incidents, the change tickets that have been pushed to prod (and at what time they were pushed), volatility on the market, and a couple other time-varying variables.
So far I've investigated a few approaches - all with their drawbacks:
1) I constructed a rolling-window classification problem where we take the features from the previous week and predict if there's going to be a major incident next week

The issue here was that the performance of this classifier was rather weak - it was better than a random baseline but it was still not great
The advantage of this approach was the resultant business output was "This is the probability that a major incident will occur on your application next week". This has a nice open and close temporal bound on the prediction

2) I used basic survival analysis. I took only the times between major incidents, constructed my probability chart, and output (based only on how much time has passed) the probability of survival at that point

The issue with this approach is my time-bound is less precise. I'm now saying the probability of survival this long rather than the probability of a major incident next week (the latter's structure has much better business implications)

This approach also doesn't allow for including exogenous variables like change ticket information or market volatility

The advantage here is the survival functions seem to make sense when looking at the distribution of major incident times & the implementation is simple

3) I used Cox's time varying proportional hazard model to use survival analysis while also including some exogenous variables such as change ticket information and market volatility

The only issue here is I'm having trouble evaluating the performance of this approach in comparison to basic survival analysis (how can I calculate relative performance of these two approaches?)

In summary - I'm asking what's the best approach to this style of data that I should investigate to determine the riskyness of these web applications? Is there a different approach that I haven't tried? Additionally - how can I evaluate if the Cox time varying proportional hazard is more effective than the basic survival analysis?


texts: What are the limitations while using XGboost algorithm?

texts:
Will XGBoost pose any problem while dealing with categorical variables with more than 2 levels.
For example, occupation variable can have values like doctor, engineer, lawyer, data scientist, farmer e.t.c.
If so what would be a better method to use in that case?


texts: How can I draw CNN Architecture like this one here:


texts:
I am trying to understand the idea of  "Generalized Max Pooling". It seems they try to make the 'pooled' representation similar to the features. If so I feel some rare discriminating features could not be captured by the 'pooled' representation. The 'pooled' representation will tend to be similar to the most frequent features. It will not capture the 'max' feature.

Could you please explain this method.


texts: What is the intuition behind decreasing the slope when using regularization?

texts:
While training a logistic regression model, using regularization can help distribute weights and avoid reliance on some particular weight, making the model more robust.
Eg: suppose my input vector is 4 dimensional. The input values are [1,1,1,1]. The output can be 1 if my weight matrix has values [1,0,0,0] or [0.25,0.25,0.25,0.25]. L2 norm would give the later weight matrix (because pow(1, 2) > 4*pow(0.25,2) ). I understand intuitively why l2 regularization can be beneficial here.
But in case of linear regression l2 regularization reduces the slope. Why reducing the slope only provides better performance, is increasing the slope also an alternative?