texts: Struggling to understand GCNNs (Graph Convolutional Neural Networks)

texts:
Although I've worked with CNN's for over a year, I am struggling to understand how GCNNs work paper on their simplification. I've read several papers, and I find myself out of my depth when they talk about Chebyshev polynomials or Fourier spaces.

The descriptions talk about using an adjacency matrix as input, and perhaps my primary confusion is how I can supply such a matrix to a convolutional neural network (if that is what is in fact what is done). I can't just convolve over the matrix as if it were an image because spatial similarity in the matrix (i.e. rows/cols that are near to each other) doesn't signify actual closeness between nodes in the graph. The question is therefore, how does an adjacency matrix, fit into the framework of CNNs which uses the spatial filters, and the logistic operations (non-linear regressions) in the layered stack/graph.

As this GCNN is less mature than CNNs, which have more explanatory material, some further explanation material as to how these methods differ would be helpful. Are there simple situations in which the usage of GCNN can be used to derive results specific to this method/framework?


texts: How to support a dynamic shape input for tf.where()

texts:
Tensorflow (tf.where) function does not support dynamic input shapes.

For example, the following function calculates the svd of a matrix A and tries to identify the singular values greater than a certain threshold to invert them while filling other values with zeros.

def func (A, threshold)
   s,u,v = tf.svd(A)
   condition = tf.greater(s, threshold)
   z= tf.where(condition , tf.reciprocal(s), tf.zeros(s.shape))
   return z

will result in an error:

  Cannot convert a partially known TensorShape to a Tensor: (?,)

Any idea how to overcome this issue?


texts: Advise resources on un-supervised learning

texts:
I have seen that people coming into data science will rush into scikit-learn or other libraries without trying to learn the knowledge behind. Its good to follow a top-down approach but most of times people don't go into depth, even if they do its in supervised-learning area. That't why I think un-supervised is neglected (or is it heavy influence of statistical inference). I would like to learn the un-supervised learning in-depth I tried some books so far (Sugiyama, Masashi - Introduction to Statistical Machine Learning-Morgan Kaufmann (2016) book) was better among others to build intuition around concepts.
If anyone have gone through same path and know other good reads or video tutorials kindly share in this thread.


texts: Retrain image classifier using MobileNet v2

texts:
I am using my own dataset to retrain mobilenet_v2_100_224 model, I currently have 4 classes where each class have more than 100 images still I'm observing overfitting even though I've used --random_scale and --random_brightness parameters, how should I overcome this overfitting problem when there's no such regularization technique available in the retrain.py script? following is my TensorBoard visualization:

Retrain command:

python retrain.py --tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/2 --how_many_training_steps 500 --random_scale=5 --random_brightness=10 --output_graph=./retrained_graph.pb --output_labels=./retrained_labels.txt --image_dir ./data --summaries_dir /tmp/log


texts: Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions?

texts:
I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly.
I am using the same hyperparameter values between each iteration, and have subsample and colsample set to the default of 1 to prevent any random variation between executions. I am using the scikit learn feature_importance_ function to extract the values from the fitted model.
Any ideas as to why this variation in feature importance could be occurring? Does this mean that some of my features may be correlated and is there a way to ensure the XGBoost outputs the same importance values each time it is called? Note that the prediction and the predicted probabilities are all constant across iterations: it is just the feature importances.
Thanks in advance!


texts: How do I choose a discount factor in Markov Decision Problems?

texts: I'm referring to the gamma in the Value function:


texts: Machine Learning - How to predict set of fixed fields based on past features

texts:
I have quite a large dataset (> 100k rows), which contains information for logistical shipments. (export shipments)

The dataset looks like this:

|shipper|consignee                    |origin|destination                                  |
|-------|-----------------------------|------|---------------------------------------------|
|6409292|288882                       |USSFO |CNPVG                                        |
|6409292|288882                       |USSFO |CNPVG                                        |
|6409292|182724                       |USSFO |HKHKG                                        |
|6409292|182724                       |USSFO |HKHKG                                        |
|8201922|948292                       |USSFO |FRCDG                                        |
|8201922|948292                       |USSFO |FRCDG                                        |
|8201922|948292                       |USSFO |FRNIC                                        |
|8201922|291222                       |USEWR |AEDXB                                        |

So what we have here is a list of past shipments. It shows the relationship between shipper and consignee, and from where the shipment was from and where it was sent to.

Based on this past data, I wish to be able to predict when a new shipment is added by looking at the consignee code and origin.

Example

Take below new booking as an example:

|shipper|consignee                    |origin|destination                                  |
|-------|-----------------------------|------|---------------------------------------------|
|1234567|948292                       |USMOB |?                                            |

How can I train a model to predict the destination? And what is this area in ML referred to?


texts: Identify members who are likely to switch where they receive drug administration

texts:
I have access to medical claim data from a large health insurance company. As some of you may know there is a large delta between the price of drug X depending on where it is administered.
My company wishes to help members to reduce the cost they pay in deductibles, co pays, and coinsurance as well as reducing the cost for the health play by identifying the lowest cost of care typically home infusion or physician administration.
I am researching various approaches and customer segmentation modeling seems like the best approach to this sort of problem.
The claim data goes back 3 years and is at a member level. For each member claim, there could be many records in the DB. One line could be for 2hrs of chemo admin, another for the drugs they were provided, others for nurse's time, etc. Each row specifies where the site of care was i.e. home infusion, physician, outpatient, specialty pharmacy. Each claim also provides details on the primary diagnosis I.e. rheumatoid arthritis (R.A) etc.
Ideally, I would like to identify those members who use expensive sites of care when there are cheaper alternatives available. I was thinking for example that for primary diagnosis for R.A I could filter down to R.A members using the diagnosis code and then cluster those members based on certain input features such as demographics, distance to home infusion center/hospital/physician, level of cover, frequency of refill, dosage, cost of a drug, the current site of service, etc.
Is my approach right or is there a better way to solve this problem other than clustering?


texts: One-hot encode multi-class multi-label sequences

texts:
Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each entry has a different number of possible values (categories).
For example the first entry has 0-3 values, the second 0-1 and so on...

I want to train an LSTM model in order to predict the next timestep. So I defined a one hot encoding of each entry by means of the maximum number of classes:

For example [2, 0, 5] becomes

[[0. 0. 1. 0. 0. 0.],
[1. 0. 0. 0. 0. 0.],
[0. 0. 0. 0. 0. 1.]]

Unfortunately this kind of representation raises the error

  ValueError: Invalid shape for y: (1, 3, 5)

I have three questions:

Is it possible to pass a 3d y target to Keras?
Should I define a single one-hot encoding which combines all the possible triplets of categorical values instead? The problem is that in this case I would lose the correlation between the occurrences of the labels in the same category, because each possible combination of labels would become independent from the other ones.
Should I only one-hot encode the target y or also the input X?


texts: Difference between MapReduce, Dryag and Pregel in terms of efficiency

texts:
What is the difference between the three parallel processing paradigms: MapReduce, Dryag, and Pregel in terms of efficiency for big data analytics applications?
Thank you.


texts: Confidence range for individual prediction from multiple regression

texts:
I've built a multivariate regression model for predicting the price of an item given a couple of other factors about that item - age, condition, etc. So, given values for age and condition I can return a single value for price.

But what I want to do is return a price range, representing, say, 95% confidence that the true value will fall within this range.

I've googled a bunch, but I can't find a clear description of how to achieve this. The closest I've come is what's described in this paper: http://pareonline.net/getvn.asp?v=7&n=2, in the section "Calculating a Predicted Score, and Confidence Intervals Around That Score".

Given that the confidence of a regression model "bows" in around the mean for each value, I feel like this is more complex than just taking +/-X% of the predicted value. The range should be greater for items that are at the extremes of age and condition, right?

Grateful for any help.


texts:
The following code

DBSCAN_score = pd.read_csv("DBSCAN_scores.csv")
sigmoid_score = pd.read_csv("Sigmoid_scores.csv")

score = pd.concat([sigmoid_score.F1_score.fillna(0), DBSCAN_score.F1_score.fillna(0)], axis=1)
score.columns = ["Sigmoid", "DBSCAN"]
ax = plt.gca()
ax.set_title("F1 score comparison")
score.plot(ax=ax)
plt.show()

Normaly I would get the output bellow

Now I am getting the following result with the same code and the same files.
I'm struggling to make it look like the output above


texts: What is the difference between regplot and lmplot in seaborn?

texts: Seaborn library in python suggests to use either lmplot or regplot to visualise a regression between two variables. What is the difference between the two plots ? The result I was able to get are slightly different but I have no idea why !


texts: How to use embedding to reduce features for a regression problem

texts:
I’m working on a regression problem in which I’d like to predict demand of different items. I have used holidays as a feature in my model, in a hot encoded format, i.e. I have 11 binary features each representing one holiday.
I’d like to reduce the size of my features and thinking to used embedding on the holiday features to represent them in lower dimensions.
I’m new to embedding. My question whether it make sense or not and if it does any hints on how to do it?


texts: Which is faster for imputing: R or Python?

texts:
I have a fairly large dataset (about 40k rows, 40 columns) with many NA's (up to 40% of each variable). All of my work has been done in R so far, but I know R (which I'm told runs only on a single core) won't be able to impute the data with any type of respectable turn around time, especially on my laptop.

While I'm looking into more powerful cloud computing to help, I'm wondering if Python could offer better imputation times (presumably it can be run on more than one core).

I'm also still deciding on the algorithm -- likely Cart, KNN, or Forest. It's survey data.


texts: Quick guide into training highly imbalanced data sets

texts:
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test samples as a majority class.

Some good answers about sub-sampling and weighted random forest are given here: What are the implications for training a Tree Ensemble with highly biased datasets?

Which classification methods besides RF can handle the problem in the best way?


texts: Strange behavior of CNN when forecasting time series

texts:
I have a time series containing 5 features. I tried to use LSTM to predict the next 112 periods in the series. However, I got very bad results. So I tried to use CNN.
First, it did not work properly when using a lot of data for training, and it was not possible to find the characteristics of the series. By decreasing the data, leaving only 224 data for training and 224 for validation, CNN can find the data pattern but results in overfitting (Final epoch: loss: 4.7838e-15 - val_loss: 0.0971).
The strange behavior observed is that even though it results in overfitting, it manages to generalize and predict better future values than when using some kind of regularizer. Does anyone have an explanation for this? Any tips to improve predictions in this case?
The code:
model = Sequential()
model.add(Conv1D(filters=32, kernel_size=2, activation='sigmoid', input_shape=(n_steps_in,n_features)))
model.add(Conv1D(filters=32, kernel_size=2, activation='sigmoid'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(32, activation='sigmoid'))
model.add(Dense(32, activation='sigmoid'))
model.add(Dense(n_output))
model.compile(optimizer='adam', loss='mse')

from keras.callbacks import EarlyStopping
stop = EarlyStopping(monitor='loss',min_delta=0.000000000001,patience=30)

# fit model
history = model.fit(X, y, epochs=1000, verbose=1,callbacks=[stop], validation_data = (test_X, test_y))


texts: Why MA model order is from acf but not pacf

texts:
For MA model in Arima, why the order references acf, but not pacf? The emphasize is why not PACF.

In https://towardsdatascience.com/significance-of-acf-and-pacf-plots-in-time-series-analysis-2fa11a5d10a8

It mentions

  Why would that be? Since our series is linear combination of the
  residuals and none of time series own lag can directly explain its
  present (since its not an AR), which is the essence of PACF plot as it
  subtracts variations already explained by earlier lags, its kind of
  PACF losing its power here

I don't still get why PACF cannot be used here, anyone can give a better intuition why it should be ACF but not PACF for giving the order for MA model?


texts: In Roberta if there is no NSP how are the weights update during training

texts: in Roberta if NSP loss is not used , what is the loss function for DOC-SENTENCES or FULL-SENTENCES task , on what basis will it update the weights ?


texts: How to evaluate data capability to train a model?

texts:
Bioengineering data comprises of 512 binary features and a single Boolean label: if the particular mixture is worth further research. There are about 1,200,000 results of previous experiments available.

Every "physical" test costs, so it is tempting to train a model to predict if a certain combination of features is worth real testing.

I tried a few models, all failed to render better accuracy than 52% on test data, which is very close to blind guess considering binary classification. Namely, a Feed-forward Multi-layer Perceptron, Random Forest, Naive Bayes models.

For a plain simple XOR example from ML starter courses, for two binary inputs, all 4 cases are required to train the network.

1.2 mln samples are, perhaps, still a too small dataset to train for 512 inputs, considering a 512-bit number of all combinations and complex cross-influence.

Thinking from the opposite, how to prove that it is impossible to train any model given available training data?


texts: Computing Image Similarity based on Color Distribution

texts:
Image Similarity based on Color Palette Distribution

I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as follows,

Img1: {'Brown': 14, 'White': 13, 'Black': 40, 'Gray': 31}

Img2: {'Pink': 82, 'Brown': 8, 'White': 7}

Where the numbers denote the % of that color present in the image. What would be the best way to compute similarity on a scale of 0-100 between the two images?


texts: Grouping domain specific words/phrases with same meaning

texts:
I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/phrases and the sentence would still have the same meaning.

Broken: damaged, ruined, busted, unfit for purpose, missing a leg

I want to do this for texts that contain domain specific and colloquial jargon so existing NLP solutions may not be suitable?

My intention is to do this as a bridging step between named entity extraction and named entity linking.


texts: Handling underflow in a Gaussian Naive Bayes classifier

texts: I am implementing a Gaussian Naive Bayes classifier (so each feature is continuous and assumed to be coming from a Gaussian distribution).  When evaluating the probability of a feature value in the test set, if the value is sufficiently far away from the mean (e.g. the mean and s.d. on the training data is say 0 and 1 but the test value is 10^10) then there is underflow.  This is an issue because then the probability will be calculated as 0.0 so the log probability is undefined.  Is there a standard way of handling underflow in this case?


texts: Is it accurate to say that "K-means clustering the vectors based on keywords weight similarity"?

texts: Long story short, I have 200 vectors as a result of TF-IDF (Term Frequency - Inverse Document Frequency) on thousands of keywords in hundreds of vectors. The total number of unique keywords that I got is 745 keywords, meaning that there are 745 dimensions/axes. Now, I was wondering how does K-means clustering work on those 200 vectors? Is it accurate to say that K-Means is clustering those 200 vectors by the keywords weight similarity?


texts: How to find the weights for weighted least squares regression?

texts: When we are doing weighted least squares how do we find the weights? Where ever I see tutorials are just using $w_i = \frac{1}{(sigma)i^2}$ and doing it with basic data. But I want to know how to find the weights for real data. Is it always the inverse of the square of variance?


texts: Autoencoder for cleaning outliers in a surface

texts:
I have been looking at autoencoders from the keras blog here: https://blog.keras.io/building-autoencoders-in-keras.html

I was wondering, what motifications would be necessary in order to be able to give it different surfaces i.e. 2-dimensional vectors, of which some of them have large spikes.

For example here we see a surface that looks clean:

How could a neural network look like, if I wanted to remove individual spikes from this surface?

Am I right in thinking that a normal fully connected feed forward propagation would be sufficient? If so, is there any way to control thresholds when spikes are should be eliminated?

Also, would you agree that the training principle would still be the same as show in the keras blog? Would it work if I simply trained it with many good examples of clean surfaces to recognize themselves?


texts:
I have a problem and I'm having trouble representing it - first I thought I should use graph theory (nodes and edges) and now I'm not sure.

My data is some tanks names and it's volumes, those tanks are connected by pipelines which I have the names and length.

------(pipeline 1)------.-----(pipeline 2)------.----(pipeline 3)---
  |                     |    |                           |         |
[R tank 1]        [S tank 1] [S tank 2]            (pipeline 4) [S tank 3]
                                                         |
                                                     [S tank 4]

R tank is sink (receiver) and S tank is source (sender)

Problem is the pipe names change doesn't occur where there is a tank - they change name because historical reasons, size or connections...

So if I want to graphically show that S tank 2 is connected to pipeline 2 at point X and pipeline 2 connects to pipeline and the content goes to R tank 1, how should I do this? (I think the point X may not be relevant but if I had some way to get the distance travelled would be great).


texts: Random accuracy results on validation set (Binary classification)

texts:
I am configuring a CNN model for a classification problem of human action in Python using TensorFlow.

My data is video frames representing human body joints. Every 3 consecutive columns represent x,y,z coordinates of one joint.

I set the placeholders in tf to be of shape (None, l, j*3) where l=fixed number of frames and j= number of joints. Now the model is giving me very random results (pretty much noise) on validation.

This is the code I used to compute loss and validation and plot results.

for e in range epochs:

# Loop over batches
        for x,y in random_batches(X_train, Y_train, batch_size):

            # Feed dictionary
            feed = {X : x, Y : y, keep_prob_ : 0.5, learning_rate_ : learning_rate}

            # Loss
            loss, _ , acc = sess.run([cost, optimizer, accuracy], feed_dict = feed)
            train_acc.append(acc)
            train_loss.append(loss)

            # Print acc & loss every 5 iter.s

            if (iteration % 5 == 0):
                print("Epoch: {}/{}".format(e, epochs), "Train acc: {:.6f}".format(acc))

            # Compute validation loss every 10 iter.s

            if (iteration%10 == 0):
                val_acc_ = []
                val_loss_ = []

                for x_v, y_v in random_batches(X_valid, Y_valid, batch_size):
                    # Feed
                    feed = {X : x_v, Y : y_v, keep_prob_ : 1.0}

                    # Loss
                    loss_v, acc_v = sess.run([cost, accuracy], feed_dict = feed)
                    val_acc_.append(acc_v)
                    val_loss_.append(loss_v)

                # Print
                print("Epoch: {}/{}".format(e, epochs), "Validation acc: {:.6f}".format(np.mean(val_acc_)))

                # Store
                validation_acc.append(np.mean(val_acc_))
                validation_loss.append(np.mean(val_loss_))

            # Iterate
            iteration += 1

    ## Plotting:

t = np.arange(iteration-1)

plt.plot(t, np.array(train_loss), 'r-', t[t % 10 == 0], np.array(validation_loss), 'b*')

So I thought maybe if I tell the system to consider that every 3 consecutive columns are representing just one joint it would solve this randomness on validation.

Not sure how to do this as tensorflow accepts arrays. And arrays don't have headers unlike dataframes.

0  0.30467  0.45957 -0.95414  1.74687  1.42338 -0.03860
1  0.27331  0.59293 -1.00874  1.74135  1.32004 -0.00701
2  0.30348  0.88129 -1.05517  1.75090  1.65138 -0.03112

What I would want the system to understand is that the first 3 columns represent the same joint and the 2nd 3 columns represent the 2nd joint.

I tried shaping the data to be (None, l, j, 3) but that also gave me random results on validation as well. And when I looked up similar works, they were feeding tf data such as (None, l, j*3).

Another area where I am doubting is wrong (although less likely to be wrong) is the shuffling of the data. I have my inputs data (X) and I have constructed the labels (Y) from the filenames. Below is a snippet of the get batches function (which does the shuffling):

# 1- Shuffle (X, Y)
m= X.shape[0]

    permutation = list(np.random.permutation(m))
    shuffled_X = X[permutation,:,:]
    shuffled_Y = Y[permutation,:]

    # 2- Partition (shuffled_X, shuffled_Y). Minus the end case.

    num_batches = m // batch_size # number of mini batches of size batch_size in the partitionning
    X, Y = shuffled_X[:num_batches*batch_size], shuffled_Y[:num_batches*batch_size]

    # Handling the end case (last mini-batch < mini_batch_size)
    if m % batch_size != 0:
        X = shuffled_X[num_batches * batch_size : m,:,:]
        Y = shuffled_Y[num_batches * batch_size : m,:]

    # Loop over batches and yield
    for b in range(0, len(X), batch_size):
        yield X[b:b+batch_size], Y[b:b+batch_size]

I am a beginner in deep learning and tensorflow. Appreciate any advice or help here.


texts: What is the purpose of untrainable weights in Keras

texts: In this page it is mentioned that when trainable=false, the weight won't be updated and is used for optimization, too. But I still do not understand how it can be useful? (e.g., I want to find out the best number of neurons or the best drop out rate, can it help?)


texts: How do I extract album and song titles from this plain text file?

texts:
Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the results into a single plain text file that looks like this:

To perform my intended analyses, I want the target file to look like this, where each row is a line of lyric with columns indicating the corresponding album (wrapped around by "*") and song (after a track number "n. ") titles.

Maybe I can just read each line as a row and manually add album and song titles. However, I was wondering if there's a more efficient way to do this.


texts: Python: Fast indexing of strings in nested list without loop

texts:
I have an array dataset from which I need to build D whose each element is a list of strings (Third column of dataset). The dataset looks like:
600,900,3418309
600,900,3418309
600,900,3418314
600,900,3418314
600,900,3418319
600,900,3418319
610,800,3418324
610,700,3418324
600,900,3418329
620,900,3418329
600,900,3418329
600,900,3418334
610,900,3418334
600,900,3418339
600,900,3418339
600,900,3418339
660,700,3418339
610,800,3418339
660,700,3418339
600,900,3418339
600,900,3418339

I want to check for every new string, and if it is not a part of any of the array element then append new string to only that array element.
Since the data on new strings to be added is large, I do not want to use loop. Is there any method to do it fast. I have to use Python.
Right now I am using the code, which is very slow:
for i in range(len(dataset)):
 for j in range(int(int(dataset[i,0]-600)),int(int(dataset[i,1]-600)+1)):

    if(str(dataset[i,2]) not in D[j]):

        D[j].append(str(dataset[i,2]))


texts:
I'm taking a course on Supervised Learning in R: Regression. There is a section where I'm supposed to predict blood pressure given age and weight. This is was MY approach

# Create the formula and print it
fmla <- lm(blood_pressure ~ age + weight, data=bloodpressure)
fmla

# Fit the model: bloodpressure_model
bloodpressure_model <- fmla

# Print bloodpressure_model and call summary()
bloodpressure_model
summary(bloodpressure_model)

It was an incorrect submission. The message error message read -  "The contents of the variable fmla aren't correct."

DataCamp's solution was this

# bloodpressure is in the workspace
summary(bloodpressure)

# Create the formula and print it
fmla <- blood_pressure ~ age + weight
fmla <- lm(blood_pressure ~ age + weight, data=bloodpressure)

fmla

# Fit the model: bloodpressure_model
bloodpressure_model <- lm(fmla, data = bloodpressure)

# Print bloodpressure_model and call summary()
bloodpressure_model
summary(bloodpressure_model)

Both of these models had  the same diagnostic results. What's the issue with MY approach?


texts: Baking perfect cookies: methods or techniques for determining optimal ranges of x variables in classification models

texts:
This is probably best explained via an analogy: let’s assume we’re baking cookies because cookies are awesome! Let’s also assume we can only control three variables in our cooking process:

time (x1)
temperature (x2)
humidity (x3)

There’s an obvious interaction between time and temperature, but let’s also assume there are additional interactions among the other variables.

Our outcome variable (y) is “cookie goodness” – for the sake of argument, let’s force it to be binary – we have “good cookies” and “bad cookies”.

There are plenty of methods and models for classifying and understanding how to determine and predict good cookies vs. bad cookies, but let’s assume we want the best “settings” for time, temperature, and humidity. Since these are settings, we can assume they must be continuous “ranges” for their settings (for example, one possible rule for settings would be: keep time between 15-25 minutes, temperature between 350 and 375, and humidity above 50…there is no other cross section of variables that will yield more “good cookies”).

Questions

Are there methods or techniques for approaching a classification model with these constraints? Initially we may think tree-based models, but a decision tree will likely return several interactions and since I need one ideal “setting” for each variable, how do we know we're not missing some subset of “settings” down a different branch of the tree?
Is this just an exercise of optimizing/pruning the tree parameters and selecting the best branch?
Is there some method or way to force a decision tree to only include one set of continuous ranges for each variable or to collapse/summarize the output of the tree to continuous rulesets for each variable?

Aside: This almost feels like a reverse Design of Experiments (DOE)?

Practical Considerations

Then there’s the problem when the tree suggests too “tight” of a range (e.g. keep temperature between 400 and 401 – my oven’s not that accurate)? I suppose we could check branches with high support and high confidence and then see if they have implementable ranges. I suppose "best/optimal" is subjective, so let's say we need to have at least 10% support (50 trials) and 75% confidence.

In case anyone wants to explain with some fake data, here is some R code:

library(tidyverse)

n <- 500
time <- runif(n, 15, 40)
temp <- runif(n, 300, 450)
humidity <- runif(n, 35, 60)

cookie_goodness <- (2*time + 3*time*temp + 4*humidity*temp  + 5*humidity + rnorm(n)) / 10000

good_cookie_lower_limit <- quantile(cookie_goodness, 0.25)
good_cookie_upper_limit <- quantile(cookie_goodness, 0.75)

df <- data_frame(
  trial = 1:n,
  time = time,
  temp = temp,
  humidity = humidity,
  good_cookie = as.factor(
    if_else(between(cookie_goodness, good_cookie_lower_limit, good_cookie_upper_limit),
            1, 0)
    )
)

# # A tibble: 500 x 5
#    trial  time  temp humidity good_cookie
#    <int> <dbl> <dbl>    <dbl> <fctr>
# 1      1  40.0   377     54.5 0
# 2      2  39.0   384     53.4 0
# 3      3  30.9   325     38.4 0
# 4      4  25.5   447     35.1 1
# 5      5  19.0   348     44.7 0
# 6      6  37.1   431     42.7 0
# 7      7  34.8   400     43.5 1
# 8      8  26.2   346     36.9 0
# 9      9  19.2   409     38.0 0
# 10    10  38.0   408     46.9 0
# # ... with 490 more rows


texts: Modeling the influence of events order on probability

texts:
The case is to model if the sequence of events influences the probability of binary target variable. We have for example five different events which occur in time (event: A,B,C,D,E). They can occur in order from 1 to 5. I would like to check if the order of their occurrence influences the target variable.

My first idea was to convert the time of occurrence into numbers from 1 to 5 and then for example use logistic regression.

Do You know any other practices?
Any whitepapers and ideas will be helpful.


texts: Calculating correlation between two time variables

texts:
I have a dataset that looks like this :

    UserID  App_open_Time(Hour_ofDay)   Email_Open_Time(hour_ofDay)
       1        1                         1
       2        1                         1
       3        1                         1
       4        1                         1
       5        1                         1
       6        2                         1
       7        2                         1
       8        3                         1
       9        3                         1
      10        3                         1

I would like to know if the App_open_time is correlated with Email open time .
How can i do this analysis in python.
I was planning to do a pearson-correlation in Python using Numpy ,Is this the best approach?


texts: Validity of cross-validation for model performance estimation

texts:
When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure, several models are created, one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

I am curious whether it is really valid to report the validation performance as estimated performance of the final (selected) model (as the performance was derived using all the other models which were created during the validation procedure, but are not considered when using the final model for predictions).

I would expect that the deviation of the selected model's performance might vary drastically from the average performance of all models (depending on several factors such as the used algorithm and validation scheme).

Why is cross-validation used to estimate the performance of a predictive model despite the given fact (e.g. in many peer-reviewed scientific publications)? Wouldn't it always be better to conduct an additional performance evaluation with the selected model on an independent test set and report the resulting performance alongside the validation performance?


texts:
I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be labelled such that rows would be paired (or even grouped). The output for each row in the test/prediction set would be the most similar row and the probability that it would have been paired with that row.
In theory there could be a matrix with a score for the pair by pair comparison but I only need the highest for my use case.
This feels somewhat like collaborative filtering but I am not quite sure what to call this class of algorithm. Bonus points if you can point me to a python library.


texts: Potential problems with expanding training set

texts:
The problem is a binary classification one. My dataset contains users with activity over multiple days, where they all start with class 0 and can become class 1 after a certain activity (which is not part of the input features). If I have 1000 class 0 users and 100 class 1 users, I will have 1100 training instances.

What I want to do is expand the data such that for each day of activity of each user, there is a row in the training set. So user 1 with 10 days of activity will have 10 rows in the training data, all with class 0, instead of 1 row. User 2 who has 10 days at class 0 and 5 days at class 1 will have 15 rows instead of 1.

This can give the model multiple times more examples to learn from. The only drawback I see is that it will change the ratio of class 0 to class 1 examples (not sure if that is a problem). Are there any unforeseeable disadvantages to this approach?


texts: Predicting next medical condition from past conditions in claims data

texts:
I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of diagnosis (ICD-9CM) and procedure codes (CPT, HCSPCS, ICD-9CM).

My goals are to:

Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease;
Identify the likelihood (or probability) that a patient will develop a medical condition based on the conditions they have had in the past;
Do the same as 1 and 2, but with procedures and/or diagnoses.
Preferably, the results would be interpretable by a doctor

I have looked at things like the Heritage Health Prize Milestone papers and have learned a lot from them, but they are focused on predicting hospitalizations.

So here are my questions: What methods do you think work well for problems like this? And, what resources would be most useful for learning about data science applications and methods relevant to healthcare and clinical medicine?

EDIT #2 to add plaintext table:

CKD is the target condition, "chronic kidney disease", ".any" denotes that they have acquired that condition at any time, ".isbefore.ckd" means they had that condition before their first diagnosis of CKD.  The other abbreviations correspond with other conditions identified by ICD-9CM code groupings.  This grouping occurs in SQL during the import process. Each variable, with the exception of patient_age, is binary.


texts: Weighted clustering of lat lon coordinates

texts:
I have millions of lat long points that have been grouped into squares. Some squares have thousands of points, others have a couple of points. The idea is that we have one set of lat long for the square with a weighting related to the square based on number of items it has instead of having millions of rows of data to cluster.

I was originally using the leadercluster algorithim which allowed me to specify the distance each of my clusters should cover. This is ideal for my use case but now i would like to use it to cluster squares factoring in the weight of the squares. This is basically a learning experience for me and any help would be greatly appreciated

I have seen this which could be useful but I'm not really able to make headway with it

Below is some sample data in R about airports and arrivals

df <- read.csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_february_us_airport_traffic.csv')

ggplot(df, aes(x = long, y = lat)) +
  geom_point()

sample <- df %>%
  select(long, lat, arrivals = cnt)


texts: Can AlexNet outperform ResNet as a feature extractor?

texts: I've been using the pre-trained Deep Learning models as feature extractors on a project involving chest x-ray images to detect TB. I extract the features from the layer just before the Softmax. I could repeatedly find that the performance of AlexNet is way better than ResNet-50. Why is it so?


texts: Handling features with multiple values per instance in Python for Machine Learning model

texts:
I have a dataset which contains medical data about children and I am developing a predictive machine learning model to predict adverse pregnancy outcomes. The dataset contains mostly features with a single value per child, e.g. gender = ["Male", "Female].
However, I have some features that have multiple values per child, such as the abdominal circumference which has been recorded multiple times per child, as such:
    ChildID     abdomcirc
0   1           273
1   1           267
2   1           294
3   2           136
4   2           248

So in the above table child 1 has 3 values for abdomcirc and child 2 has two values for abdomcirc. Adding this feature to the remaining dataset (comprised of single observational features) will result in nearly duplicate rows, apart from the different values for abdomcirc, like so:
    ChildID     gender  diabetes  birthroute  abdomcirc
0   1           Male    No        Normal      273
1   1           Male    No        Normal      267
2   1           Male    No        Normal      294
3   2           Female  Yes       csection    136
4   2           Female  Yes       csection    248

I am unsure what the best way to deal with these features is, without merging the data and having near-duplicate rows. I have considered the following:

Using python list type for abdomcirc. However, I do not know if a machine learning model can handle this data type. So my data will look something like this
      ChildID     gender  diabetes  birthroute  abdomcirc
 0    1           Male    No        Normal      [273, 267, 294]
 1    2           Female  Yes       csection    [136, 248]

Transforming abdomcirc into a single observational feature by calculating the mean (although I am not sure how useful this information would be for my predictive model) like so:
      ChildID     gender  diabetes  birthroute  abdomcirc
 0    1           Male    No        Normal      278
 1    2           Female  Yes       csection    192

I have tried looking for resources to help me with this but have not been very successful, maybe because I am not typing the correct keywords or something. So, I would appreciate your opinions and helpful resources. Many thanks!


texts: Calculate Similarity using User's Personal Data?

texts:
I want to find out which users are similar to each other using their personal/organisational data, such as department, company, site, etc.

I have this data in a boolean format, as shown below:

       Dep1 Dep2 Comp1 Comp2 Site1 Site2
    U1  0    1     0     1     0     1
    U2  1    0     0     1     1     0
    U3  1    0     1     0     1     0
    U4  0    1     0     1     0     1
    U5  0    1     0     1     1     0
    U6  1    0     1     0     0     1

I want to select a user and identify the other users that are similar to them, so that I can recommend them software/hardware that those similar users are using.

I have looked into Cosine and Jaccard similarity but I have not seen much help when it comes to calculating them on Boolean data.

I'm using Python and am not new to the language, but I am fairly new to Data Analytics and ML. Any advice is appreciated!


texts: Can I use a GAN to increase my Dataset used for Image detection?

texts: I am currently working on a machine learning project where I use the YOLO Algorithm to detect an object inside of a picture or video. The problem I face is that the specific image set (side-scan sonar) that I am working with is mostly classified, thus there is not a wide range of images available to the public to be used for training. Would I be able to implement a GAN to produce a larger data-set of side-scan sonar imagines to be used for training an image detection Algorithm? I understand that there may be image distortion from using a GAN, but for the purpose of having a larger training set would this be possible?


texts: Regression problem - too complex for gradient descent

texts:
I try to predict temperatures values as function of time and different parameters. The temperature curve look like a "ramp" with some "gauss peaks" on regular intervals.

So, I try to build a regression model for the following equation:

$$
\hat{T}_{a, \mu, \sigma, s, c, m, p}\left(t\right) = a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p
$$

My goal is to establish the values of $a$, $\mu$, $\sigma$, $s$, $c$, $m$, $p$.

My first intention was to use the gradient descent, but I'm pretty sure that the derivative of the RSS based cost function won't be convex, so I might get stuck at local mimina.

I'm also wondering if there is no better alternative to solve that problem. It concerns especially methods not involving minimizing derivatives. I think (but not sure) that neuronal networks can help me with that.

So, in short, how would you solve the following?

\begin{equation}
 \left[\begin{array}{l}
   a \\
   \mu \\
   \sigma \\
   s \\
   c \\
   m \\
   p
 \end{array}\right] \leftarrow \textrm{argmin}_{a, \mu, \sigma, s, c, m, p}\frac{1}{2N} \sum_{i=1}^N{\left(a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t_i-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p - T_i\right)^2}
\end{equation}

Thanks a lot for your help.

P.S.: I'm using python 3 with scipy stack


texts: Text representation using TFIDF .toarray() freezes my computer. Too many features to handle?

texts:
I'm new to Data Science, so hopefully this question makes sense.

I have a dataset with ~50,000 rows. It consists of one column that has item category in it and one column with item description in it.

I'm trying to get bigrams from the item description with the highest correlation to each category.

i'm using TfidfVectorizer to do this.

I compress the item descriptions into an array using this:

features = tfidf.fit_transform(df.ItemDescription).toarray()

In Spyder, when i even click on the features under variable explorer, the memory immediately shoots up to 99% and my computer freezes.

If I try to call features at all, it will also freeze.

Here is how i'm calling features later in my code after making it into an array:

from sklearn.feature_selection import chi2
import numpy as np
N = 2
for UNSPSC, category_id in sorted(category_to_id.items()):
  features_chi2 = chi2(features, labels == category_id)

The array size is 50000X12000

Is that too big? Is there anything I can do to get the bigrams without my computer freezing? As a last resort, I will trim down my data. But i feel like that will reduce the accuracy of any model i try to train later.

Please let me know if you need full code.


texts: Injecting random values as one input feature for feature selection results in a odd beaviour

texts: I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the features that are scoring below the value of the random feature in the importance ranking. But once I removed the features, the performance drop. I don't understand what is happening. I checked this behavior in a gradient boost and in a random forest. Someone has a clue of what is happening?


texts:
I have transaction details for credit data (bank transfers, peer to peer transfers, etc). Currently, I have one year worth of data which I cannot properly classify.

I'm looking for input and suggestions about these two questions:

How do I detect anomalies in the last hour with previous credit transaction data that I have?
How can I detect anomalies in real-time for the current transaction?


texts: Should the input data be normalized using keras pre-trained models

texts: I want to use a pre-trained VGG16 in keras. My question is simple. Should I normalize the input image before predicting its label?


texts:
Is there any comprehensive list of past, current and future NLP challenges?

E.g. for NLP conferences, Joel Tetreault's unofficially official conference calendar and WikiCFP are pretty good.

The "Competitions and Challenges" page on the ACL wiki quite incomplete.


texts: Can there be in-active neuron in output layer

texts:
I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a neuron in hidden layer. Suppose if we use 'Relu' as activation function and our weighted sum is less than 0, then that particular neuron will not be activated right!
So similar to that what if some neuron in output layer have some value and that value is not accepted by a particular activation function means the value is below the threshold for that activation function, so will that particular neuron remain in-active.


texts: Bayesian Neural net with non probibalistic Data?

texts:
Is it possible to construct a Bayesian Neural Network without Probability Distributions as dependent Variable for purpose of predictive modeling?

I mean, if id like to Infer on a Specific Value, like y(e.g. y=5), with a vector of explanatory Variables X(e.g. X=[3,5,1.3,(.....)])
The Bayesian Neural Network infers on a distribution of ymean with standard deviation sigma(e.g. ymean =5, sigma =0.5).

Does it even make sense? Is the loss function of the Neural net able to work by comparing y with ymean, without taking simgma into account?

partial answer:
I think sigma is the result of the distributions in the weight matrices of the Neural net, and it should work. But I want to be sure and understand.

PS: I work in ecology, so getting a probability distribution as result would serve my goals.


texts: How to change a cell in Pandas dataframe with respective frequency of the cell in respective column

texts:
I have a pandas dataframe with binary value columns. I would like to replace values in each cell with its frequency in rspective column in place. My question is how to keep track of the current column while using apply on the subset of columns like here: (to be applied from 8th columns to the end) :

train_data.ix[:,8:] = train_data.ix[:,8:].apply(x: what should come here?)

I know that train_data.ix[:,col_number].value_counts()[0] will return number of zeros in col_number but how can I use it inside apply function?


texts: Feature importance using logistic regression in pyspark

texts:
I am using logistic regression in PySpark. I have after splitting train and test dataset

LR = LogisticRegression(featuresCol = 'features', labelCol = 'label', maxIter=some_iter)
LR_model = LR.fit(train)

I displayed LR_model.coefficientMatrix but I get a huge matrix. How do I select the important features and get the name of their related columns ?


texts: Value error array with 0 features in linear regression scikit

texts:
My input and output data are written in an 6xn row-column excel file,thatI read them using pandas

using this code :

from sklearn import linear_model
import os
import pandas as pd
import numpy as np
import openpyxl as pyx
import numpy as np
import matplotlib.pyplot as plt

path = r"E:\"
os.chdir( path )

data1 = pd.read_excel("input.xlsx", header=None,index=False)
data2 = pd.read_excel("target.xlsx", header=None,index=False)

# print df.head()

le= 20;lg=len(data1)
x_train=[];x_t=[];y_train=[];y_t=[];x_test=[];x_ts=[];y_test=[];y_ts=[];
for i in range(le):
    x_t = data1.iloc[i,:]
    x_train.append(x_t)
    y_t = data2.iloc[i,:]
    y_train.append(y_t)
    if i > le :
        x_ts = data1.iloc[lg-i,:]
        x_test.append(x_ts)
        y_ts = data2.iloc[lg-i,:]
        y_test.append(y_ts)

ols = linear_model.LinearRegression()
model = ols.fit(x_train, y_train)

print (model.predict(x_test)[0:5])

I get this error :

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
  line 880, in runfile
      execfile(filename, namespace)

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
  line 102, in execfile
      exec(compile(f.read(), filename, 'exec'), namespace)

  File "E:/Scikit-Learn.py", line 46, in
      print (model.predict(x_test)[0:5])

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\base.py",
  line 268, in predict
      return self._decision_function(X)

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\base.py",
  line 251, in _decision_function
      X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py",
  line 424, in check_array
      context))

  ValueError: Found array with 0 feature(s) (shape=(1, 0)) while a
  minimum of 1 is required.


texts: Error while trying glmnet() in R: "Error in storage.mode(xd) <- "double" : 'list' object cannot be coerced to type 'double'"

texts:
I'm trying to create a logistic regression model using Ridge, this is the code:
glmnet(X_Train, Y_Train, family='binomial', alpha=0, type.measure='auc')

And this is the error message I'm getting:
Error in storage.mode(xd) <- "double" : 'list' object cannot be coerced to type 'double'

I tried converting all the variables into "numeric" but still doesn't work.
I'm going to post the code for those two datasets so you can reproduce it:
libraries:
library(dplyr)
library(fastDummies)
library(missForest)
library(glmnet)

Data:
url <- 'https://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.data'
crx <- read.csv(url, sep = ",", header = F)

Getting rid of null-values:
crx[crx == "?"] <- NA
crx <- type.convert(crx, as.is=FALSE)
crx.i <- missForest(as.data.frame(crx))
crx <- crx.i$ximp

Data transformations:
crx <- crx %>%
rename(Gender = V1,
         Age = V2,
         Debt =  V3,
         Married = V4,
         BankCustomer = V5,
         EducationLevel = V6,
         Ethnicity = V7,
         YearsEmployed = V8,
         PriorDefault = V9,
         Employed = V10,
         CreditScore = V11,
         DriversLicense = V12,
         Citizen = V13,
         ZipCode = V14,
         Income = V15,
         ApprovalStatus = V16)

crx = subset(crx, select = -ZipCode)

crx <- crx %>%
mutate(ApprovalStatus = recode(ApprovalStatus,
                  "+" = "1",
                  "-" = "0"))

# Normalizing numeric variables:
crx$Age <- scale(crx$Age)
crx$Debt <- scale(crx$Debt)
crx$YearsEmployed <- scale(crx$YearsEmployed)
crx$CreditScore <- scale(crx$CreditScore)
crx$Income <- scale(crx$Income)

crx$Gender <- NULL
crx$DriversLicense <- NULL

Creation of dummy variables:
df <- dummy_cols(crx, remove_selected_columns = T)

df$ApprovalStatus_0 <- NULL
df$ApprovalStatus_1 <- NULL
df$Married_l <- NULL
df$BankCustomer_gg <- NULL

df$ApprovalStatus <- crx$ApprovalStatus

Creation of Training datasets and Test datasets:
X <- df %>% dplyr::select(-ApprovalStatus)
Y <- df$ApprovalStatus

X_Train <- X[0:590, ]
Y_Train <- Y[0:590]

X_Test <- X[591:nrow(X), ]
Y_Test <- Y[591:length(Y)]

And trying to use the glmnet:
glmnet(X_Train, Y_Train, family='binomial', alpha=0, type.measure='auc')

I did some research and I found an article saying that you have to convert everything into numeric class, so I tried converting everything into numeric variables like this:
Y_Train <- as.numeric(Y_Train)
X_Train <- as.data.frame(apply(X_Train, 2, as.numeric))

And still doesn't work. What am I doing wrong exactly?


texts: How to automatically disable register_hook when training model is in eval() phase in PyTorch?

texts:
I require to update grads of an intermediate tensor variable using the register_hook method. Since the variable isn't a leaf-variable, I require to add the retain_grad() method to it after which, I can use the register_hook method to alter the grads.
score.retain_grad()
h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))

This works perfectly fine during the training (model.train()) phase. However, it gives an error during the evaluation phase (model.eval()).
The error:
File "/home/envs/darthvader/lib/python3.6/site-packages/torch/tensor.py", line 198, in register_hook
    raise RuntimeError("cannot register a hook on a tensor that "
RuntimeError: cannot register a hook on a tensor that doesn't require gradient

How could the model automatically disable the register_hook method when it in eval() phase?


texts: How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

texts:
I am new in making recommendation systems . I am using the  surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute the Hit Rate of my top n recommender system.

I know the formula for hit rate is:

(no items users have already purchased)/(no of users)

But this does not makes sense to me because to train and test the user vs item ratings I have only those users in my dataset which have already rated all the items that are in the dataset. (**After all this is what enables me to measure accuracy of my rating predictions **)

This means I will get a HIT RATE OF 100% everytime !!!

Is my theory wrong if not what can I do to properly measure hit rate
I know there is one method called Leave one out cross validation
which can help me in this case . If that's the answer how do they work with top n recommender systems?


texts: Outlier detection with non normal distribution

texts: What are some techniques that I can use for anomaly detection given a non-Normal distribution? I have less than twenty available observations.


texts: How do I add together multiple columns based on header names?

texts:
Here's what my dataframe looks like

Server  Performance123      Performance456
server1       1024             0
server2       110              0
server3       0                1024

Here's what I want it to ultimately look like

Server        Tier1
server1       1024
server2       110
server3       1024

Now obviously I could just add the two columns together but I can't be sure what the "123" or "456" part of the CSV I'm importing will look like as it's the last part of the UID of the datastore.

I've tried this

df['Tier 1'] = df.filter(like='Performance')

But I can't assign that as a new column in the dataframe. Any advice? I'm a bit of a newcomer to this so ran out of ideas quickly.


texts: Feature usage for machine learning algorithm

texts:
Given a list of software installed by users as features, e.g.,

Microsoft_VC80_DebugCRT_x86_x64 1.0.0; Microsoft_VC80_DebugCRT_x86 1.0.0; ;Windows UPnP Browser 0.1.01;Adobe Acrobat Professional 10;

I want to predict whether the student will buy certain product.

Now the question is: what are the ways to turn the list of software into something learnable by a machine learning algorithm?


texts: Calculating probability from a distribution

texts:
I have a histogram/count plots of average time a property is in the market. Given this distribution, I'd like to calculate the probability that a similar property will be leased within 'x # of weeks' or less.

Would a cumulative probability distribution make sense, in this case or there are other ways of calculating?
i.e P(X <= x), where X is a similar property and x = # of weeks).
Say, I want to calculate the P(X <= 4), i.e probability that the property is on the market for less than 4 weeks.
Additionally, I'd also like to calculate conditional / bayesian prb. i.e given a set of property features, calculate the P(X <= 4) based on prior distribution. How would we formulate this as a conditional probability distribution problem?


texts: What are natural (computed) pre-images useful for?

texts:
I've just been reading

  Zeiler, M.D. and Fergus, R., 2014, September. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (pp. 818-833). Springer International Publishing. (Link, my summary)

and (partially)

  Mahendran, A. and Vedaldi, A., 2016. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, pp.1-23. (link, my summary)

They are both about visualizing features being learned in CNNs.

Although both papers have an introduction where you can read things like:

our understanding of [CNN features] remains limited
While the performance of representations has been improving significantly in the past few years, their design remains eminently empirical
In this paper, with the aim of obtaining a better understanding of representations, we develop a family of methods to investigate CNNs and other image features by means of visualizations
there is no clear understanding of why they perform so well, or how they might be improved
there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance

Although I like the images, I don't see how these methods are better than simply pushing all images through the network and showing the top-$n$ images which activate the neuron of interest most. Was this evaluated? Did the authors have any insights into the features which other authors didn't have before / without those techniques?

(The Zeiler&Fergus paper at least added the occlusion sensitivity analysis which does help. However, a big part of the paper is this filter visualization by deconv-nets. And I don't see how this helps to address any of the points mentioned above)


texts: How to supply validation set for keras multitask learner?

texts:
I have a multitask classifier with two outputs. The model is defined as follows:
def create_model(shape,
                activation1='relu',
                layer1=256,
                layer2=512,
                dropout1=0.3,
                weight1=1.,
                weight2=1.,
                lr = .001):
    # define model
    inputs = Input(shape=shape, name='main_input')
    main_branch = Dense(layer1, activation=activation1)(inputs)
    main_branch = Dropout(dropout1)(main_branch)
    main_branch = Dense(layer2, activation=activation1)(main_branch)
    nrar_branch = Dense(1, activation='sigmoid', name='nrar_output')(main_branch)
    nrahr_branch = Dense(1, activation='sigmoid', name='nrahr_output')(main_branch)

    model = tf.keras.Model(inputs=inputs, outputs=[nrar_branch, nrahr_branch])
    # compile model
    model.compile(loss={'nrar_output': masked_loss_function,
                        'nrahr_output': masked_loss_function},
                  loss_weights={'nrar_output': weight1,
                                'nrahr_output': weight2},
                  optimizer=tf.optimizers.Adam(learning_rate=lr),
                  metrics=[tf.keras.metrics.AUC(curve="PR")]
                 )
    return model

and loss:
def masked_loss_function(y_true, y_pred):
        mask_value = -1
        mask = K.cast(K.not_equal(y_true, mask_value), K.floatx())
        return K.binary_crossentropy(y_true * mask, y_pred * mask)

I want to provide validation data in the model.fit method, but I'm not sure how to do this. Right now I am trying
model.fit({'main_input': X_t},
              {'nrar_output': y_t['NR.AR'], 'nrahr_output': y_t['NR.AhR']},
              epochs=100, batch_size=256,
              verbose=1,
              callbacks=[PlotProgress(entity='auc', num=129)],
              validation_data=(X_val, (y_val['NR.AR'], y_val['NR.AhR']) )
             )

However, I'm not sure if this is working as I expect it to. The learning curves look reasonable but I'm still not sure. Is this functionality supported and if so is this the correct way to achieve it?


texts: is it possible to implement LSTM with input shape (sample,timestep,timestep,feature)?

texts: I'm new to Keras. I am trying to implement this model https://www.aclweb.org/anthology/D15-1167 for document classification, and I want to use LSTM for getting sentence representation. I have trained vector representation separately with the skip-gram model on my dataset. now after converting each document to separate sentence and then converting each sentence to separate word and then converting each word to the corresponding integer in the dictionary, I have something for example like this for each document: [[54,32,13],[21,43,2]...[28,1,9]] which I should feed each sentence to an LSTM to get a sentence vector and after that I should feed each sentence vector to a diffrent LSTM on the higher layer in order to get a document representation and then apply classification to it. my problem is in the first layer. how should I feed each sentence simultaneously to each LSTM (therefore at each time step each LSTM should be applied to a word vector from each sentence) ?


texts:
I have a table with this features:
   'tourney_id', 'tourney_name', 'surface', 'draw_size', 'tourney_level',
   'tourney_date', 'match_num', 'winner_id', 'winner_seed', 'winner_entry',
   'winner_name', 'winner_hand', 'winner_ht', 'winner_ioc', 'winner_age',
   'winner_rank', 'winner_rank_points', 'loser_id', 'loser_seed',
   'loser_entry', 'loser_name', 'loser_hand', 'loser_ht', 'loser_ioc',
   'loser_age', 'loser_rank', 'loser_rank_points', 'score', 'best_of',
   'round', 'minutes', 'w_ace', 'w_df', 'w_svpt', 'w_1stIn', 'w_1stWon',
   'w_2ndWon', 'w_SvGms', 'w_bpSaved', 'w_bpFaced', 'l_ace', 'l_df',
   'l_svpt', 'l_1stIn', 'l_1stWon', 'l_2ndWon', 'l_SvGms', 'l_bpSaved',
   'l_bpFaced'

You can see that they are duplicate features with the difference that one half is for winners e.i winner_age and the other half for losers e.i loser_age. I not sure which approach to take as is not a classification problem in the sense that I don't have clear binary labels (of course I could categorize some of them) or regression because I don't have a clear target (each feature is both winner and loser). I'm trying to do some EDA first but in order to do it I have to have a clear understanding on which approach to take to start with in the first place.


texts: How to calculate latent vector for a user in ALS based on some new input?

texts:
So I have an ALS trained in pyspark but then I get some interactions from a new user that wasn't in the training set. I want to give recommendations to that new user without retraining the ALS based on their ratings. I've found some information here, but as I need it for explicit ALS I've found that I need to calculate this:

x_u is the user's latent vector, r_u contains the user's ratings and Y is the ALS model's item factor matrix.

This is the code I've come up with:

def calculate_user_latent_vector(R_u, Y, rank, reg):
    R_u_Y = R_u @ Y
    term = Y @ Y.T + reg * np.eye(rank)
    x_ut = R_u_Y @ term
    x_u = np.squeeze(np.asarray(x_ut))
    return x_u

But this doesn't work. When I created R_u I made a vector that is the length of the number of items and R_u[i] is the rating that the new user has given to item i or 0 otherwise.

After some thinking I tried another solution using np.linalg.lstsq method.

def calculate_user_latent_vector_lstsq(R_u, Y):
    A = np.vstack([Y[i] for i in range(len(R_u)) if R_u[i] != 0])
    b = np.array([r for r in R_u if r != 0])
    return np.linalg.lstsq(A, b, rcond=None)[0]

Here I only optimize for items that were given a rating and don't include 0 ratings for unknown ones. This works much better and if I try it on data that matches some user's interaction who was in the training set then the latent vector almost matches the one in the ALS's user factor matrix. (no regularization in my method that's why they aren't exactly the same)

I feel like the two methods are kinda the same thing mathematically but from a bit different perspective and that I shouldn't use 0 ratings where the ratings are unkwown in the first method. But I can't come up with any solution to fix this.
The second approach works but I don't really know how to formalize it, and I also want to include regularization in it, but I don't know how.


texts: Principal components analysis with compositional data

texts:
Another beginner question: I'm trying to do PCA on compositional data. In other words, all the variables in the group add up to 100%.

I've since learned on this forum that compositional data poses a problem for linear regression. Will compositional data also pose a unique problem when doing PCA?


texts:
I am new to NLP, I found a format named ConLL which seems a tab-separated file,
like

ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL

I found a dependency Treebank in this format for Persian. I would like to do some experiment on it. I am familiar with C# and C# has great libraries for XML document files.

I just wonder why such a corpus is not in XML format! I don't say a sophisticated one but they could put each line in a node of XML. Then at least, I would know what is the tag for each element.

However, I can convert it to XML.


texts: Lack of activation function in output layer at regression?

texts: I supposed that the output layer should have certain kind of activation function (preferably linear or tanh) for regression, but I recently read that in case of regression this is not necessary. This seems me reasonable, after all, the node(s) of the output layer produce(s) numeric values themselves. Which solution is better: with or without activation func?


texts: Similarity of perceptron criterion and SVM

texts:
In the book "Neural Networks and Deep Learning" by Aggarwal there is an exercise 2.10.1:

  Consider the following loss function for training pair $(\overline{X},y)$:
  $$L=max(0, a -y(\overline{W} \cdot \overline{X}))$$
  The test instances are predicted as $\hat{y}=sign(\overline{W} \cdot \overline{X})$. A value of $a=0$ corresponds to the perceptron criterion and a value of $a=1$ corresponds to the SVM. Show that any value of $a>0$ leads to the SVM with an unchanged optimal solution when no regularization is used.

I am trying to solve this exercise and from the previous chapters of the book I drew a conclusion that the stochastic update for such a loss function will be $$\overline{W} \Leftarrow \overline{W} + \alpha y \overline{X} [I(y \hat{y} < a)]$$
so it's different from original SVM for cases where $y \hat{y} \in (a,1)$.  For this range, the weights in original SVM will be updated, but for "modified" SVM the wegiths will not be updated. So I don't see a reason why the "modified" SVM (with $a$ instead of $1$) should be equal to the original SVM. How to solve this exercise?


texts:
Well, I post the same question in the main stack before finding the right place, sorry.

A friend of mine is working with more than a 100 videos as sample for his neural network. Each video last more than a couple of minutes with around 24 frames per second. The objective, using deep learning, is to detect movement through all the samples.

The problem for him is the quantity of data he is dealing with. The training part require/consumes too much time. I'm no expert with data preparation, but I thought maybe he could turn all frame into dataframe, clean them from mono color image (full black/white), turn them into gray instead of full rgb and compress them but, I'm not sure if it will be enough.

Do you think of better method to reduce the training sample?


texts: How do I space out the grouped bar plots in plotly?

texts:
I have the following graph that I created using plotly but I would like to space the grouped bar plots (based on quarter) a little bit apart.

This is the code I used:
df = pd.melt(billed_rev['EMEA'][7:13], id_vars=['Fiscal Year/Period'],value_vars=['Val1','Val2','Val3','Val4','Val5'])
fig = go.Figure()

fig.update_layout(
    template="simple_white",
    xaxis=dict(title_text="Quarter"),
    yaxis=dict(title_text="Revenue"),
    barmode="group",
    bargap=0.2,
    bargroupgap = 0.0)

colors = ["darkgoldenrod", "darkblue","darkcyan","olive","lightblue"]

for r, c in zip(df['variable'].unique(), colors):
    plot_df = df[df['variable'] == r]
    fig.add_trace(
        go.Bar(x=[plot_df['Fiscal Year/Period'], plot_df['variable']], y=plot_df['value'], name=r, marker_color=c,text=plot_df['value'],
        textposition='outside'),
    )

fig.show()

How do I space these out?


texts: Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

texts:
I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification.
I applied cv (5 iteration loop) mean target encoding on the categorical variables and i got a certain accuracy. Ordinal encoding of the categorical features is giving a better accuracy than the mean encoding. How is that possible? If my understanding is correct, mean target encoding does not only numerically encode 'object'-type variables but it orders them using their impact on the target value, and the difference between the numerically-encoded new variables is also based on the categories' impact on the target  Why do GBDT's perform better on a randomly encoded variable rather than a "well-encoded" one? Over-fitting ? or do GBDT's ( catboost/xgboost ) handle the ordinal encoding well enough that mean encoding is not needed? or something else?

Here's how i'm doing cross-validation mean encoding with a smoothing value of alpha = 10

Edit : Got a slightly better result by increasing the smoothing value to 30, but mean-encoding is still underperforming compared to the ordinal one.

   ## inside the loop
   means1 =  X_val.groupby(column1).bank_account.agg('mean')
   nrows1 = X_val.groupby(column1).size()
   score1 = (np.multiply(means1,nrows1)  + globalmean*alpha) / (nrows1+alpha)
   X_val.loc[:,encoded_column1] = X_val[column1]
   X_val.loc[:,encoded_column1] = X_val[column1].map(score1)
## After the loop is over, i average the encodings for each category across all folds and update the value for my new encoded column
meanz1 = train_new.groupby(column1)[encoded_column1].mean()
train_new[encoded_column1] = train_new[column1].map(meanz1).copy()


texts: Feed forward neural network, output as list of targets and associated probabilities

texts:
I am working through an FNN tutorial, right now it outputs a sigmoid probability from 0-1 (0.8956 for example). My own data has 3+ possible targets so i need the output to be a list of the targets and the associated probability for testing on new samples. (In this instance it would be: 0 (probability), 1 (probability)). How would i go about this? Here is the code. Thanks.

import numpy as np

# sigmoid function
def nonlin(x,deriv=False):
    if(deriv==True):
    return x*(1-x)
    return 1/(1+np.exp(-x))

# input dataset
X = np.array([  [0,0,1],
                [0,1,1],
                [1,0,1],
                [1,1,1] ])

# output dataset
y = np.array([[0,0,1,1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
syn0 = 2*np.random.random((3,1)) - 1

for iter in xrange(10000):

# forward propagation
l0 = X
l1 = nonlin(np.dot(l0,syn0))

# how much did we miss?
l1_error = y - l1

# multiply how much we missed by the
# slope of the sigmoid at the values in l1
l1_delta = l1_error * nonlin(l1,True)

# update weights
syn0 += np.dot(l0.T,l1_delta)

print "Output After Training:"
print l1


texts:
So i started building my own neural network framework in node.js, just to understand the concept of neural networks better.

Currently i implemented Neurons, Connections between these Neurons and Layers.

Given this network:

N3s output for example, would be Sigmoid((N1 * W1 + N2 * W4) + B) where B is a randomly generated Bias between -1 and 1.

To train the network, a dataset which contains the inputs an their desired output is used. The network will be fed the inputs (Every input getting assigned to one Neuron in the first layer, for example [1, 0] would result in N1 beeing 1 and N2 beeing 0). Then for each Connection, the weight will be increased or decreased by 1% if the resulting output of the network would be closed to the desired one.

For example if 1 is the desired output and the current output would be 0.5, the output with the weight increased would be 0.37 and the output with the weight decreased would be 0.46. The weight would be increased.

Same thing is done for every Neurons Bias.

My Problem:

If i train above network with the dataset:

input: 0, 0
desired Output: 0

input: 0, 1
desired Output: 1

input: 1, 0
desired Output: 1

input: 1, 1
desired Output: 0

for 10000 iterations, it will give me something similar to the following output:

[ 0, 0 ] => [ 0.13500189926566186 ]
[ 0, 1 ] => [ 0.14749124528600177 ]
[ 1, 0 ] => [ 0.14238677719564685 ]
[ 1, 1 ] => [ 0.15412548826116046 ]

I am kinda new to Neural Networks, so sorry if this is a dumb question.


texts: Deep Q Network gives same Q values and doesn't improve

texts:
I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit of debugging, I figured out that the Q values the network predicts are the same every time. The action space is [up, right, down, left] and the network predicts [0, 0, 1, 0]. The training loss does go down over time, but it doesn't seem to make a difference. Here's the training code:

def train(self):
    tf.logging.set_verbosity(tf.logging.ERROR)
    self.build_model()
    for episode in range(self.max_episodes):
        self.current_episode = episode
        env = SnakeEnv(self.screen)
        episode_reward = 0
        for timestep in range(self.max_steps):
            env.render(self.screen)
            state = self.screenshot()
            #state = env.get_state()
            action = None
            epsilon = self.current_eps
            if epsilon > random.random():
                action = np.random.choice(env.action_space) #explore
            else:
                values = self.policy_model.predict(state) #exploit
                action = np.argmax(values)
            experience = env.step(action)
            if(experience['done'] == True):
                episode_reward += experience['reward']
                break
            episode_reward += experience['reward']
            self.push_memory(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
            self.decay_epsilon(episode)
            if self.can_sample_memory():
                memory_sample = self.sample_memory()
                X = []
                Y = []
                for memory in memory_sample:
                    memstate = memory.state
                    action = memory.action
                    next_state = memory.next_state
                    reward = memory.reward
                    max_q = reward + (self.discount_rate * self.replay_model.predict(next_state)) #bellman equation
                    X.append(memstate)
                    Y.append(max_q)
                X = np.array(X)
                X = X.reshape([-1, 600, 600, 2])
                Y = np.array(Y)
                Y = Y.reshape([self.batch_size, 4])
                self.policy_model.fit(X, Y)
        food_eaten = experience["food_eaten"]
        print("Episode: ", episode, " Total Reward: ", episode_reward, " Food Eaten: ", food_eaten)
        if episode % self.target_update == 0:
            self.replay_model.set_weights(self.policy_model.get_weights())
    self.policy_model.save_weights('weights.hdf5')
    pygame.quit()

Here's the network architecture:

    self.policy_model = Sequential()
    self.policy_model.add(Conv2D(8, (5, 5), padding = 'same', activation = 'relu', data_format = "channels_last", input_shape = (600, 600, 2)))
    self.policy_model.add(Conv2D(16, (5, 5), padding="same", activation="relu"))
    self.policy_model.add(Conv2D(32, (5, 5), padding="same", activation="relu"))
    self.policy_model.add(Flatten())
    self.policy_model.add(Dense(16, activation = "relu"))
    self.policy_model.add(Dense(5, activation = "softmax"))
    rms = keras.optimizers.RMSprop(lr = self.learning_rate)
    self.policy_model.compile(optimizer = rms, loss = 'mean_squared_error')

Here are the hyperparameters:

learning_rate = 1e-4
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = 1e-5
memory_size = 100000
batch_size = 2
max_episodes = 1000
max_steps = 100000
target_update = 100

I've let it train for the full 1000 episodes and it's pretty bad at the end. Am I doing something wrong with the training algorithm?

EDIT: Forgot to mention that the agent receives a reward of 0.5 for going towards the food, 1 for eating the food, and -1 for dying

EDIT 2: Added updated code for models and hyperparameters


texts: Analytic Hierarchy Process - consistency ratio over 50%

texts:
I have such data of pair-wise comparison. I calculated consistency ratio for this data of 56%. That's too high for relevant results.

Whether is possible to modify initial data to improve consistency if I have no other sources what factor should I fix?


texts: Data scaling before PCA: how to deal with categorical values?

texts:
I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal with them (basically I created dummy variables). Now, in order to apply PCA I have to scale the data matrix such that I have mean equal to 0.
My question is: does it make sense to normalize also the categorical values (which now are numbers, but they are actually categorical values)? I think it doesn't, but in that case how do I proceed? I do the PCA without scaling these variables?

Thanks!


texts: GAN - am I seeing mode collapse? Common fixes not working

texts:
I have a 2 part question.

Context
I am learning about GANs and writing my own starting from the very simplest example of adversarial learning (1-parameter node), then implementing a very simple 1-dimensional pattern (1010) learning GAN .. and now I am trying to implement an MNIST learning GAN, before I proceed to more realistic photos.

I have some background in machine learning and data mining (masters from a long time ago) and moderately understand how neural networks work.

You can read about my progress on the initial steps here:

GANs - Part 1
GANs - Part 2

I've read lots of the latest blogs, articles, and watched youtube tutorials but still can't get past 2 key problems with implementing MNIST learning GANS.

Q 1. Am I seeing mode collapse?

After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.

You can see a recent iteration of my pytorch code here: github notebook.

When fed random noise, the trained GAN always generates the same, or extremely similar image. Feeding it 1-shot noise (00001000...) also generates a similar image.

Are GANs supposed to generate just one image? Are the different images we see reported from separate training of the GAN? I thought the idea was a single trained GAN could generate many different images from random input noise. Have I misunderstood it?

Q 2. How To Escape Mode-Collapse?

If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.

I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:

with / without batch normalization
with / without maxpooling
with / without dropout
with / without label softening
with / without noise added to both input and target labels, including noise that decays over training time
various widths and depths for the generator, less so the discriminator
increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)

What I've observed or noted is:

the discriminator width/depth/architecture is tested first to ensure it has the capacity to learn the multi-class MNIST before it is used in the GAN to avoid the case of a discriminator that can't actually learn to discriminate MNIST
plotting the error as training proceeds (D error, D error on G input) is useful to show that the D error approaches 1/2 as it should, and that D error on G input approaches 1 (well more like 0.8).
plotting errors also shows stability or collapse of some sort which helps tune the learning rates for example
I would have thought adding noise to the input or the target labels might kick the GAN out of any local minima which is mode-collapse but I suspect the theory is more complicated than this
trying different suggestions for the optimiser parameters doesn't help, I have to find my own optimal tuning, and the learning rates are much lower than what others use, eg Adam lr=0.00002 not 0.001 which causes instability
higher training epochs result in high contrast images which don't look like the MNIST data which has softer edged strokes .. I was hoping higher training epochs would improve diversity of outputs

One area where I can't find much guidance is on the actual architecture of the discriminator and generator:

Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
The use of deconvolution is common in the generator, but some examples online use simple fully connected mapping. Computationally the deconvolutions have fewer learning parameters and intuitively make sense in building an image.

I'd welcome your thoughts and suggestions.

Example output which I think looks good, but is mode-collapsed:


texts: Master of Statistics or Master of Data Science

texts: Many Universities now offer Masters Degrees in data science; I'm curious as to people's opinions on whether a traditional master of statistics is a better to fit for working in the field of in data science.


texts: Suggest DataSet for Sentiment Analysis on Dutch languages

texts: Can anybody provide any links for any training dataset for sentiment analysis on Dutch languages? I googled trying to find maybe movie review on Dutch but didn't find website with persons review like in IMDB


texts: Using ARIMA parameters when transforming time series to Supervised Learning

texts:
When forecasting time series one can change the problem from a classical time series (ARIMA type of models) to supervised learning (by adding lag features).

When the time series is long and you convert it to supervised learning you end up having a lot of features.

In case that we had done previous modeling with ARIMA and found the AR, I, MA, parameters.

Could we use this knowledge/parameters to select a subset of lag features?


texts: What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?

texts:
For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.

Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.

If label ordering is [apple, pear, big, yellow] then a “big yellow apple” picture should have the label [1,0,1,1]. First two part is mutually exclusive, one-hot, however last two are not.

So, what loss function should I use for this problem?


texts: I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually I have to add rows and cols. Trying to select the best for my problem, I want to understand which are the best cases to use each of this types: lil_matrix, coo_matrix, csr_matrix, csc_matrix, dok_matrix. Can someone explain me? Its not necessary to show examples of all the types in the same answer.


texts:
Cross-validation
Hi, I'm deploying machine learning models in my MSc thesis using Weka. I have noticed that when I use 10-fold cross-validation in the training dataset I get low evaluation metrics compared to training the model on the entire dataset without cross-validation, this can make sense if I get better performance for the model when I test it on the testing dataset, however, it keeps the same performance. For example, I trained a random forest model for a regression problem and these are the results I have got:
1. Without cross-validation:
Training:
R = 0.97; Mae = 1.31; Rmse = 1.78
Testing:
R = 0.91; Mae = 2.70; Rmse = 3.57
2. With 10-fold cross-validation:
Training:
R = 0.76; Mae = 3.55; Rmse = 4.77
Testing:
R = 0.91; Mae = 2.70; Rmse = 3.57

As you can see the performance of the model on the testing dataset is the same, can someone explain to me what is the purpose of using it if the model is the same in either case?


texts: Classify documents using a set of known vocabularies

texts:
I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents).

One way I am thinking about is to go online and search for the most popular words in soccer articles to make a list of vocabularies (for example: score, shoot, World Cup, etc). Then somehow use that list of vocabularies to classify the documents (maybe if a particular contains 30% of the words in that list of vocabularies, then that document talks about soccer).

I am wondering whether it is a valid method or there are better existing methods. Really appreciate any help.


texts: How to do feature reduction for a log-linear regression model

texts: I'm building a log-linear regression model and I have 18 different variables in my model. 13 out of 18 variables I'm using are hot-encoded variables for holiday, e.g. showing which holiday it is. I think 18 variable for a log-linear model are a lot and I'd like to reduce them. I'm thinking of applying feature reduction, e.g. PCA or Autoencoder, on the holiday flags and convert them to 3 or 4 features. Does this makes sense or any better way or dealing with these number of features in my model?


texts: So I've been trying to iterate over rows of my dataframe and my goal is to find matching rows based on two particular columns (say C and P). Then i need to do some manipulations as well in the data for the rows. I've read quite a few answers here telling to use iterrows() or itertuples() but that doesnt serve my purpose because I cannot manipulate my data using them. Same goes for functions like groupby since it only allows manipulation on the whole groups not elements of those groups(correct me if I am wrong here because thats what I have seen in the articles on groupby(). What approach should I use to match rows in my data frame to each other based on columns and then manipulate them.


texts: The problem of K-Means with non-convex function

texts:
In all articles I've read say "K-means is not converge with non-convex function".I just know that convex function is the function that able to find the global minimum. However , I do not understand how this relate to K-Means algorithm.

I need to ask

Which situation cause the non-convex problem ? (If possible please visualize it and show the equation)
In some articles say that the bad initialization( the poor seeds) cause the not-convex problem, How are they related?
I'm not quite sure the referred function is subject to which variables.Are that clusters and their centroid ?

Ok , It's my misunderstanding. K-Means is not converged with non-convex shape data set .Nothing relate with convex function,sorry.


texts: Advantages of Binary Rating System for Collaborative Filtering Recommender Systems

texts:
I notice that Netflix, which I think used to use a five-star scale for rating content and give predicted ratings for unrated content on the same scale, now just has basic like/dislike buttons.  Music streaming services like Pandora and Spotify seem to employ the same approach.  My question is what makes a binary response superior to a likert-type scale when it comes to generating the best recommendations?

Is it related to the nature of the algorithms (e.g., classifications are more accurate when there are fewer possible responses)? Or is the advantage more to do with human psychology (easier for users to accurately choose between like/dislike as opposed to love/like/neutral/dislike/hate)?


texts:
I'm trying to find a way to predict/calculate how a shape (e.g. outline of a glacier) will change in the future—based on its history (previous shape) and additional factors (e.g. Δtemperature).

In my example: I have the shape/coordinates of a glacier and an average temperature at 1970, 1985, 2000, 2015. How can I give an estimate on how that shape will look like in 2030, based on the previous shapes and a predicted temperature?

The shapes would ideally come in a format similar to this:

[
  [
    [-113.74926783355818, 48.32440832757374],
    [-113.74926767330584, 48.32440813255405],
    [-113.74926748239692, 48.32440458296232],
    [-113.74926717549286, 48.32439887665663],
    [-113.74926686858888, 48.32439317035087],
    ...
  ],
  [
    [-113.75763099733634, 48.32877681033601],
    [-113.75763955455557, 48.32877660502744],
    [-113.75764546550784, 48.32878027200765],
    [-113.75765402272751, 48.32878006669797],
    [-113.75766003497893, 48.32878563609638],
    ...
  ]
]

But I imagine it could be necessary to convert this to coordinate offsets (Δx, Δy) similar to what sketch_rnn is doing.

Any pointers would help. Even that this is too ambitious. Thanks!

Disclaimer: I'm a beginner with this and hope the question isn't too naïve :)


texts: Modeling when the response variable has too many 0's and few continuous values?

texts:
For problems where the data represents online fraud or insurance (where each row represents a transaction), it is typical for the response variable to denote the value of fraud committed in dollars. Such a response value might have less than 5% non-zero values denoting fraudulent transactions.

I have two questions regarding such a dataset:

What algorithms can we use to ensure that the model not only predicts the fraudulent transactions accurately, but also predicts the value of fraud associated with these.
Assuming that we can quantify the cost involved in each false positive (tagging a non-fraudulent transaction as fraudulent) and cost incurred due to a false negative (tagging a fraudulent transaction as non-fraudulent), how can we optimize the model to maximize savings (or minimize losses)?


texts:
Let's say I have a dataset like thisX<-matrix(rnorm(30), nrow=100, ncol=6). I am trying to find a way to apply the sum function to every four rows of column 3. This means I want to sum rows 1-4, rows 4-8 and so on.

Are there any functions such as apply or lapply I could use?


texts: Plot dataframe with two columns on the x axis

texts: how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, should I combine them first?


texts: Machine learning model bundled with a library vs. an API

texts:
I am thinking to "deploy" a machine learning model (in pickle it is sized 3 megabytes) and after discussing with my developer colleagues, they said it would be better if the model is packed as a python library instead of a microservice (like a rest API).

I wanted to ask what's your view on this: Pickled model packed in a library specifically meant for it vs. a rest API, pros and cons?

I was thinking that having it as a library could possibly be easier to use and wouldn't require to have to worry about deployment, web address etc.


texts: Evaluating Model Accuracy on a testing data set for a DecisionTreeReegressor Model

texts:
I am trying an exercise where I have been asked to "Evaluate each model accuracy on testing data set for a max_depth parameter value changing from 2 to 5".

The model here is DecisionTreeRegressor. I just wanted to know what is the metric for calculating the Accuracy for a DecisionTreeRegressor model.

My understanding is that it's same as Score which can be calculated simply as regressor.score(X_test, Y_test)

Please let me know what should be used to calculate the Accuracy of the DecisionTreeRegressor Model.


texts: Does partial transfer learning require a lot of computer power?

texts:
I want to be sure my understanding of the problem is correct.
I want to do image classification and current state of the art in my field is achieved by transfer learning with VGG16.

Since image on this field are totaly different than image used by VGG16 trainig image, we should unfreeze "a lot" of layer.

Original VGG16 model has more than 130M parameter. it's has something like 13 hidden layer and 3 fully connected layer at the end.
If I unfreeze only the fully connected layer i'll still have more than 80M parameter to train!! and i'll probably need to unfreeze some hidden layer to get better performance since my data are really different

So it's look like transfer learning with VGG16 require a lot of computation power

am I right?
Is it possible to use transfer learning with image really different than training data without a lot of computational power?


texts:
So much of what we export is in CSV and JSON files.

Is there any useful tools you know of that can automatically perform data analysis on flat file formats. For example

Basic statistics if numeric types: avg, stdev, mode, median
Column type and Cardinality detection
Find relationships between columns, if column A is X, then column B is always Y

Any of these things would be useful, even if there'd be an intermediary step of just loading the flat file into some software...


texts: SKLearn - Understanding Discrepancy Between LogisticRegressionCV classification_report and scores_

texts:
Cross-posting from Stack Overflow:
I'm running into a weird situation where my sklearn LogisticRegressionCV model is apparently getting 100% accuracy (the lack of shuffling is intentional). However, when I ask the model to report its accuracy scores, the accuracy scores per inverse regularization are much lower than 100%. What am I doing wrong?
The model:

    decoder = LogisticRegressionCV(
        penalty='l1',                       # want sparse parameters
        cv=KFold(n_splits=5, shuffle=False),
        Cs=np.logspace(-1, 2, 3),
        random_state=0,                     # reproducibility
        dual=False,                         # Prefer dual=False when n_samples > n_features.
        solver='liblinear',                 # for l1 penalty, must use liblinear or saga
        max_iter=1e5,                       # give plenty of time to get good performance
        fit_intercept=False,
        class_weight='balanced',
    ).fit(X, y)

    y_hat = decoder.predict(X)

    y_hat_probs = decoder.predict_proba(X)

    print(classification_report(y, y_hat))
    print(confusion_matrix(y, y_hat))

outputs
              precision    recall  f1-score   support

         0.0       1.00      1.00      1.00       374
         1.0       1.00      1.00      1.00       105

    accuracy                           1.00       479
   macro avg       1.00      1.00      1.00       479
weighted avg       1.00      1.00      1.00       479

[[374   0]
 [  0 105]]

Asking for decoder.scores_ yields values nowhere near 100%:
{1.0: array([[0.35416667, 0.60416667, 0.61458333],
        [0.64583333, 0.64583333, 0.65625   ],
        [0.59375   , 0.65625   , 0.64583333],
        [0.54166667, 0.57291667, 0.59375   ],
        [0.17894737, 0.61052632, 0.65263158]])}

Why is there this discrepancy?