texts: Feature selection for nonlinear regression on high dimensional data set

texts: Can anyone tell that if there are any feature selection techniques for non linear regression?


texts:
I have downloaded a dataset from Amazon. http://jmcauley.ucsd.edu/data/amazon/ Dataset involves feature vectors of images. There are around 1.5 M feature vectors.

Dataset consists of 10 characters (the product ID), followed by 4096 floats (repeated for every product).

Every product image involves feature vectors with (4096x1) size. Feature vectors involve float numbers.

What do these float numbers mean?

What I understood is, there are at total 4096 features, and each index of feature vectors indicate a specific feature. The values in feature vectors indicate the frequency of regarding feature in all specific image.

Is it so? Or, if it is not, what might be the right explanation?

Thanks,


texts: What are the implications for training a Tree Ensemble with highly biased datasets?

texts:
I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on this data but it's difficult to create training datasets that contain enough examples of the positive class.

What would be the implications of doing a stratified sampling approach to normalize the number of positive and negative examples? In other words, is it a bad idea to, for instance, artificially inflate (by resampling) the number of positive class examples in the training set?


texts:
I'm trying to train a model which in my opinion is taking too long compared to other datasets given that it's taking about 9s to complete a step. I think that the problem is because the dataset is not being stored on ram, but I'm not sure of this.

The code is the following:

def load_data():

    train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
    train_generator = train_datagen.flow_from_directory(path1, target_size=(200, 200), batch_size=32, class_mode="binary")

    test_datagen = ImageDataGenerator(rescale=1./255)
    test_generator = test_datagen.flow_from_directory(path2, target_size=(200, 200),batch_size=32, class_mode="binary")

    return train_generator, test_generator

Model:

Sequential model
2 Convolutional layers with 32 neurons, activation = relu.
1 Convolutional layer with 64 neurons, activation = relu.
Flattening and Dense layer, activation = relu.
Dropout of 0.5
Output layer (Dense) with sigmoid activation.
Adam optimizer.
Loss: binary cross entropy.

Fit:

model.fit_generator(x, steps_per_epoch=37, epochs=50, validation_data=y, validation_steps=3, callbacks=[tensorboard])

My dataset has 1201 images and 2 classes.
I built the model following this tutorial.
My GPU is a GTX 1060 3gb.
8gb of ram.
The images are being reshaped to 200x200.

If you could help me I'd appreciate it. Thank you very much!


texts: Binning data in one of the columns of a dataframe(Using R)

texts:
I have a data frame that contains random values between 0-60 (inclusive 0 & 60). These values denote months. I want to bin the data into three categories
(x<=6, 6< x <=12, x>12) and generate a new single columns which will be a factor containing 3 values (0,1,2) denoting the respective bins.

I am able to generate 3 columns one-hot encoded style but I am unable to think of a way to generate single column having 3 factors.


texts:
I have database of sequential events for multiple animals.
The events are represented by integers so it looks something like:
Animal A: [1,6,4,2,5,7,8]
Animal B: [1,6,5,4,1,6,7]
Animal C: [5,4,2,1,6,4,3]

I can manually see that for each event 6 event 1 first happens.
And event 4 happens quickly after a 1,6 combination.
But these are easy to spot in such a small dataset, the real lists are 10000+ events per animal.
Is there a way to use an algorithm or machine learning to search for these kinds of patterns?


texts: Intractability in Variational Autoencoders

texts:
I'm having difficulty understanding when integrals are intractable in variational inference problems.
In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize data likelihood $p_\theta (x) = \prod_{i=1}^N p_\theta (x_i)$ which is
$p_\theta (x) = \int p_{\theta_1} (z) p_{\theta_2} (x|z) dz$
We know $p_{\theta_1} (z)$ (usually a Gaussian with mean $\mu$ and covariance matrix $\Sigma$), and we also know $p_{\theta_2} (x|z)$ which is usually a Gaussian distribution with mean $\mu_z$ and covariance matrix $\Sigma_z$ modeled as a neural network. We want to maximize the likelihood with respect to $\theta = \{\theta_1, \theta_2\}$. I understand that it is not possible to analytically compute this integral to optimize for $\theta$. It is possible to obtain a sample-based approximation of the integral for a stochastic gradient ascent update. Denote by $z_l$ samples from $p(z)$ and write log likelihood to optimize:
$\nabla_{\theta_1} \log p_\theta (x_i) =  \frac{\int \nabla_{\theta_1} p_{\theta_1}(z) p_{\theta_2}(x_i|z)dz}{\int p_{\theta_1}(z) p_{\theta_2}(x_i|z)dz} \approx \frac{\frac{1}{L} \sum_{l=1}^L \nabla_{\theta_1} \log p_{\theta_1}(z_l)}{\frac{1}{L} \sum_{l=1}^L p_{\theta_2}(x_i|z_l)}$
where we used the following general property: $p_\theta (x) \nabla_\theta \log p_\theta (x) = \nabla_\theta p_\theta (x)$. As I understand, we don't like this approach because it requires sampling $L$ samples $z_l$ for every observation which is inefficient. I'm confused to see that this kind of sampling is still used later.
Now, we assume to have an encoder network $q_\phi (z|x)$ (again typically a Gaussian) and write the following objective:
$E_{z \sim q_\phi(z|x)} \big[ \log p_{\theta_2} (x|z) \big] - D_{KL}(q_\phi(z|x)||p_{\theta_1}(z))$.
In the paper, authors say that it is possible to calculate gradient updates for $\theta$ and they use a reparametrization trick to calculate updates for $\phi$. I tried to derive the updates for $\theta$ below. To optimize with respect to $\theta_1$ we can sample $z_l$ from $q_\phi(z|x_i)$
$\nabla_{\theta_1} \Big(\int q_\phi(z|x_i) [\log p_{\theta_1} (z) + \log p_{\theta_2} (x_i|z)] dz \Big) = \int q_\phi(z|x_i) \frac{\nabla_{\theta_1}p_{\theta_1}(z)} {p_{\theta_1}(z)} \approx \frac{1}{L} \sum_{l=1}^L \frac{\nabla_{\theta_1}p_{\theta_1}(z_l)} {p_{\theta_1}(z_l)}$
and similarly for $\theta_2$
$\nabla_{\theta_2} \Big(\int q_\phi(z|x_i) [\log p_{\theta_1} (z) + \log p_{\theta_2} (x_i|z)] dz \Big) = \int q_\phi(z|x_i) \frac{\nabla_{\theta_2}p_{\theta_2} (x_i|z)} {p_{\theta_2} (x_i|z)} \approx \frac{1}{L} \sum_{l=1}^L \frac{\nabla_{\theta_2}p_{\theta_2} (x_i|z_l)} {p_{\theta_2} (x_i|z_l)}$
I don't understand why the last two updates are ok but the original update is not. I'd appreciate it if you'd point out my mistakes/misinterpretation.


texts: Azure Data Studio opens Jupyter Notebook / Lab HTML

texts:
When I start Jupyter Lab or Notebook from the terminal, Azure Data Studio opens the auto-redirect HTML file instead of Firefox.

Is there a way to correct this behavior?  I'm experiencing this issue on Ubuntu 18.04 on my personal computer, but do not see this behavior on my work computer, which is a Macbook Pro.  Currently running Azure Data Studio 1.5.4.

jupyter                   1.0.0
jupyter_client            5.2.4
jupyter_console           6.0.0
jupyter_core              4.4.0
jupyterlab                0.35.3
jupyterlab_launcher       0.13.1
jupyterlab_server         0.2.0

EDIT: After starting the HTML in ADS, this is the output I see in the terminal:

[main 11:17:33 PM] update#setState idle
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell


texts: AWS EC2 Segmentation Fault (core dumped) issue with fast-style-transfer

texts:
I'm trying to utilize AWS EC2 p2.xlarge instance to convert images using style-transfer code given in this git repo:https://github.com/lengstrom/fast-style-transfer.git, yet when the input file becomes large, I keep running into this error:

2018-09-12 02:55:25.797741: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports
instructions that this TensorFlow binary was not compiled to use:
SSE4.1 SSE4.2 AVX AVX2 FMA
2018-09-12 02:55:25.797880: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread
pool with default inter op setting: 2. Tune using
inter_op_parallelism_threads for best performance.
Segmentation fault (core dumped)

This happens when the file size of the input is bigger than 130kb...yet works fine when the input is smaller than a certain size around 130kb. This defeats the purpose of using AWS, as any file of that small size would convert quite quickly on my computer too.

If anyone wants the files that I've used for testing, I could also upload it on the question.


texts: How does neural network solve XOR problem

texts:
I'm reading a wonderful tutorial about neural network. This is the best tutorial I've ever seen but I can't understand one thing as below:

In the link above, it is talking about how the neural work solves the XOR problem.

It says that we need two lines to separate the four points. But I don't know the second table.

XOR:

input1    input2    output
  0         0         0
  0         1         1
  1         0         1
  1         1         0

First table:

input1    input2    output
  0         0         0
  0         1         1
  1         0         1
  1         1         1

In my opinion, the first table is OK because it includes the XOR, which means that what the second table need to do is to remove the forth input. So I think the second table should be as below:

input1    input2    output
  0         0         1
  0         1         1
  1         0         1
  1         1         0

How in the link it says the second table is like this:

input1    input2    output
  0         0         0
  0         1         0
  1         0         0
  1         1         1

In a word, I can understand why the single layer neural network can't solve the XOR problem but I can't understand how the two layers neural network work to solve it.


texts: What is the difference between handcrafted and learned features

texts:
I am having difficulty understanding what the differences are between handcrafted and learned features.

Is it just the case that the handcrafted features are the input variables, and that the learned features would refer to the output variable? Or are they (handcrafted and learned) both types of input variables?

Can anyone help with this?

Thanks.


texts:
I am experimenting with Keras and I managed to build a simple CNN to classify blue images (300x300 images of the same shade of blue) vs. red images (same size, just red). This is a dummy problem I assumed the NN would have solved immediately but this doesn't seem the case. In fact, even after 20+ epochs accuracy is still exactly 50%.

I assume there's a lot of stuff I could try to do differently but is there anything extraordinary wrong I am doing here that could result in such poor performance on such an easy task?

# Create a Keras model.
model = keras.Sequential()
model.add(
    keras.layers.Conv2D(
        input_shape=(300, 300, 3),
        filters=64,
        kernel_size=(3, 3),
        activation='relu',
    )
)
model.add(
    keras.layers.Conv2D(
        filters=64,
        kernel_size=(3, 3),
        activation='relu',
    )
)
model.add(
    keras.layers.MaxPooling2D(
        pool_size=(2, 2),
        strides=(2, 2),
    )
)
model.add(keras.layers.Flatten())
model.add(
    keras.layers.Dense(
        units=128,
        activation='relu'
    )
)
model.add(keras.layers.Dropout(0.5))
model.add(
    keras.layers.Dense(
        units=1,
        activation='sigmoid',
    )
)

# Train the model.
sgd = keras.optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(
    optimizer=sgd,
    loss='mean_squared_error',
    metrics=['binary_accuracy']
)
model.fit(images, labels, epochs=20, batch_size=5)

Which outputs:

20/20 [==============================] - 18s 887ms/step - loss: 0.5954 - binary_accuracy: 0.4000
Epoch 2/20
20/20 [==============================] - 16s 794ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 3/20
20/20 [==============================] - 16s 781ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 4/20
20/20 [==============================] - 17s 853ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 5/20
20/20 [==============================] - 18s 877ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 6/20
20/20 [==============================] - 18s 891ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 7/20
20/20 [==============================] - 17s 825ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 8/20
20/20 [==============================] - 17s 861ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 9/20
20/20 [==============================] - 17s 846ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 10/20
20/20 [==============================] - 17s 835ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 11/20
20/20 [==============================] - 16s 800ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 12/20
20/20 [==============================] - 16s 806ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 13/20
20/20 [==============================] - 16s 811ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 14/20
20/20 [==============================] - 17s 827ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 15/20
20/20 [==============================] - 16s 806ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 16/20
20/20 [==============================] - 16s 786ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 17/20
20/20 [==============================] - 16s 795ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 18/20
20/20 [==============================] - 16s 796ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 19/20
20/20 [==============================] - 16s 788ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 20/20
20/20 [==============================] - 16s 794ms/step - loss: 0.5000 - binary_accuracy: 0.5000


texts: what is BIO Tags for creating custom NER Named entity recognization?

texts: I want to create custom NER Named entity recognition but im confused with this part of what is BIO Tags. Can anyone please explain the steps for creating NER and about this B, I, O tag.


texts: Combine two DataFrames column wise in Pandas

texts:
I have 2 Dataframes as shown and I want a new DataFrame where the 1st column is the 1st column of the 1st DataFrame and 2nd column from the 1st column of the 2nd DataFrame. I have tried pivot table etc but have no luck.

    A   B
0   1   2
1   3   67
2   4   54

    C   D
0   0   19
1   23  6
2   55  5

I need a new DataFrame like this.

    A   C   B   D
0   1   0   2   19
1   3   23  67  6
2   4   55  54  5


texts: How to assign a new level to many levels of a categorical variable

texts:
I have a model which has many categorical variables. For each categorical variable there are many levels, like 50~. But not all of them have significant counts. I got these counts using the function value_counts() in Python:

A                 50
B                 38
C                 26
D                 18
E                 10
...
T                 1
X                 1
Z                 1

How can I change the levels with count (say) less than 5 to a new level "others"?

for x in data.class:
    if x.value_counts() <30:
        x = "others"


texts: A few ideas to parse "events" from a text document

texts:
I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.

Input:

Output:

datetime;location;title;description
2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...
2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old

I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. => assume the nearby words are a date, etc.). But this would require lots of work for every different press release.

I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).

What general techniques do you see to address such a big (and difficult topic) concern ?
Are there open-source projects dedicated to that?
Some Python libraries that would make this task easier?
Any general idea?


texts: Semantic Segmentation using Keras: loss function and mask

texts:
I am about to start a project on semantic segmentation with a grayscale mask. The thing is, we have to detect for each pixel of the image if its an object or the background (binary class problem). I struggle to relate this pixel binary classification task with a mask labelling, since each pixel will be in a range [0,255]. I have started implementing an U-net with Keras according to this methodology (being fairly new with keras).

What kind of loss would you use? - I was thinking of binary_crossentropy
What kind of labelling would you use? And therefore, what would be the output shape of the CNN if I do this binary classification pixelwise?

Sorry if I do not use the proper technical terms.


texts: does smaller training set always lead to better training accuracy?

texts:
Does anyone know any existing research or have observed some experimental results in deep learning about the following:

For a fixed data set, if you subset the dataset from 50% to 90% as training set, using the same number of epochs, will the training error be bigger as the training set becomes bigger?

would be great if anyone could give some concrete experimental results or paper about this?
For example, this one https://arxiv.org/abs/1901.04169v1
and this one
https://www.researchgate.net/publication/333671691_Resampling-based_Assessment_of_Robustness_to_Distribution_Shift_for_Deep_Neural_Networks

but both papers have not drawn any conclusions on this issue.


texts: How to transform specific type feature to yield better prediction?

texts:
I have a dataset with 5K records focused on binary classification problem. I have about 60 features.

Out of 60 features, around 45-46 features are of 'Min' and 'Max' type.

For example, minimum blood pressure, maximum old pressure, minimum heart rate, maximum heart rate, minimum potassium, maximum potassium etc.

Several other vital parameters like sodium, urea etc. follow this pattern of min and max.

Any suggestions on how can I transform them without losing info . currently when I use as-is,  I get only around 86% accuracy with recall value of minority class only around 70. Resamplin didn't help much.

Your suggestions and experience in how to transform this to yield better predictions would really be helpful.


texts: How to interpret the mean for output clusters for expected-maximization?

texts:
I am trying to cluster data using scikit's expectation-maximization. So I created two different data sets from a normal distribution which is I have shown in the graph below.

The mean for each of the distribution is:

Mean of distr-1: 0.0037523503071361197
Mean of distr-2: -0.4384554574756237

But after I run the EM using scikit, I get the mean as follows:

Mean after EM: [[-0.12327634  0.39188704]
[-1.31191255 -4.4292102 ]]

How am I supposed to interpret this mean? I am trying to create two clusters from the data. Here is my code:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

distr_1 = np.sin(2 * np.random.randn(100) + np.random.randn())
distr_2 = (3 * np.random.randn(100)) + np.random.randn()

x = list(range(0,100))

X_train = np.concatenate((distr_1, distr_2))

plt.scatter(x,distr_1)
plt.scatter(x,distr_2)
plt.gca().legend(('sin', 'linear'))
plt.savefig('cluster_data.png')
plt.clf()

print("Mean of distr-1:",np.mean(distr_1))
print("Mean of distr-2:",np.mean(distr_2))

gmm = GaussianMixture(n_components=2)
gmm.fit(X_train.reshape(100,2))

print("Mean after EM:",gmm.means_)

Am I doing this incorrectly? What does the output mean?


texts: Audio dataset preprocessing to perform cry detection

texts:
I am building a neural network to perform cry detection (i.e., binary classification of cry/non-cry situations) when capturing sound in a house environment.

To do so, I performed the following steps:

Dataset building: I collected some samples in publicly available datasets, and enriched this sample set with some samples I registered with my own device (including speech, sounds from the home, sounds from the street, music...); I have a few samples (around 2500), which could be further enriched with new samples
Feature extraction: I used librosa to extract MFCC descriptors from my samples
Model creation and training: I built the model of my NN (using Keras, one recurrent layer, one dense layer and one final sigmoid node), and trained it on the features extracted at step 2.

Alas, although the model is performing very well on test set (achieving ~98% of accuracy), in the real world (i.e., when I put the trained model in a mobile application, stream the audio through my device and use it to classify the sounds I capture in the home environment I am at that moment) the network seems to perform very poorly, giving more or less random "cry"/"non-cry" labels over time.

I was wondering if this poor accuracy I achieve in the real world is related only to the small size of the dataset.

If this is not the case, is there some preprocessing I can do on my dataset? I learned from other questions that for speech recognition one would apply some preprocessing to remove some unwanted situations (e.g., to reduce the effect of volume change or to remove noise). Can you suggest me some preprocessing to do on the dataset, before training, either on the raw audio or on the extracted MFCC coefficients, so that this processing is adapt to my case study?

Thanks.


texts: How can I train my CNN to learn (numerically) smaller values better?

texts:
I'm using a CNN to model a problem that involves precise numerical values from a physical simulation.

After months of design/redesign and optimization, I've noticed that the majority of the "error" in this network is accumulated because of lower accuracy for smaller valued cases, whereas it seems to work fine when the values are numerically larger.

The input values are always normalized to $(0,1)$ when fed into the network.

The error is measured on the denormalized output values.

How can I make sure that the network learns these "small" values better?

I believe that the key problem lies in the fact that the network is "blinded" to the actual size of the I/O pairs because of the normalization process.

Is this an inherent problem with the normalization process, or is this a problem in how the network is designed?

The error is quantified using MAPE, in this case.


texts:
I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "replay" them through our neural network. This makes sense because we are trying to build out our rewards matrix and see if our episode ended in reward, scale that back through our matrix.

I would think the sequence of actions that led to the reward state is what is important to capture - this sequence of actions (and not the actions independently) are what led us to our reward state.

In the Atari-DQN paper by Mnih and many tutorials since we see the practice of random sampling from the memory array and training. So if we have a memory of:

$(action\,a, state\,1) \rightarrow (action\,b, state\,2) \rightarrow (action\,c, state\,3) \rightarrow (action\,d, state\,4) \rightarrow reward!$

We may train a mini-batch of:

[(action c state 3), (action b, state 2), reward!]

The reason given is:

  Second, learning directly from consecutive samples is inefficient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates.

or from this pytorch tutorial:

  By sampling from it randomly, the transitions that build up a batch are decorrelated. It has been shown that this greatly stabilizes and improves the DQN training procedure.

My intuition would tell me the sequence is what is most important in reinforcement learning. Most episodes have a delayed reward so most action/states do not have a reward (and are not "reinforced"). The only way to bring a portion of the reward to these previous states is to retroactively break the reward out across the sequence (through the future_reward in the Q algorithm of reward + reward * learning_rate(future_reward))

A random sampling of the memory bank breaks our sequence, how does that help when you are trying to back-fill a Q (reward) matrix?

Perhaps this is more similar to a Markov model where every state should be considered independent? Where is the error in my intuition?


texts:
I'm working on NLP task using RoBERTa model. As training last very long I saved my model, but now for some reason, part of my code doesn't work with this pre-trained model (getting an error), and before saving and uploading newly it, the code worked fine with the trained model. My code:
model = model.from_pretrained("/content/drive/MyDrive/model")
model = model.cuda()
model.eval()

def to_list(tensor):
    return tensor.detach().cpu().tolist()

def run_prediction(question_texts, context_text):
    """Setup function to compute predictions"""
    examples = []

    for i, question_text in enumerate(question_texts):
        example = SquadExample(
            qas_id=str(i),
            question_text=question_text,
            context_text=context_text,
            answer_text=None,
            start_position_character=None,
            title="Predict",
            is_impossible=False,
            answers=None,
        )

        examples.append(example)

    features, dataset = squad_convert_examples_to_features(
        examples=examples,
        tokenizer=tokenizer,
        max_seq_length=384,
        doc_stride=128,
        max_query_length=64,
        is_training=False,
        return_dataset="pt",
      #  threads=1,
    )

    eval_sampler = SequentialSampler(dev_dataset)
    eval_dataloader = DataLoader(dev_dataset, sampler=eval_sampler, batch_size=10)

    all_results = []

    for batch in eval_dataloader:
        model.eval()
        batch = tuple(t.to(device) for t in batch)

        with torch.no_grad():
            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                "token_type_ids": batch[2],
            }

            example_indices = batch[3]

            outputs = model(**inputs, return_dict=False)

            for i, example_index in enumerate(example_indices):
                eval_feature = features[example_index.item()]
                unique_id = int(eval_feature.unique_id)

                output = [to_list(output[i]) for output in outputs]

                start_logits, end_logits = output
                result = SquadResult(unique_id, start_logits, end_logits)
                all_results.append(result)

    output_prediction_file = "predictions.json"
    output_nbest_file = "nbest_predictions.json"
    output_null_log_odds_file = "null_predictions.json"

    predictions = compute_predictions_logits(
        examples,
        features,
        all_results,
        n_best_size,
        max_answer_length,
        do_lower_case,
        output_prediction_file,
        output_nbest_file,
        output_null_log_odds_file,
        False,  # verbose_logging
        True,  # version_2_with_negative
        null_score_diff_threshold,
        tokenizer,
    )

    return predictions

context = "New Zealand (Māori: Aotearoa) is a sovereign island country in the southwestern Pacific Ocean. It has a total land area of 268,000 square kilometres (103,500 sq mi), and a population of 4.9 million. New Zealand's capital city is Wellington, and its most populous city is Auckland."
questions = ["How many people live in New Zealand?",
             "What's the largest city?"]

predictions = run_prediction(questions, context)

# Print results
for key in predictions.keys():
  print(predictions[key])

The received error:

Any ideas how to solve this issue?
(this code part mostly is based on this notebook:
https://colab.research.google.com/github/spark-ming/albert-qa-demo/blob/master/Question_Answering_with_ALBERT.ipynb)


texts: How does the GAN based prediction in K. Zhang et al. (2018) improve performance?

texts:
In Stock Market Prediction Based on Generative Adversarial Network by K. Zhang et. al, the authors feed financial data (X0...Xt) into an LSTM to predict Xt+1.
Then, they evaluate whether the series (X0...Xt+1) is real or not (with as Xt+1 either the predicted or the one appearing in the data).
This is done in a GAN system, with the predictor trained to create an Xt+1 so that the series would fool the discriminator.
What is the benefit of training the predictor as a generator in GAN, with the discriminator in providing the loss function, when it would be simpler to use a loss based on |predXt+1 - Xt+1| ?
After all, the aim is to predict the target values - so why not train using the fit of the prediction directly?


texts: I am new to deep learning and trying to build a Grad-cam from time series data. Shape of my input sample  is (188,1), its an ECG signal and I have a cnn-1D model for classification. Keras provides function  Grad-cam for images(2D). I am trying to convert this function for 1D time-series data. Could anyone please help me how to do it. Thank you in anticipation.


texts: Polynomial Regression Not Overfitting as Expected

texts:
I am trying to implement polynomial regression in Python and have it mostly working, but it is not overfitting enough for higher degree polynomials. This is a weird thing to be upset about, I know, but it may point to an underlying problem with my implementation.
In this case I'm using a toy example where I'm sampling uniformly in the region $[0,1]$ and setting output values to $\sin(2\pi x) + \mathcal{N}(0, 0.3)$, i.e. just sin with added Gaussian noise with variance $0.3$.
I am using the normal equation form rather than gradient descent, so solutions should be optimal least-squares solutions. By Lagrange's interpolation theorem, we should have a polynomial of degree n fits n + 1 points exactly. However, for 10 samples and a 9th-degree polynomial (or even higher degrees), my implementation does not fit every point exactly.

You'll notice in my code that I'm not representing all the values of x in the graph exactly (i.e., I may be off after the ten-thousandths place), but the effect should not be as noticeable as it is if that is causing it. Because I'm sampling the x values uniformly in $[0, 1]$, I could have two overlapping x values within my error region which would cause some possible poor fitting in the graph, but this is extremely unlikely for low sample sizes (see my own question and solution to this problem here )
    import numpy as np
    import scipy as sp
    import matplotlib.pyplot as plt
    import math
    import random
    plt.rcParams['figure.figsize'] = [10, 5]

    def sample(n_samples):
        ''' Samples uniformly from x axis and returns y values given by sin(2pi)
        with Gaussian noise'''

        x = np.random.uniform(0, 1, size=n_samples)
        y = np.sin(2*np.pi*x) + np.random.normal(0, .3, n_samples)

        return [x,y]

    def poly_regress(data, n_param):
        ''' Implementation of polynomial regression with n powers. Uses normal
        equation rather than gradient descent.

        data: 2 by n matrix with data[0] the x values and data[1] the y values.
        n_param: degree of returned polynomial
        '''

        coeff = np.zeros(n_param) # Coefficient vector initialization

        # Calculate the vandermonde matrix of input x
        vand_x = np.vander(data[0],N = n_param + 1, increasing=True)

        # Computes the coefficient vector via the normal equation
        coeff = np.linalg.pinv(vand_x.T @ vand_x) @ (vand_x.T @ data[1])

        # This is for plotting purposes. If we wanted to predict a particular x
        # or range of values, then we'd use an almost identical form.
        x = np.arange(0,1, .0001)
        y = 0
        for i in range(n_param + 1):
            y += coeff[i]*np.power(x, i)

        # plots the true value of x. This only works for this toy example.
        true_x = np.arange(0,1, .01)
        true_y = np.sin(2*np.pi*true_x)

        plt.plot(true_x, true_y, color='g', label= "True Curve")
        plt.plot(x, y, color='b', label="Prediction")
        plt.plot(data[0], data[1], 'r+', label="samples")

        plt.ylim(-2,2)
        plt.legend()
        plt.show()
        return [x,y]

Any help is greatly appreciated. This is just base case testing before extending to more general forms of regression, so I want it to be functioning ideally at this point.
Cubic case:

High degree case with no noise:

With similar results for lower degree polynomials.


texts: Negative log-likelihood not the same as cross-entropy?

texts:
The negative log-likelihood
$$
\sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})
$$
can be multiplied by $\frac{1}{m}$ after which the law of large numbers can be used to get
$$
\frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}))
$$ as the sample size $m$ tends to infinity. This expectation is the "cross-entropy".
Now here comes my question: The book I am reading(Deep Learning by Goodfellow et al) mentions several attractive properties of using the negative log-likelihood(like consistency). But meanwhile, it also also uses cross-entropy directly as the loss function of maximum likelihood estimators:

This doesn't make sense to me - to talk about negative log-likelihood and cross-entropy as being identical. It would make sense for me to talk about NLL as an approximation of the cross-entropy.

I mean, they give different results - so why use one over the other? This seems like a valid question when they do not give the same results and must thus also affects the performance. Like, I am only aware of neural networks that use cross-entropy and not ones that use NLL - how come?

Maybe cross-entropy even holds other properties than negative log-likelihood?


texts:
Short version:
How can I find a function that maps X to Y when data looks like this.
Note:

For a pair of emissivity and distance relation between temperature and
raw_thermal_data is linear.

Long Version:
I am working on a project which uses thermal(IR) camera. Now we extract temperature from sensor reading (raw thermal data )
For some reason I need to find a function that maps temperature data to raw thermal data.
Now,
temperature = F ( raw_thermal_data, emissivity, distance )

I am trying to find,
raw_thermal_data = F1 ( temperature, emissivity, distance )

For a pair of emissivity and distance relation between temperature and raw_thermal_data is linear.
Looks like for every pair of emissivity and distance, intercept of the line is different.
Any thoughts?


texts: Keras RetinaNet implementation and width, height order

texts:
I'm trying to use the excellent open source implementation of RetinaNet with keras (https://github.com/fizyr/keras-retinanet) and I've noticed a little detail that I would like to clarify. In the preprocessing stage image is read (function read_image_bgr in keras-retinanet/utils/image.py link to code) and returned as an array in height,width order. The code is :
from PIL import Image

def read_image_bgr(path):
    """ Read an image in BGR format.
    Args
        path: Path to the image.
    """
    # We deliberately don't use cv2.imread here, since it gives no feedback on errors while reading the image.
    image = np.ascontiguousarray(Image.open(path).convert('RGB'))
    return image[:, :, ::-1]

Example my (640,480) image (width,height order) is returned as a numpy array of shape (480,640,3). This image array will later be fed to the model that expects a tensor whose dimensions are (batch,width,height,channels). I understand that this is simple convention and as long as the model processes the bounding boxes the same way, it is probably harmless. But I would like to be sure if it's desired behavior or not.
EDIT: as I'm using resnet backbone and want to do transfer learning, a corollary question is: in what order the resnet model expects images, (w,h) or (h,w)? I suppose that if the learned weights that I'm planning to use have been derived from learning of images in (w,h) order, then I should use this order too right?


texts: OCR Spell Check based upon previously found words

texts: I have a bunch of OCR produced text files with numerous spelling errors. I want to automatically correct these spelling based upon the prevalence of the closest correctly spelled words. For example, if the text reads "junp" and the options are "jump" or "june", then it would pick between the two based upon both context and the frequency of the two words. Is there any software like this?


texts: Unbalanced multi-class : distribution might change as more data come in

texts:
I am currently working on a problem of multi-class classification on testing logs data.

Basically, I have the context data from tests' execution saved, and want to automate the analysis of the failed tests.
The two targets I have are the page (which corresponds to the working team) responsible for the error, and the type of error.

So, for this matter I got the data at a time t and worked on it locally in order to get the best results possible.

I trained a few models, optimized them through hyper-parameter optimization with grid search, and got good results.

The f1_scores are around 90+% for the target page and 80+% for the target type.

These results seem good so I was working on ensembling methods to then get even better results from the combinations of my different models.

On the following figures, you can see the f1 scores and variances of different models for the target page.

But as I was working on ways to do ensembling, I got the idea of getting the latest testing data (remember I was working with the same data from the time t).

I went from having only 2162 samples to 4367, so a bit more than double the amount of data.

This gives my data a distribution different from before (on the following figures we can see the number of values per class for the target page).

Above, we can see that there are 11 highly unbalanced classes.

And on the second figure, we can see that we have now 14 classes, also very unbalanced.

(The differences with the new data are also true for the target type). It resulted in my models loosing between 6 and 10% of f1_score.

So, my questions are the following :

How do I work around data in this case in order for the models to work well with new data even when the distribution might change ? Will I need to check the evolution of the scores and when it goes too low, redo the parameter optimization ?
I am for now working with each target independently. Should I try multi-labeling between both targets ? If so, do you have any recommendation on algorithms or ways to go around it ?
Any suggestion on how to improve the work, maybe different models to try, or ensembling methods would be greatly appreciated ☺

Thanks for reading this long post !


texts: Convergence speed of perceptron algorithm

texts:
I was reading the convergence proof for the perceptron algorithm. It says under the assumption that there are some $R$, $\theta^*$ with $|\theta^*| = 1$ and $\gamma > 0$, such that $y_t(x_t\cdot \theta^*) \geq \gamma$ and $|x_t|\leq R$ for $t = 1, 2, \dots n$, the perceptron algorithm makes at most $\frac{R^2}{\gamma^2}$ errors.

What I didn't fully understand how $\theta^*$ was related to $x_t$ and how it affected the convergence of PLA. If I scale down all $x_t$ by a factor $k$, then I have $|x_t| \leq \frac{1}{k}R$, but what happens to $\theta^*$ and $\gamma$? Does scaling down $x_t$ gives a smaller upper bound and thus PLA converges faster? I personally believe how fast PLA converges is decided by how data is distributed rather than $|x_i|$, is it correct? Any hint or answer is appreciated, thanks in advance.


texts:
I am confused by the definition of the likelihood function in different contexts.

In the case of linear and logistic regression, it is defined as y given x

In the case naive bayes and LDA, it is defined over x and y

In the case of the EM algorithm (with observed variable x and unobserved variable z), it is defined over x

How do we know over what it should be defined?


texts: What are standard ways to normalize lengths for multiple time series?

texts:
An example of some time series input I have in mind. Suppose I have multiple users working on a 20-minute search task, using a commercial search engine like Google, Yahoo, or Bing. It's the same task, but users may finish after 10 minutes if they feel they have completed the task to the best of their ability. The y-axis is a "performance metric" (like normalized discounted cumulative gain), and measures are taken either:

At each query
At each minute

Both the # of queries and time are obviously not constant across users.  The goal here is for visualization/visual comparison, but if there's methods that are more suited to recommendation/prediction, I'm happy to hear those solutions, too!


texts: Can we use pca for supervise classification?

texts:
My questions are:

Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?
If I understand it right when we use pca for
feature extraction  after do it we have old features but in new
dimension then can we say these are the new features or mixed
features?
Is "feature extraction" different with "feature
selection" in pca? I mean they are two things to do with different
ways.
How can select effective features with pca?
There is a lots of references on the internet but I couldn't find
any MATLAB sample code or video (on youtube) about pca feature
selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).


texts: Why does "Depth = Semantic representation" in convolutional neural networks?

texts:
I was watching some videos online about convolutional networks, and the speaker was discussing the concept of running a filter over an image.

He said, and it is also shown in the image below, that "Depth = Semantic representation".

This sounds nice, but I'm not sure exactly what is so special about reducing the spatial dimension of the image in favor of increased depth? What makes depth more important than any other dimension? A dimension is just dimension, right?

EDIT: I have a hunch that the reason is going to have something to do with translation independence...


texts: Escaping from overfitting hell: introducing regularization vs increasing training data

texts:
I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn.
Here is a typical spectrum of the data that I am working with:

In this example, the data between 16:00 and 20:00 UTC -when the local railway system stops for the night- are assumed to be "clean" (1), while the remainder of the data are assumed to be "noisy" (0). This interval slightly changes from day to day, hence the need for a method allowing to automatically discriminate between clean and noisy data.
I train my model with 2 years of data. In order to get the same number of clean and noisy samples, for each day, I select 10 spectra between ~08:00 and ~09:40 as my noisy data and 10 spectra between ~17:00 and ~18:40 as my clean data. I therefore end up with a feature array X containing 14,600 samples (365x2x10x2):
for idx in np.arange(number_of_days):
               date=start_date+datetime.timedelta(int(idx))
               rd=RawData()
               rd.populate(station_id,date,data_type, decimation_level)

               pxx,  freq, t, cax = plt.specgram(rd.decimated_data, Fs=decimated_frequency, detrend='mean', cmap='jet', scale='dB')

               noisy_data=pxx[:,44:54]
               clean_data=pxx[:,94:104]

               if idx==0:
                    X_noisy=np.transpose(noisy_data)
                    X_clean=np.transpose(clean_data)
               else:
                    X_noisy=np.vstack((X_noisy,np.transpose(noisy_data)))
                    X_clean=np.vstack((X_clean,np.transpose(clean_data)))

 X=np.vstack((X_noisy,X_clean))
 y_noisy=np.zeros((X_noisy.shape[0]))
 y_clean=np.ones((X_clean.shape[0]))
 y=np.hstack((y_noisy,y_clean))

I then "whiten" my data and perform a 80/20 test-train split:
      X_scaled = preprocessing.scale(X)
      X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

Finally, I fit my model and print out the score:
      clf = LogisticRegression()
      clf.fit(X_train, y_train)
      print("Model accuracy: {:.2f}%".format(clf.score(X_test, y_test)*100))

Alas, the score that I obatain is close to 99%, which as far as I understand means that I am overfitting my data. I know two remedies to overfitting:
1) Introduce regularization:
clf = LogisticRegression(C=regularization)

Regardless of what C value I select, I still end up being stuck with a score of ~99%.
2) Increase the size of the training dataset
If on each day I consider 12 clean spectra (from 08:00 to 10:00 UTC) and 12 noisy spectra (from 17:00 to 19:00 UTC), instead of the 10 originally selected, the score decreases from ~99% to ~68%. When considering this marginally larger dataset, the effect or regularization can be seen, with the score linearly increasing with the C value:

I should be pretty happy for nailing down this solution but I am surprised by how suddenly the score drops with a mere 20% increase in dataset size. Plus there is something fishy: if I consider slightly different intervals (say from 07:40 to 09:40UTC instead of from 08:00 to 10:00UTC), I revert back to a score of 99%. What am I doing wrong here?


texts: Data Annotation: "labeling" target vs features

texts: I understand how one would use a data annotation tool to label targets for a given sentence, for example though, I'm not clear on how placing labels on features can be used to improve model performance.  For example, in this text annotation tool , you can add "labels" to a body of text like person, location, event ...etc .  Given that you must create Word Embeddings to work with the data, and the vector representation is not human-readable, how would you be able to improve model performance by annotating feature variables?


texts: Hi, im currently working for a company that has some inventory control problems

texts:
First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing Received cases vs produced here in my country, produced out of the country, total produced and % of advancement.
Later on in a meeting she asked me to show the graph and table I made to the production supervisors, and she explained that somehow they could predict inventory with it, after the meeting I've been thinking a lot about it, I made another 2 graphs one dependant on the day of the week vs received items, and another dependant on time of the day vs received cases, but now im thinking of doing a regression analysis to be able to predict inventory.
This is not my field, I am a chemist, but I know that if I somehow do a regression analysis and while doing so I succesfully take into account all the variables that affects inventory I should be able to come up with some equation that gives me the answer I need right?
What do you think of my approach? any advice ?


texts: Calculate Q parameter for Deep Q-Learning applied to videogames

texts:
I am working on Deep Q-learning applied to Snake, and I am confused on the methodology. Based on the DeepMind paper on the topic and other sources, the Q-value with the Bellman equation needs to be calculated as follows:

Q(s,a) = r + γ(max(Q(s',a'))

While calculating the Q-value for Q-table is straightforward, it's not clear to me the process in Deep Q-learning. According to some sources, the future possible states for the actual state need to be processed with Deep Learning and the highest Q-value needs to be chosen (input: future state, output: Q-value, or input: current state, output: Q-value for future states). Then, a table containing [state, action, reward, future_state] is stored in memory to reduce critical forgetting. I don't understand how we get the Q-values to predict in first place, if we need the states in order to compute the Q-values. Is this approach correct, or I am missing something?

Thank you!


texts: Machine learning algorithms for Geometrical objects shapes identification

texts:
Are there Machine learning algorithms which will take input dataset of all geometrical objects shapes as images in gif,jpg,tiff formats & output the geometrical shapes names?
i.e. Geometrical Objects pattern identification.
Examples : Square, Rectangle, Kite, Circle, Sphere, Cone, Rhombus, Ellipse, Cylinder, Pentagon etc
These are examples of potential inputs with corresponding labels


texts: How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

texts:
For several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all.

How does IQN differ from Quantile Regression Network, in plain language?

In Reinforcement Learning, a DQN would simply output a Q-value for each action. This allows for Temporal Difference learning: linearly interpolating the current estimate of Q-value (of the currently chosen action) towards Q' - the value of the best action from the next state.

Quantile-Regression Network takes it a step further, outputting a range of values for each action. So if with DQN we had a vector of Q values, QR-DQN gives us a vector of sub-vectors of Q values. These Q-values are equally likely to occur when compared amongst each other (in that sub-vector).

For example, if we have 3 possible actions, QR-Network might output the following:

Q estimates for Action0:  -15,  -5,  120,  121   130
Q estimates for Action1:   20,  25,  200,  210   300
Q estimates for Action2:   18,  40,   41,   54   120

Once again, when working with QR-Network, we treat -15,  -5,  120,  121,  130  as having the same probability of occurring: in my case  0.2  or 20%

In other words, each action now has values sitting at the middles of 5 separate quantiles:  at $\tau$ = 0.1, $\tau$=0.3, $\tau$=0.5, $\tau$=0.7, $\tau$=0.9

We can therefore say, that for the Action0:

with probability of 20% values within [$-\infty$, -10] will occur
with probability of 40% values within [$-\infty$, 57.5] will occur
with probability of 60% values within [$-\infty$, 120.5] will occur
with probability of 80% values within [$-\infty$, 121.5] will occur
with probability of 100% values within [$-\infty$, $+\infty$] will occur

To conclude, with QR-Net we demand it to output correct sub-vectors, knowing that each Q-score must have the same probability as others in that sub-vector.

Now, how does IQN work?

From what I understood after checking the paper, we have to feed the network the $\tau$ itself (in my example either 0.1, 0.3, 0.5, 0.7 or 0.9),

Or should we instead feed it several random $\tau$, pulled from [0,1] range?
Say, we pulled-out 0.35,  0.05,  0.1,  0.2,  0.11, should we continue getting them, or should we stop once we taken out 5 of those $\tau$ values? Would that mean we have to do a forward-pass for every $\tau$ or can we use a vector of them all, at once?

As I understood, we have to feed this $\tau$ as an input, into the network, along with the state vector (concatenating).

Does IQN output only 1 value per action, or a vector of vectors, like QR-Network does?

The IQN paper gives a picture, but I am unable to grasp the intuition behind the 4th image:

Why does the above image show the $f$ function being used in DQN and in IQN but not in C51 or QR-DQN?  Does that mean that the output is no longer a vector of vectors, but is just a vector, like in a usual DQN?

The gist of IQN is in the following paragraph, but I just can't grasp it fully. What's the intuition behind the cosine? I know it's a periodical function, why does it get computed 64 times with i=0 to i=64, with the same $\tau$ (for example $\tau=0.4$ etc)


texts: How to make a numpy ndarray from several numpy arrays

texts:
I am not to figure out how to form a ndarray from existing numpy arrays.
Suppose I have three arrays - index, distA, and distB.
Now the I want to form a ndarray of these three arrays

What I have tried -

>>> indices = np.array([[5,7,4], [6,4,8]])
>>> distances = np.array([[0.2, 0.3, 0.4], [0.3, 0.4, 0.5]])
>>> np.column_stack((indices, distances))
array([[5. , 7. , 4. , 0.2, 0.3, 0.4],
   [6. , 4. , 8. , 0.3, 0.4, 0.5]])

But I want to form a table-like structure so that I can retrieve values in distA and distB.

5, 0.2, 0
7, 0.3, 0
4, 0.4, 0.4
6, 0, 0.3
8, 0, 0.5

Is it possible to do this with numpy?


texts: Can anyone recommend me any GitHub code repo/s or textbook/s that walks through all the major ML algorithms in great detail with code to get hands-on with? I have a data engineering background and transitioning to data science and took multiple courses and got the theoretical background to get hands-on and want to create my own git repo for each Major Algorithms so I have a quick reference on the theory and coding skills handy for visual reference. I am researching here and there and taking too much time to collect and add up on my list. Thanks for your suggestions


texts: Keras trained model exported with older version of Keras ( < 2.2.0 )

texts:
Is it possible to update a trained model saved in a file without retraining it ?

I found the model on the web and I would like to use it but it uses Merge layers which are not supported by newer version of Keras, making it impossible to load with new versions.


texts: Using SVM as a binary classifier, is the label for a data point chosen by consensus?

texts: I'm learning Support Vector Machines, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in each dimension of the separating hyperplane?


texts: LSTM model prediction scaling with loaded model

texts:
I am deploying a LSTM pytorch model for production and I have issue with scaling the LSTM output correctly. While the model was tested the output was scaled with label data:
y_scaler = MinMaxScaler(feature_range=(-1, 1))
y_test_scaled = y_scaler.transform(y_test.reshape(-1, 1))

Now when I am loading the model from state_dict and making a prediction how I should inverse scale the data? I am not having the newest label data during the forecast. For the input data I am using QuantilieTransformer. I can make predictions this way:
y_test_scaler = MinMaxScaler(feature_range=(-1, 1))
y_minmax = label_data # For example past 40 days of the label data

y_test_scaled = y_test_scaler.fit_transform(y_minmax.reshape(-1, 1))

If I use the whole label-data as y_minmax the results are not good enough. But If I use less label data for scaling predictions are decent.
# Inverse transform predictions from LSTM model
y_actual = y_test_scaler.inverse_transform(y.reshape(-1, 1))

Should the output of LSTM be somehow inversed with x_scaler or how? :)
EDIT:
I got good results by making prediction for large amount of sequences. (Predicting 1 sequence to future and 300 sequences of history data). It is not a problem to load a lot of history data during the prediction, but I believe there is a smarter way to get the same result.


texts: How to handle correlated variables in a dataset?

texts:
I am wondering how to handle correlated fields in a dataset.

Some people suggest to drop this correlated fields because they don't provide any inside, other claim that a lot of information is getting lost. I am not sure if I should keep this correlated fields or not.

How to handle correlated data. Drop it, keep it, or does it depend on additional parameters? If yes, which? Are there rules I can follow?
What if the fields are sparse? (because the dataset is incomplete)
Does 1) depend on what kind of model is used? Regression, classification, NN?

To give a concrete example, the correlated values from the Zillow competition on Kaggle.


texts:
I have a dataset with 5K records and 60 features focused on binary classification. Class proportion is 33:67

Currently I am trying to increase the performance of my model which is stuck at F1-score of 89% (majority) and 75% (minority) class and AUC of 80%.

I tried Gridsearchcv and feature engineering. Though I don't explicity call out the best parameters on Gridsearch below, I guess when I fit, it takes the best parameters only. But nothing seems to help.

Does this mean my data has issues? When I mean issue, I am not talking about missing values. I mean the way the data was extracted. Can it be data entry issues?

This is what I tried for gridsearchcv. Am I doing it right?

import xgboost as xgb
parameters_xgb = {
    'learning_rate': (0.1,0.01,0.05,0.5,0.3,1),
    'n_estimators': (100,200,500,1000),
    'max_depth':(5,10,20),}
xg_clf = xgb.XGBClassifier()
xgb_clf_gv = GridSearchCV(xg_clf,parameters_xgb,cv=5)  # using cross validation with best hyperparameters
xgb_clf_op = xgb_clf_gv.fit(X_train_std,y_train)
y_pred = xgb_clf_op.predict(X_test_std)
cm = confusion_matrix(y_test, y_pred)
print(cm)
print("Accuracy is ", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))

I also tried catboost and gb. The AUC is only around 80-82% throughout in test data.


texts: Best Model for Multi-label image classification

texts: Usually using CNN architecture with a Sigmoid function as an activation function in the last layer and using binary cross entropy can be used to output a probability for each class. However, the problem is when we have more than 2 labels per image, the model will result in lower accuracy. Anyone has a suggestion for a robust state-of-art models that do Multi-label image classification  with a high accuracy.


texts: Supervised Learning: time estimation of bike repair

texts:
I would like to train a model that estimate the time a given shop would take to repair for a bike using the data below:

shops.csv

| variable | description              |
+++++++++++++++++++++++++++++++++++++++
| shop_id  | unique shop identifier   |
| country  | country of the shop      |
| city     | city of the shop         |
| nb_empl  | number of employees      |
| review   | google review            |

repairs.csv

| variable  | description                             |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
| repair_id | unique repair identifier                |
| price     | value of the repair in local currency   |
| type      | what to repair (brakes, wheel...)       |
| bike_model| model of the bike to repair             |
| size      | size of the bike to repair              |
| shop_id   | shop that carries out the repair        |
| duration  | time it took to repair the bike (TARGET)|

My first thought was to combine the datasets and treat the problem as a supervised learning problem. Meaning that I would use regression, XGBoost, random forest to fit my model in order to get the estimated time of repair, all the shops in the training set being mixed.
But my concern is that my shops might be similar, it doesn't mean that they would spend the same amount of time to repair a bike.
My second thought was to treat each shop independently, so basically one model per shop.

My questions are:

How would you approach the problem yourself?
Is there a way to combine my two thoughts together to get a better model?


texts: CNN image to image translation: multiple image inputs to one image output

texts:
I am interested in training a CNN to take in inputs where each input is a set of low-resolution images and each ground truth is a single high-resolution image. The ground truth high-resolution image was generated by averaging the information from a set of 80 low-level images. What I would like the CNN to do is to generate the same high-resolution image from a smaller set of low-level images, for example 5 low-level images.

I was looking into TensorFlow GAN and was thinking I could input each training image as a concatenated array of low-resolution images and each ground truth as a single high-resolution image. However, I am unsure whether this is the best method since the dimensions of the input and output will be different. Could someone advise what is the best approach using TensorFlow (or alternative deep learning frameworks if there is a better option)? Thanks!


texts: How do I deal with non-IID data in gradient boosted random forest (for stock market)?

texts:
I am working on a stock market decision system. I have currently centered on gradient boosting as the likely best machine learning solution for the problem.

However, I have 2 fundamental issues with my data owing to it being from the stock market having to do with it not being IID.

First, because of the duration of average in some indicators use, some data-points are highly correlated. For example, the 2-year trailing return of a stock is not very different if measured a month ago. My understanding is that this requires a sampling (for ensembles) where I choose datapoints that are "far away" in time to make trees more independent. From what I can tell so far, Matlab does not have functionality to pick a random subspace with this criteria. When I was previously thinking of using simple bagging, I figured I would just build the trees myself from custom subspaces and aggregate them into an ensemble, but this won’t work if I want to do gradient boosting. Now, on this point I am not totally sure that it is so critical to have samples “far away.” My intuition is that it is better if they are, but even if they are not perhaps by right-sizing the percent of data sampled and having enough trees it gives the same result. I would love any insight on that issue and how I might be able to use LSboost in matlab on custom samples.  (One idea I have is to just to create a small number, like 5-10 custom samples, use LS-Boost on each, and then average them.)

The second fundamental problem is that data from a given stock is correlated/related to itself. I realized after thinking about it that this is of critical importance. Consider, it would likely be better, if there is enough data, to make a prediction for stock A from training data only or mostly from stock A than to use the entire market. Thus, I had been thinking of a “system” where I train on stock-specific data, stock-group data (where I use a special algorithm to group stocks), and the entire market, and then use a calculation (I can elaborate if interested) that determines which of these models is more likely to give the better result. If the input looks very different from the stock-specific training data, for example, then it will use the group or entire market. I am pretty convicted that some form of taking into account which stock the system is looking at is important to optimizing performance.

Now, on the second issue the question is what is the best way to organize this. Thinking naively, it would be great to simply feed categories to the predictor that indicate what stock it is looking at. However, my belief here from what I know about these algorithms is that this will have poor results on new data, because this predictor will assume that it has seen the full universe of potential outcomes for each stock, when many times this isn’t the case. (Say there is a stock with only a one year history with a big rally – the system will think the rally will continue regardless of how different the new data looks). So I feel like I have to do something like in the previous paragraph. I don’t know if there is some way for the system to “automatically” recognize when new data is sufficiently similar to stock-specific data to focus on a stock-specific prediction vs. when it is different and it should go to the default system with multiple stocks.

If you have any insights on these issues and/or how to address them in Matlab or otherwise, I would very much appreciate. Thanks in advance.

Best,
Mike


texts: Text annotating process, quality vs quantity?

texts:
I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)


texts:
Consider any network, say network of flights between many cities or Internet or brain or social network. Is it correct to say that it can be considered as discrete dynamical system provided we have an expression that defines edge as a function of time ?
Will it differ if the nodes are dynamic and consequently the edges ?


texts: Difference between training and test data distributions

texts:
A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?

How does one check whether two distribution are statistically different?
Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference?
Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?


texts: creating dataset/storing vectors from c++ to python

texts:
I am new to the data science field.
I need to create dataset which consists of n inputs (vectors of size 5) and n outputs (vectors of size > 10,000). These vectors are generated using c++ code.
What is the best way to store them, such that they can be easily readable in python and use to train tensorflow/keras network?
I am looking for recommendation on format and tools to be used.


texts:
I'm looking to use google's word2vec implementation to build a named entity recognition system.  I've heard that recursive neural nets with back propagation through structure are well suited for named entity recognition tasks, but I've been unable to find a decent implementation or a decent tutorial for that type of model. Because I'm working with an atypical corpus, standard NER tools in NLTK and similar have performed very poorly, and it looks like I'll have to train my own system.

In short, what resources are available for this kind of problem?  Is there a standard recursive neural net implementation available?


texts:
I need to build a credit risk scorecard using logistic and linear regression. The variables using to predict are all dummies, where each dummy is a bin of some variable.
Let's say the variable age, I have 4 dummy bins for "age": age_20 (1 when client has less than 20), age_2030 (1 when client betwen 20 to 30), age_3040 (same for 30 to 40), age_4050 (same for 40 to 50).
I have more variables like this.

To run the linear or logistic regression I need to take out one dummy, because if not there will be colinearity on the variables. I also apply backwards selection, so only significant coeficients are kept.
The thing is that afterwards, after getting the parameter estimates I need to scale them so each category would be a "score".
The score formula is as follows:

-(woe*Beta+alpha/n)*factor+offset/n

where the WOE is a statistic for each bin, n the number of variables.
My question is, what should I do with the base case when it comes to the dummies?
And, let's say that my regression removed 2 more dummies because they were non significant, what does that mean to the score calculation?


texts: Latent loss in variational autoencoder drowns generative loss

texts:
I'm trying to run a variational auto-encoder on the CIFAR-10 dataset, for which I've put together a simple network in TensorFlow with 4 layers in the encoder and decoder each, an encoded vector size of 256. For calculating the latent loss, I'm forcing the encoder part of my network to output log variances instead of standard deviations, so the latent loss function looks like:

latent_loss = -0.5 * tf.reduce_sum(1 + log_var_vector - tf.square(mean_vector) - tf.exp(log_var_vector), axis=1)

I found this formulation to be more stable than directly using the logarithms in the KL-divergence formula since the latter often results in infinite loss value. I'm applying a sigmoid activation function on the last layer of the decoder, and the generative loss is computed using mean-squared error. The combined loss is simple a sum of both latent and generative losses. I train the network in batches of 40 using Adam Optimizer with a learning rate of 0.001.

The problem is that my network doesn't train. The latent loss immediately drops to zero, and the generative loss doesn't go down. However when I only optimize only for the generative loss, the loss does reduce as expected. Under this setting, the value of the latent loss quickly jumps to very large values (order of 10e4 - 10e6).

I have a hunch that the culprit is the extreme mismatch between the magnitudes of both losses. The KL-divergence is unbounded, whereas the mean-squared error always remains <1, so when optimizing for both, the generative loss basically becomes irrelevant.

Any suggestions to solve the problem are welcome.


texts: How to model the unknown in data science

texts:
I have been asked this question in interview which I was not sure how to respond.
Situation: My company provides personal loans where consumer income is minimum $10,000 per annum and we have data to analyze the default/ delinquency history of these customers.
Problem: is that if my CEO wants the benchmark reduced to $5,000 - what tools should I use to analyze the impact of reducing threshold on default / delinquency?


texts:
I'm reading through Hands-On Machine Learning with Scikit-learn and Tensorflow by Geron. I am creating a simple polynomial regression using sklearn's PolynomialFeatures.

First, I create an X and y set using numpy random numbers with quadratic shape:

m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)

Then I plot the scatterplot distribution:

plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([-3, 3, 0, 10])
plt.show()

Then I use PolynomialFeatures to add the 2nd degree:

poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)

Then I fit the LinearRegression:

lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
lin_reg.intercept_, lin_reg.coef_

Then I plot the same distribution with the quadratic regression line. My question is with the following code:

X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(X, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])
plt.show()

Why do we create X_new (np.linspace(-3,3,100).reshape(100,1) and X_new_poly? Why does this not work with the X_poly that I've already created? (I tried plotting it with the original X_poly and it definitely does not work. It's just oscillating lines up and down over and over. I'm just not sure why this is the case.)


texts: Are there good hyperparameter optimization networks for Generative Adversarial Nets?

texts: Finding good hyperparameters, as the learning rate for gradient descent, is crucial for good performance in deep learning. There exist various automatic methods for tasks as segmentation or classification, like Baysian optimization or HpBandSter. Are there good hyperparameter optimization networks for Generative Adversarial Nets (GANs)? If yes, which ones?


texts: Understanding Locally Weighted Linear Regression

texts:
I'm having problem understanding how we choose the weight function. In Andrew Ng's notes, a method for calculating a local weight, a standard choice of weights is given by:
What I don't understand is, what exactly is the x here? Apparently

  Note that the weights depend on the particular point x at which we’re trying
  to evaluate x.

But I don't get it.

Take the example of housing prices predicted by number of rooms and size in square feet. So each x^(i) is a [roomnum, size] array. So what's in x? I guess that also should be a [roomnum, size] array, but what's in it? Is it even a vector? Or is it the target variable? If so, why isn't it marked with y? I don't get it, please help!

EDIT
Ok so what I want is to make create a regression line like this:

How would I choose the x-es? What would they be in an algorithm? Do I have to make guesses for each x? How can I produce a line like this?


texts: K-means clustering of word embedding gives strange results

texts:
I'm trying to cluster words based on pre trained embeddings. I ran a simple experiment where I obtained around 100 words relating to "food taste", obtained word embeddings from a pre-trained set, and tried to run k-means on the result.

I do get sensible results on certain clusters, but very strange results on others. E.g:

Cluster 1: [fatty, oily, greasy] -- (good)
Cluster 2: [crumbly, powdery, grainy, flakey, chalky] -- (good)
Cluster 3: [flavorful, hearty, unflavored, savory, full-bodied] -(bad)
Cluster 4: [seasoned, unseasoned] -- (bad)

Can anyone suggest as to why seemingly opposite words like (seasoned, unseasoned) and (flavorful, unflavored) would be clustered together?

What I tried:

1) Using fasttext embeddings and Glove embeddings. The latest results are from concatenating fasttext wikipedia and common crawl embeddings.

2) Normalizing the vectors to same length before doing k-means (using eucledian distnaces). I think this is somewhat similar to having cosine distances. Tried not normalizing s well, but normalizing gave better results.

3) Tried some other clustring methods like DBSCAN but k-means seems better.

4) Tried PCA to reduce the dimensionality of the word vectors - didnt change the results much. Tried PCA for the selected embeddings in my vocab that needs to be clustered- not the whole word set.

Any suggestions to improve my results are welcome. If someone came across research articles discussing similar issues please post them as well.

Thanks!
TNC


texts:
So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, for both I got worse accuracy to what I got from making it 1(which is basically just SGD). Now I understand that I need to train it on more epochs but I'm not noticing any speed up in runtime which from what I read should happen. Why is this?

Here is my code(Java)

 public void SGD(double[][] inputs,double[][] expected_outputs,int mini_batch_size,int epochs, boolean verbose){
    //Set verbose
    setVerbose(verbose);

    //Create training set
    TrainingSet trainingSet = new TrainingSet(inputs,expected_outputs);

    //Loop through Epochs
    for(int i = 0; i<epochs;i++){
        //Print Progress
        print("\rTrained: " + i + "/" + epochs);

        //Shuffle training set
        trainingSet.shuffle();

        //Create the mini batches
        TrainingSet.Data[][] mini_batches = createMiniBatches(trainingSet,mini_batch_size);

        //Loop through mini batches
        for(int j = 0; j<mini_batches.length;j++){
            update_mini_batch(mini_batches[j]);
        }
    }

    //Print Progress
    print("\rTrained: " + epochs + "/" + epochs);
    print("\nDone!");
}

   private Pair backprop(double[] inputs, double[] target_outputs){
    //Create Expected output column matrix
    Matrix EO = Matrix.fromArray(new double[][]{target_outputs});

    //Forward Propagate inputs
    feedForward(inputs);

    //Get the Errors which is also the Bias Delta
    Matrix[] Errors = calculateError(EO);

    //Weight Delta Matrix
    Matrix[] dCdW = new Matrix[Errors.length];

    //Calculate the Deltas
    //Calculating the first Layers Delta
    dCdW[0] = Matrix.dot(Matrix.transpose(I),Errors[0]);

    //Rest of network
    for (int i = 1; i < Errors.length; i++) {
        dCdW[i] = Matrix.dot(Matrix.transpose(H[i - 1]), Errors[i]);
    }

    return new Pair(dCdW,Errors);
}
private void update_mini_batch(TrainingSet.Data[] mini_batch){
    //Get first deltas
    Pair deltas = backprop(mini_batch[0].input,mini_batch[0].output);

    //Loop through mini batch and sum the deltas
    for(int i = 1; i< mini_batch.length;i++){
        deltas.add(backprop(mini_batch[i].input,mini_batch[i].output));
    }

    //Multiply deltas by the learning rate
    //and divide by the mini batch size to get
    //the mean of the deltas
    deltas.multiply(learningRate/mini_batch.length);

    //Update Weights and Biases
    for(int i= 0; i<W.length;i++){
        W[i].subtract(deltas.dCdW[i]);
        B[i].subtract(deltas.dCdB[i]);
    }
}


texts: Splitting a sentence to meaningful parts

texts:
I have the following sentence

query = u'tell me about people in konoha who have wind style chakra and are above jonin level'

I want to split the above sentence into the following three parts

[
 "tell me about people in konoha",
 "who have wind style chakra",
 "and are above jonin level"
]

I have looked into sent tokenizers in spacy and nltk, but they are not giving me the desired results.

I am just a beginner in nlp and machine learning and have very limited knowledge so far. It would be awesome if you could direct me to some techniques or available packages through which I could achieve the above results.

EDIT

I'll be having question-like queries like the one above where I would be asking for details about the subject based on some constraints

In the above query the subject was people in Konoha and the constraints are have wind style chakra and above jonin level

I want to extract such relationships from similar queries.


texts: Backpropagation: how do you compute the gradient of the final output with respect to any loss function?

texts:
I'm trying to train a simple neural net but I'm unsure of the correct backprop value to use.

I'm using a 2 layer NN (1 input, 1 hidden and 1 output layer), using ReLU activations. Here is my h2

h1 = np.maximum(0, X.dot(W1) + b1.T)
h2 = h1.dot(W2) + b2.T
score = h2

In this case, if I were to backprop my score with respect to the loss function (cross entropy loss), what should be the correct computation I should use? I'm under the impression that this should always be 1 or at least usually equals to 1 since the final output is linearly related to the loss? On this writing, it doesn't really make sense. But if that's not the case, then what would be the correct computation?

Note that the score has not been directly linked to the cross entropy loss function, as I have not exponentiated the score matrix and do the necessary additions. If I were to link it to the cross entropy loss, then I'd do

loss_mat = np.exp(scores) / np.sum(np.exp(scores))
loss_mat = np.log(losses)
loss_mat[:,y] -= 1 #Minus one for the correct label? Why do this?
                   #read this as one of the solutions

Thank you.


texts:
I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years.
I just found the following statistics:

The poll of KDnuggets that actually shows that over the years is the following:

Gigabytes still rule: Majority of answers (56% in 2018, 57% in 2016, 56% in 2015, 54% in 2014, 53% in 2013) are in Gigabyte range. The overall median response was again between 11 and 100 GB (which comfortably fits on one laptop) for each year since 2012.

Do you know where to find better statistics?
I guess that the average size of the dataset has increased.
If not, I would like to see if at least the size of company-data available has increased.


texts:
I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a continuous variable X is:

But how could that be correlated? I mean I can take $r(s,a)$ as variable $x$, but why that $\pi_{\theta}(s,a)$ could be taken as $f(x)$? Thanks a lot for any hint or link or book for reference.


texts: What is the reason behind taking log transformation of few continuous variables?

texts:
I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or log of continuous variable like loan_amount or applicant_income etc.

I just want to understand the reason behind it. Does it help improve our model prediction accuracy. Is it mandatory? or Is there any logic behind it?

Please provide some explanation if possible. Thank you.


texts: Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset

texts:
I am trying to train an image classifier on an unbalanced training set. In order to cope with the class imbalance, I want either to weight the classes or the individual samples. Weighting the classes does not seem to work. And somehow for my setup I was not able to find a way to specify the samples weights. Below you can read how I load and encode the training data and the two approaches that I tried.
Training data loading and encoding
My training data is stored in a directory structure where each image is place in the subfolder corresponding to its class (I have 32 classes in total). Since the training data is too big too all load at once into memory I make use of image_dataset_from_directory and by that describe the data in a TF Dataset:
train_ds = keras.preprocessing.image_dataset_from_directory (training_data_dir,
                                                             batch_size=batch_size,
                                                             image_size=img_size,
                                                             label_mode='categorical')

I use label_mode 'categorical', so that the labels are described as a one-hot encoded vector.
I then prefetch the data:
train_ds = train_ds.prefetch(buffer_size=buffer_size)

Approach 1: specifying class weights
In this approach I try to specify the class weights of the classes via the class_weight argument of fit:
model.fit(
    train_ds, epochs=epochs, callbacks=callbacks, validation_data=val_ds,
    class_weight=class_weights
)

For each class we compute weight which are inversely proportional to the number of training samples for that class. This is done as follows (this is done before the train_ds.prefetch() call described above):
class_num_training_samples = {}
for f in train_ds.file_paths:
    class_name = f.split('/')[-2]
    if class_name in class_num_training_samples:
        class_num_training_samples[class_name] += 1
    else:
        class_num_training_samples[class_name] = 1
max_class_samples = max(class_num_training_samples.values())
class_weights = {}
for i in range(0, len(train_ds.class_names)):
    class_weights[i] = max_class_samples/class_num_training_samples[train_ds.class_names[i]]

What I am not sure about is whether this solution works, because the keras documentation does not specify the keys for the class_weights dictionary in case the labels are one-hot encoded.
I tried training the network this way but found out that the weights did not have a real influence on the resulting network: when I looked at the distribution of predicted classes for each individual class then I could recognize the distribution of the overall training set, where for each class the prediction of the dominant classes is most likely.
Running the same training without any class weight specified led to similar results.
So I suspect that the weights don't seem to have an influence in my case.
Is this because specifying class weights does not work for one-hot encoded labels, or is this because I am probably doing something else wrong (in the code I did not show here)?
Approach 2: specifying sample weight
As an attempt to come up with a different (in my opinion less elegant) solution I wanted to specify the individual sample weights via the sample_weight argument of the fit method. However from the documentation I find:

[...] This argument is not supported when x is a dataset, generator, or keras.utils.Sequence instance, instead provide the sample_weights as the third element of x.

Which is indeed the case in my setup where train_ds is a dataset. Now I really having trouble finding documentation from which I can derive how I can modify train_ds, such that it has a third element with the weight. I thought using the map method of a dataset can be useful, but the solution I came up with is apparently not valid:
train_ds = train_ds.map(lambda img, label: (img, label, class_weights[np.argmax(label)]))

Does anyone have a solution that may work in combination with a dataset loaded by image_dataset_from_directory?


texts: Consolidating multivariate time-series information from many data sets

texts:
I am having trouble setting up a problem with regards to time series analysis. I have 30 data sets, where each set corresponds to a certain project. Each project has 7 features, and each feature has time series information sampled every week from 2018 to today.
One of the features is how much the project is under/over budget and I wish to use this as the label.
If I was learning on a single project it could be a straight forward multivariate time-series. As an example I could transform the data from this:
Week   X1   Y1
1      0.5  3
2      1    5
3      1.5  8

to this:
X1   X2   X3   Y
-    -    0.5  3
0.5  3    1    5
1    5    1.5  8

Then I would use the second table as my input data. However, with 30 different projects all with the same time-steps I'm not sure how to combine this information so a single model could learn it. One solution I thought I could do was a bagging approach. I would train 30 models and I could do a voting/weighted average for predictions, but I feel like this isn't the best approach. If anyone has dealt with a problem like this before, please let me know. Thanks in advance!


texts:
I am new to GANs, but I was able to train a DCGAN decently. I decided to try a WGAN (not the improved one). I seem to get outputs, but my loss doesn't seem to converge for the generator.

I am using the loss function

from keras import backend as K
def wasserstein_loss(y_true, y_pred):
    return K.mean(y_true * y_pred)

for both the critic and generator. I am labelling real images as -1 and fake ones as 1.

Here's my loss over time:

Is the generator loss supposed to behave like that? Even though the discriminator loss seems to behave as expected, is a swing that large normal?


texts: Do i need sort the data by the eigenvalues with descend (PCA)

texts:
after computing the eigenvectors and eigenvalues.
the eigenvectors sort by eigenvalues on descending.
data * eigenvectors = transformed data
how about the standardized raw data, do I need to sort them by the eigenvalues too?
before calculating the transformed data.
or are they just the same result?


texts: Recommended papers on Deep Ordinal Regression?

texts:
Can people please recommend papers on Deep Ordinal Regression?

I'm looking for both the basic papers and the current important state-of-the-art ones - basically, everything that one "must" know in this field.

Thanks :)


texts: Multilingual Bert sentence vector captures language used more than meaning - working as interned?

texts:
Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then translated them via Google Translate, passed them through the model and saved their sentence vectors.

I then compared the results using cosine similarity.

I was surprised to see that each sentence vector was pretty far from the one generated from the sentence translated from it (0.15-0.27 cosine distance) while different sentences from the same language were quite close indeed (0.02-0.04 cosine distance).

So instead of having sentences of similar meaning (but different languages) grouped together (in 768 dimensional space ;) ), dissimilar sentences of the same language are closer.

To my understanding the whole point of Multilingual Bert is inter-language transfer learning - for example training a model (say, and FC net) on representations in one language and having that model be readily used in other languages.

How can that work if sentences (of different languages) of the exact meaning are mapped to be more apart than dissimilar sentences of the same language?

My code:

import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
MBERT = AutoModel.from_pretrained(bert_name)

#Some silly sentences
eng1='A cat jumped from the trees and startled the tourists'
e=tokenizer.encode(eng1, add_special_tokens=True)
ans_eng1=MBERT(torch.tensor([e]))

eng2='A small snake whispered secrets to large cats'
t=tokenizer.tokenize(eng2)
e=tokenizer.encode(eng2, add_special_tokens=True)
ans_eng2=MBERT(torch.tensor([e]))

eng3='A tiger sprinted from the bushes and frightened the guests'
e=tokenizer.encode(eng3, add_special_tokens=True)
ans_eng3=MBERT(torch.tensor([e]))

# Translated to Hebrew with Google Translate
heb1='חתול קפץ מהעץ והבהיל את התיירים'
e=tokenizer.encode(heb1, add_special_tokens=True)
ans_heb1=MBERT(torch.tensor([e]))

heb2='נחש קטן לחש סודות לחתולים גדולים'
e=tokenizer.encode(heb2, add_special_tokens=True)
ans_heb2=MBERT(torch.tensor([e]))

heb3='נמר רץ מהשיחים והפחיד את האורחים'
e=tokenizer.encode(heb3, add_special_tokens=True)
ans_heb3=MBERT(torch.tensor([e]))

from scipy import spatial
import numpy as np

# Compare Sentence Embeddings

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_heb1[1].data.numpy())

print ('Eng1-Heb1 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng2[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Eng2-Heb2 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng3[1].data.numpy(), ans_heb3[1].data.numpy())

print ('Eng3-Heb3 - Translated sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_heb1[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Heb1-Heb2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Heb1-Heb3 - Similiar sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Eng1-Eng2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng3[1].data.numpy())

print ('Eng1-Eng3 - Similiar sentences',result)

#Output:
"""
Eng1-Heb1 - Translated sentences 0.2074061632156372
Eng2-Heb2 - Translated sentences 0.15557605028152466
Eng3-Heb3 - Translated sentences 0.275478720664978

---

Heb1-Heb2 - Different sentences 0.044616520404815674
Heb1-Heb3 - Similar sentences 0.027982771396636963

---

Eng1-Eng2 - Different sentences 0.027982771396636963
Eng1-Eng3 - Similar sentences 0.024596810340881348
"""

P.S.

At least the Heb1 was closer to Heb3 than to Heb2.
This was also observed for the English equivalents, but less so.

N.B.
Originally asked on Stack Overflow, here


texts: Using training data generated with pure regular expressions - Can machine learning surpass the accuracy of your regular expression?

texts:
For text classification with machine learning - If your training data was generated purely with regular expressions, is it possible to train a machine learning model with this training data which will be more accurate than your original regex?

By 'accuracy' I mean to say the percentage that are correctly labelled as compared to the human truth of the question.

(E.g. we have a set of Tweets that we've labelled as happy or sad with regex. I'm wondering if a machine learning classifier trained on this data will be able to surpass the accuracy of the regex that generated the training data)


texts: Bias-variance tradeoff and the uncertainty principle

texts: Bias variance tradeoff seems to behave like the uncertainty principle, is it just another name for the same principle?


texts:
I dont know if this kind of question is allowed but i kinda hit a wall. I know about some clustering algorithms. I already implemented Fuzzy C-Means and Gaussian Mixture Model, but I dont really know what's the efficient way to cluster customer data and there is no label at all.

Since it's company data I can't say the detail, but if this helps here is the columns of the data:

First of all, I group by using panda's for each customer data so there's no duplicate. And then i just soft cluster to 15 clusters. Why 15 ? I tried to cluster it with number of product categories as number of clusters (since this what my supervisor asked me unless i can propose better method).

Is this the right way to do it ? or there is some papers that explain better methods?

what I make will be used for real marketing on e-commerce so I'm scared if my method screws the company up or something.


texts: checking model stability - Performance for different class

texts:
I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up with final data to train a classifier. I make sure that the data is balanced for all the 3 class.

To train a classifier I did XGB Classifier, Logistic Regression, SGD Classifier and Normal DNN(Tensorflow Estimator). I checked the metrics for all the classifiers and I am picking out the best one from the classifier.

Linear SGD Classifier Performance on Validation Set

     Class, Precision, Recall,    spe,       f1,      geo,      iba,      sup

      A       0.58      0.69      0.79      0.63      0.74      0.54       275
      D       0.51      0.61      0.66      0.55      0.63      0.40       338
      H       0.81      0.50      0.94      0.62      0.69      0.45       315
   Avg/mean   0.63      0.60      0.79      0.60      0.68      0.46       928

Model Performance  for Test Dataset

              pre       rec       spe        f1       geo       iba       sup

      A       0.87      0.55      0.97      0.67      0.73      0.51        84
      D       0.43      0.69      0.66      0.53      0.67      0.45        83
      H       0.80      0.69      0.86      0.74      0.77      0.58       139

We can see this model is stable over the class A and H but the precision is so poor for class D. I think because of a lack of feature the model is not performing well for class D. Though, I did several EDA and Feature Engineering to increase the recall for class D.

My question is, Is this model is considered stable?


texts: Understanding why shuffling reduces weirdly the overfit

texts:
I am a student currently trying to create a classification model, however I am having difficulty understanding a weird overfitting problem.

A dataset of about 30 000 entries, 30 features. The data is sorted by date of entry. I split my data to 80% training & 20% testing. I get a training accuracy score of about 98% and a test accuracy of about 71% using random forest. When I remove 3 specific parameters, the overfitting disappears, and I get to 73% training accuracy and 68% test accuracy. Which means these 3 parameters cause a big deal of overfitting. The weird thing happens when I shuffle the data. With all the 30 parameters, the training accuracy remains 98% and the test accuracy gets up to 92%. Which for me indicates that these 3 features values change unexpectedly during the last month or so of the data (the data was sorted by date before shuffling) and shuffling them gives the model enough examples from this last month to pick up the sudden change. But plotting the mean of their values/day, for the whole spectrum of dates shows that they follow the same seasonality throughout the whole data, and there are no weird changes. Can someone please give me some ideas to explain why shuffling the data helps to reduce the overfit massively?