texts: Using bert (or fitbert) for predicting masked words from word candidates

texts:
Fitbert (which is based on Bert) can be used to predict (fill in) a masked word from a list of candidates as below:
from fitbert import FitBert

# currently supported models: bert-large-uncased and distilbert-base-uncased
# this takes a while and loads a whole big BERT into memory
fb = FitBert()

masked_string = "Why Bert, you're looking ***mask*** today!"
options = ['buff', 'handsome', 'strong']

ranked_options = fb.rank(masked_string, options=options, with_prob= True)
# >>> (['handsome', 'strong', 'buff'], [0.02543771266937256, 0.0005182072054594755, 1.3243795365269762e-05])

There is also another instruction that assigns a probability to these sentences similarly.
However I need the model to predict multiple masked words as below:

I see [masked1] playing [masked2] with his team mate.

masked1 candidates ['her', 'him', 'his']
masked1 candidates ['football', 'laptop', 'paper']

It doesn't seem that I can modify the code so that I can pass all combination of candidate words and get their probabilities. e.g. ('her', 'football'), ('her', 'laptop') , ... , ('his', 'paper').
What should be done instead?


texts: How do you determine if a value is statistically significant?

texts:
I have collected some data I need to analyze. The data is the result of a survey in which I asked approx. 180 sellers at a bazaar, how important a certain buyer's characteristic is in relation to their price setting on a scale from '1 = absolutely unimportant' to '10 = extremely important' (for instance, how important is a buyer's nationality in relation to the price a merchant is offering his goods).

I now have analyzed my results and clustered the data, where it was necessary, but I do not know at which point a value gets statistically important.

Should I separate the data into two groups, one for answers below 5 and one for above, or in three groups (1-3,4-7,8-10) and at how many percent is something statistically important? Is one characteristic significant important, if 80% of the answers were 8 and above, is it still if there are just 60% above?

Edit: Ok let me clearify my question: I do not want to compare two chracteristics to each other. I am using R and I created histograms with density lines and also cummulative density lines to present my data. The problem is, that I do not know how I can say wether  one particular characteristic is important or not. Do I need to focus on their means? And how does their standard deviation affect the interpretation?
I have conducted a survey in one particular bazaar with approx. 3600 merchants, meaning I have the answers of 5% of them.


texts: Why could my DDQN get significantly worse after beating the game repeatedly?

texts:
I've been trying to train a DDQN to play OpenAI Gym's CartPole-v1, but found that although it starts off well and starts getting full score (500) repeatedly (at around 600 episodes in the pic below), it then seems to go off the rails and do worse the more it plays.

I'm pretty new to ML so I'm not really sure what could cause this so I'm not sure how to start debugging (I've tried tweaking some of the hyper-parameters, but nothing seems to stop this trend).
If it helps, here's the (probably) relevant parts of my agent:
def _build_model(self):
    model = Sequential()
    model.add(Dense(24, input_dim=self.state_size, activation="relu"))
    model.add(Dense(24, activation="relu"))
    model.add(Dense(self.action_size, activation="linear"))
    model.compile(optimizer=Adam(lr=self.learning_rate), loss="mse")
    return model

def get_action(self, state):
    # Use random exploration for the current rate.
    if np.random.rand() < self.epsilon:
        return random.randrange(self.action_size)

    # Otherwise use the model to predict the rewards and select the max.
    q_values = self.model.predict(state)
    return np.argmax(q_values[0])

def replay(self, batch_size):
    if len(agent.memory) < minibatch_size:
        return

    # Decay the exploration rate.
    self.epsilon *= self.epsilon_decay
    self.epsilon = max(self.epsilon_min, self.epsilon)

    minibatch = random.sample(self.memory, minibatch_size)

    state_batch, q_values_batch = [], []
    for state, action, reward, next_state, done in minibatch:
        # Get predictions for all actions for the current state.
        q_values = self.model.predict(state)

        # If we're not done, add on the future predicted reward at the discounted rate.
        if done:
            q_values[0][action] = reward
        else:
            f = self.target_model.predict(next_state)
            future_reward = max(self.target_model.predict(next_state)[0])
            q_values[0][action] = reward + self.gamma * future_reward

        state_batch.append(state[0])
        q_values_batch.append(q_values[0])

    # Re-fit the model to move it closer to this newly calculated reward.
    self.model.fit(np.array(state_batch), np.array(q_values_batch), batch_size=batch_size, epochs=1, verbose=0)

    self.update_weights()

def update_weights(self):
    weights = self.model.get_weights()
    target_weights = self.target_model.get_weights()

    for i in range(len(target_weights)):
        target_weights[i] = weights[i] * self.tau + target_weights[i] * (1 - self.tau)

    self.target_model.set_weights(target_weights)

And the full notebook is here.


texts:
I want to implement a racing bib number recognition application (for study purposes) using Keras API. I can manage training a CNN model to detect different numbers but how to localize only specific regions from the image and make sure that this region is a bib number?

Steps for this task:

Digit localizer - ?
Digit recognizer - trained CNN model using MNIST dataset.

What solution would be the best choice for localizing only specific regions? Should I use Faster R-CNN, YOLO Framework, Semantic Segmentation or smth else?


texts:
I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth.

Year
Company Name
FH_PBIDT
Growth
Avg_growth

2009-10
Aayush Food & Herbs Ltd.
0.044881
0.000000
-0.773403

2010-11
Aayush Food & Herbs Ltd.
0.063545
0.415839
-0.773403

For that, I am using the below code:

from scipy.stats.mstats import gmean

I created a column name:

df['Geometric Mean'] = gmean(df.iloc[:,3])
df['Avg_Growth Mean'] = gmean(df.iloc[:,4])

I am getting an error that gmean is not defined. Could someone help me here?


texts: How to deal with a biased feature in Machine Learning (date)

texts:
I have a model that predicts the lifespan of a horse. The dataset has samples from 1980 to 2019 and among the features there is one called birth_date, labeled with the lifespan in years for each horse. The problem is knowing that a horse usually lives between 20 and 30 years, if we look from 1980 to 1990 we have the full list of horses and their lifespans, but from 2000 to 2019 we see only samples of horses that were born and died within this time span but not the ones that are currently living, therefore the birth_date is a biased feature.

Is there a way to use the birth_date feature without having to worry about the biased data, or some technique to minimize his effects on the final predictions?


texts: Best approach for this unsupervised clustering problem with categorical data?

texts:
I'm a software engineer new to Machine Learning. I've read about basic non-supervised techniques like k-means and hierarchical clustering and now I'm trying to put them into practice with a basic problem.

Say I've got a lot of rows of data with each row looking something like this:

"employeesRange": "11-50",
"category": "Lobbying",
"categoryGroup": "Polication Action",
"sector": "Industrials",
"subCategory": "Direct",
"tags": [ "Progressive", "Libertarian", "PAC" ]

I want to analyze this data and look for patterns, i.e. clusters of information that frequently appear together. For example maybe it's a common pattern to find smaller groups of lobbyists 11-50 people for "Progressive", "Libertarian" causes but "Presidential" or "Local Politics" groups tend to skew larger.

Or maybe there's another link to be found between categories and sector, that kind of thing.

This seems a little harder then the examples I've read about because some of my data is a tag cloud (the "Tags" field) which is unstructured can contain multiple entries and the more structured fields of categories and employees range.

It seems a good place to start is using hierarchical clustering (since I don't know k a-priori) maybe following the recommendations here:

https://stackoverflow.com/questions/23943391/how-to-cluster-users-based-on-tags

I have a few questions about this. First, is this a good approach? And is Mahout the best tool for this? Any obvious ways to simplify? And finally, how can I combine the tag clustering approach with the other more structured data in the rows?


texts: Formal proof of vanilla policy gradient convergence

texts:
So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included. Looking at Sutton,Barto- Reinforcement Learning, they claim that convergence of the REINFORCE Monte Carlo algorithm is guaranteed under stochastic approximation step size requirements, but they do not seem to reference any sources that go into more detail.
I am curious whether or not anybody actually has a formal proof ready for me to read. I found a paper, which goes into detail for proving convergence of a general online stochastic gradient descent algorithm, see, section 2.3.
However, I am not sure if the proof provided in the paper is applicable to the algorithm described in Sutton's book. In the mentioned algorithm, one obtains samples which, assuming that the policy did not change, is in expectation at least proportional to the gradient. However, the analytic expression of the gradient
$$\nabla J(\theta) \propto \sum_s \mu(s)\sum_a q_{\pi}(s,a)\nabla \pi(a|s,\theta)$$
depends on the on policy state distribution $\mu(s)$ which changes when we update $\theta$. Therefore, when updating during the algorithm, the distribution changes.
Any help would be greatly appreciated. Bottou's paper, which I linked above states that the event is drawn from a fixed probability distribution, which is not the case here.
EDIT:
So after reading some more papers, I found this, which is a paper of Bertsekas and Tsitsiklis. They argue that under certain assumptions convergence to a stationary point is guaranteed, where one has an update rule of the form
$$x_{t+1} = x_t +\gamma_t (s_t + w_t)$$
and $w_t$ is some error with
$$\mathbb{E}[w_t  | \mathcal{F}_t] = 0$$
for ascending $\sigma$-fields $\mathcal{F}_t$, which can be thought of conditioning on the trajectory $x_0,s_0\dots,x_{t-1},s_{t-1},w_{t-1},x_t,s_t$.
I believe that this might be a solution since we need an expected gradient update given the past parameter $x_t$, which determines the sampling distribution which is exactly what the policy gradient theorem guarantees.
I'd be happy if someone could verify this.


texts: What is the difference between cross_validate and cross_val_score?

texts: I understand cross_validate and how it works, but now I am confused about what  cross_val_score actually does. Can anyone give me some example?


texts: Choosing regularization method in neural networks

texts:
When training neural networks, there are at least 4 ways to regularize the network:

 L1 Regularization
 L2 Regularization
 Dropout
 Batch Normalization

plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.

But how would one choose which of those regularization methods to use? Is there a more principled way than "just try everything and see what works"?


texts: Training a model that has both 2D and 1D features using a CNN

texts:
I'm looking to pre-train a model for an RL agent but I'm having some trouble figuring some stuff out.

Dataset: Minerl MineRLNavigateDense-v0

The observation space includes :

2D screen input (64,64) (+ 3 channels of color)
1D (scalar) compass angle
1D (scalar) number of dirt blocks
+ this is all over time.

I am also given the reward based on the action the human took.

When training a model using a CNN for time series classification my understanding is that each feature (out of k total features) at a given point in time can be represented as a scalar value (visualized as each box in each row) and then the "time" component can be visualized by each row being a new timestep:

My question is:
How do you set up the training data in such a way where this sort of CNN can be used to train on, particularly because each datapoint has a different dimension, do we just flatten the screen input into 64*64*3=12288 new features? Because that feels wrong.

UPDATE:
My first problem, combining the 2D image and the scalar data points was answered on Discord group I am in. In Keras at least, there's these special layer types called "merge layers" https://keras.io/layers/merge/ A CNN can be applied to the image first, then merged with the scalar data, this could then be passed into a 1D CNN to add the temporal component, however I haven't actually done any of this yet. :)


texts: How to determine irrelevant data in binary classification?

texts:
Suppose I was looking at social media to find users' intents on trading stocks. I might have a binary classification model that predicted "buy" and "sell". However, it's obvious that most social media posts mentioning a company are not related to buying or selling its stock. Even if I was to look specifically at places in the Internet where the main topic of discussion is buying and selling stocks, there would be handfuls of posts that were in a sense "off-topic" (e.g. "I applied to Microsoft today." or "What does everyone here think about Alphabet?")

My question is, how would one go about recognizing when a social media post does not suggest a user would buy or sell the stock. I had three quick ideas:

Create rules that would be able to differentiate relevant from irrelevant posts
Create a second binary classifier, that differentiates between relevant and irrelevant posts, and then uses the main classifier on only the relevant posts
Change the binary classifier into a classifier that can detect buy, sell, and off-topic documents.

Is there a customary approach to this problem?


texts: Python - accessing dictionary values for math operations

texts:
I have this dictionary:
stocks = {'FB': 255, 'AAPL': 431, 'TSLA': 1700}

and this script:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

for name in stocks.keys():
    ticker = (stocks[name])
    if name == stock:
        print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)
print(amount)

I wanted to access the value of the key when it is selected in the "stock" input and multiply the ticker chosen in the input, but it always multiplies 'TSLA' value 1700 by the number of shares selected, even if I choose FB as stock input.
The right output should be:
How many shares? 10
What's the stock? FB
2550
Instead I get 17000
I'd like to understand why.. thank you!


texts: How do we use a Hierarchical clustering model with DTW again?

texts:
I've been trying to cluster time series of shape (1, 400), so 1 row and 400 columns which correspond to 400 timesteps. My train set is of size (1000, 400) so 1000 time series. I have calculated a distance matrix with DTW and used it with scipy fcluster :

I'm quite satisfied with the result. I wanted to know how I can use this trained model to cluster new time series? I have 23 clusters.
Should I add the new time series with the old and calculate a new distance matrix? How do you save such a model?


texts: What could make a set of the train data more predictive than the whole train data

texts:
I took a sample of my training data and balanced it and then trained my model. The results obtained are more accurate than using the whole set of train data (balanced or imbalanced). My question is: what could explain this result? (the entire set of data does not contain any noise).

Thanks in advance.


texts: Reversed Naive Bayes - likelihood and parameter estimation

texts:
What happens if we flip the arrows in a Naive Bayes classifier? To clarify - from what I have found naive Bayes is defined for the following network structure:

I'm interested to understand what happens if instead from y->x the child will be y. As shown in the following:

I'm not entirely sure if we get the same results as a regular naive Bayes classifier (intuitively we don't), and if not how to estimate both the likelihood of the network and the parameters?

For further clarification: lets say there is a training set of size n; each data point consisting of (x,y) where x is a vector of size m of binary values and y is the class and is also binary. E.g. (0,1,0,1,1) where the last index is the class. I'm trying to figure out the likelihood of the network (where y is the child) and how to estimate the parameters in the case of y being the child.


texts: Reinforcement Learning different patients

texts:
I have historical data with labels and features about medical data about treating cancer. The labels represents:

How happy is the patient after the treatment (other patients did a survey in the past). From scale 1 till 7.

There features are numerical:
Via facial recognition we extracted six types emotions on a scale from zero till one.

This is performed for patients who are not able to talk and are in electric wheelchair with a tablet.

I trained a random Forrest Regressor. That predicts how happy will be patient on a scale from 1 till 7.

After the patient is done with the treatment the patient fills in a survey behind a screen. Each patient has a unique user ID. How would you apply personalization via reinforcement learning in order to know happy the patient will be during the next treatment.

The happiness of the patients it's face should be updated if this is not correct for this user.

For example:
I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, we want to estimate the patients happiness with the new updated model.

My model can be correct for patient A, and incorrect for patient B. Because for patient B some emotion was more or less important compared to patient A. How would you solve this problem and it should be scalable for 2000+ patients. Patients with cancer are coming back to the hospital for treatment (on average 10 or 20 treatments). And each patient has an Unique ID.

The model is making the prediction. Based on the predictions the doctors "receive" an happiness estimation.

The model makes: the wrong prediction for patient B but the correct one for patient A. Because patient B indicated after the treatment that the estimation was incorrect based on the scale scores. But Patient A indicated that his prediction was correct.


texts: How to check if sentiment analysis is required?

texts:
I have a CSV file having a bunch of sentences related to science. Before I do sentiment analysis on the sentences I want to programatically decide whether sentiment analysis is required on the sentence or not.

Basically some of the sentences are opinions of a particular topic in which case doing sentiment analysis makes sense. However some of the sentences are just definitions and sentiment analysis is not required on such sentences. So is there any way to detect the presence of a sentiment in a sentence?

(Note that the length of a sentence varies from 9 to 30 words.)


texts: How to remove background (watermark) logo from image

texts:
I have been scratching my head for a while. What I have is a scanned PDF document with text and water marked logo at the back as in the below image.

I want to do OCR over this, which becomes very difficult because of the logo. All the ratchet I've done so far is for coloured images where they can find contrast difference. I've hit a wall when solving the same for an B&W image as shown. Would love any help/suggestions for an approach/method to achieve what I am looking for.


texts:
I have seen senior data scientists doing data scaling either before or after applying PCA.

What is more right to do and why?


texts:
I have a dataset with 60k images in three categories i.e nude, sexy, and safe (each having 30k Images). I am using ResNet50 and observed that the training accuracy and validation accuracy is ok (around 0.82-0.88) although, the validation loss fluctuates a bit.
But, on testing, the precision and recall are too low for each of the classes, around(0.30, 0.25)
I have tried the following things:

Adding DropOut: I have tried adding DropOut with different rates (0.2 - 0.8)
Freezing and unfreezing batchNorm layers.
Using Inception Network
Adding data augmentation (with and without augmentation)

Here is the code
1. Model
model = Sequential()
base_model =ResNet50(include_top = False, pooling = RESNET50_POOLING_AVERAGE)

base_model.trainable = False
# un-freeze the BatchNorm layers
for layer in base_model.layers:
    if "BatchNormalization" in layer.__class__.__name__:
        layer.trainable = True

model.add(base_model)
#model.add(Dropout(0.7))
model.add(Dense(512, activation = 'relu'))
#model.add(Dropout(0.8))

model.add(Dense(3, activation = DENSE_LAYER_ACTIVATION))
sgd = keras.optimizers.SGD(lr = 0.001, decay = 1e-6, momentum = 0.9, nesterov = True)
model.compile(optimizer = sgd, loss = OBJECTIVE_FUNCTION, metrics = LOSS_METRICS)
model.summary()

2. Data Generator
NUM_CLASSES = 3

# Fixed for Cats & Dogs color images
CHANNELS = 3

IMAGE_RESIZE = 224
RESNET50_POOLING_AVERAGE = 'avg'
DENSE_LAYER_ACTIVATION = 'softmax'
OBJECTIVE_FUNCTION = 'categorical_crossentropy'

# Common accuracy metric for all outputs, but can use different metrics for different output
LOSS_METRICS = ['accuracy']

# EARLY_STOP_PATIENCE must be < NUM_EPOCHS
NUM_EPOCHS = 10
EARLY_STOP_PATIENCE = 3

# These steps value should be proper FACTOR of no.-of-images in train & valid folders respectively
# NOTE that these BATCH* are for Keras ImageDataGenerator batching to fill epoch step input
BATCH_SIZE_TRAINING =BATCH_SIZE_VALIDATION= BATCH_SIZE

# Using 1 to easily manage mapping between test_generator & prediction for submission preparation
BATCH_SIZE_TESTING = 1

image_size = IMAGE_RESIZE
data_generator = ImageDataGenerator( rescale=1./255,
                              horizontal_flip=True,
                              vertical_flip=True,
                              validation_split=0.2,
                              preprocessing_function = preprocess_input)
train_generator = data_generator.flow_from_directory(
        'Data/training/',
        target_size=(IMAGE_RESIZE, IMAGE_RESIZE),
        batch_size=BATCH_SIZE,
        shuffle= True,
        class_mode='categorical')

validation_generator = data_generator.flow_from_directory(
        'Data/validation/',
        target_size=(IMAGE_RESIZE, IMAGE_RESIZE),
        batch_size=BATCH_SIZE,
        shuffle= True,
        class_mode='categorical')

3. Fit
import keras.backend as K
K.set_learning_phase(1)
fit_history = model.fit_generator(
        train_generator,
        steps_per_epoch=num_train_steps,
        epochs = NUM_EPOCHS,
        validation_data=validation_generator,
        validation_steps=num_valid_steps,
        callbacks=[cb_checkpointer,cb_early_stopper]
)


texts: How to balance dataset using fit_generator() in Keras?

texts:
I am trying to use keras to fit a CNN model to classify 2 classes of data . I have imbalanced dataset I want to balance my data equally. How I can do that ??
Any help would be appreciated
The main code:
def generate_arrays_for_training(indexPat, paths, start=0, end=100):
    while True:
        from_=int(len(paths)/100*start)
        to_=int(len(paths)/100*end)
        for i in range(from_, int(to_)):
            f=paths[i]
            x = np.load(PathSpectogramFolder+f)
            x = np.expand_dims(x, axis=0)

            if('P' in f):
                y = np.repeat([[0,1]],x.shape[0], axis=0)
            else:
                y =np.repeat([[1,0]],x.shape[0], axis=0)
            yield(x,y)
history=model.fit_generator(generate_arrays_for_training(indexPat, filesPath, end=75),
                                validation_data=generate_arrays_for_training(indexPat, filesPath, start=75),
                                steps_per_epoch=int((len(filesPath)-int(len(filesPath)/100*25))),
                                validation_steps=int((len(filesPath)-int(len(filesPath)/100*75))),
                                verbose=2,
                                epochs=15, max_queue_size=2, shuffle=True, callbacks=[callback])

```


texts:
we see in every description of siamese network only anchor image and test/ query image is used but why is image pair of different dataset is made and trained ? cant we use only two image one mix image of different dogs and cat as anchor image and only one test/query image of only one dog and perform the similarity measure and if yes how to ?
 this two image and do siamese network test and trial only not image pair


texts:
I have some text files containing movie reviews I need to find out whether the review is good or bad. I tried the following code but it's not working:
import nltk
with open("c:/users/user/desktop/datascience/moviesr/movies-1-32.txt", 'r') as m11:
    mov_rev = m11.read()
mov_review1=nltk.word_tokenize(mov_rev)
bon="crap aweful horrible terrible bad bland trite sucks unpleasant boring dull moronic dreadful disgusting distasteful flawed ordinary slow senseless unoriginal weak wacky uninteresting unpretentious "
bag_of_negative_words=nltk.word_tokenize(bon)
bop="Absorbing Big-Budget Brilliant Brutal Charismatic Charming Clever Comical Dazzling Dramatic Enjoyable Entertaining Excellent Exciting  Expensive Fascinating Fast-Moving First-Rate Funny Highly-Charged Hilarious Imaginative Insightful Inspirational Intriguing Juvenile Lasting Legendary Pleasant Powerful Ripping Riveting Romantic Sad  Satirical Sensitive  Sentimental Surprising Suspenseful Tender Thought Provoking Tragic Uplifting Uproarious"
bop.lower()
bag_of_positive_words=nltk.word_tokenize(bop)
vec=[]
for i in bag_of_negative_words:
    if i in mov_review1:
        vec.append(1)
    else:
        for w in bag_of_positive_words:
            if w in moview_review1:
                vec.append(5)

So I am trying to check whether the review contains a positive word or a negative word. If it contains a negative word then a value of 1 will be assigned to the vector vec else a value of 5 will be assigned. But the output I am getting is an empty vector.
Please help. Also, please suggest others way of solving this problem.


texts: ggvis vs. ggplot2+Shiny; which one to choose for interactive visualization?

texts:
There is a similar question here in CrossValidated, and I have read the answers. My question is a bit different. I don't want to merely visualize my data, and indeed what I want to visualize is not easy to visualize with either package.

I have two sets of points ($x,y$ coordinates) on my plot. I want to add edges and make it similar to a graph. If I have $n$ points, then based on the nature of my problem, I need to add $(n/2)^2$ edges to the plot.

Then I want to add some interactive features to my plot. For instance, by clicking on one point (vertex), all other vertices which are not connected to this clicked vertex must disappear to have a more lucid and focused plot.

I tried to find a network visualization package suited to my project, but it seems all of them use the data about connection of edges or nodes and provide a network based on some specific layout algorithm. My data is different. I have the coordinates of nodes and I want to establish some edges between some of them. In my data the nodes have their own fixed locations and must not be moved around.

Finally, I narrowed down the packages to ggvis and ggplot2. ggvis is new and fresh in addition to having intentional design to generate interactive plots. But I found out that ggplot2+Shiny can also produce interactive plots. (Is that correct?) Besides, I figured that ggplot2's capabilities are more varied and matured than ggvis: for instance ggplot2 has a zooming feature while ggvis does not have it. More important, as ggvis is growing and changing, I don't want to invest a lot of time on a work that becomes technically obsolete or buggy in a year or so. I thought this is the case if I code based on ggvis.

Now may you please evaluate my choice (ggplot2+shiny) and advise me about possibility to do what I want to do with it?

As I did not know any of the mentioned packages, I have started learning ggplot2 from Wickham's book and I love it! But I am afraid of spending a few weeks and finding out that this amazing package cannot do what I want to do even with the aid of Shiny.

P.S: I want to put the final code on github so probable users can download and run it. Therefore, I do not need to make a web application, there is no need to any website or putting the interactive map online. I just need to have an interactive plot which can be run from inside of Rstudio.


texts: How can I select data from a Tensorflow Dataset data collection?

texts:
Is there any way to select features or labels from a tensorflow Dataset without using numpy conversion methods or iterate through?

The simplest example I found is:

import tensorflow_datasets as tfds

ds = tfds.load('mnist', split='train', shuffle_files=True)
print(ds)
>> <_OptionsDataset shapes: {image: (28, 28, 1), label: ()}, types: {image: tf.uint8, label: tf.int64}>

In this case, _OptionsDataset holds data in a dict, but the collection can be a tuple, list or even a np.ndarray.

If there is not a way to select data directly from a tensorflow Dataset, what's the most commonly used alternative?


texts: What is the bellman equation update rule for the average reward reinforcement learning? I searched a few articles, but could not find any practical answer.


texts: Perform clustering on individual data in categories

texts:
I am looking for ways to perform clustering on an aggregated category while still using individual data.

For example, assume we have a multitude users that we have acquired through different channels. I would like to cluster groups of channels together, so that it is easier to deal with similar channels in similar ways. However, if I aggregate the user data into averages for each channel, then because these averages are heavily driven by outliers (high-spenders), it doesn't give that much insight to the behavior of the majority of users from that channel.

Is there a way to use the individual user data but still gain insight into and cluster based on the channel they come from?


texts: Is it recommended to compute the average of metrics for classification?

texts: For computing the performance metrics of the models used for classification. Is it recommended to repeat the experiments many times (for example 10 times) and compute the average? In my case, I used 116745 samples for training and 125927 for testing.


texts: I would like to ask if the Pearson correlation between fields (but not the class field) of a dataset affects somehow the performance of Naive Bayes when applying it to the dataset in order to predict the class field.


texts:
I have trained and saved a model :

import numpy as np

# load the dataset
dataset = np.loadtxt("modiftrain.csv", delimiter=";")

# split into input (X) and output (Y) variables
X_train = dataset[:,0:5]
Y_train = dataset[:,5]

from sklearn.naive_bayes import GaussianNB

# create Gaussian Naive Bayes model object and train it with the data
nb_model = GaussianNB()

nb_model.fit(X_train, Y_train.ravel())

# predict values using the training data
nb_predict_train = nb_model.predict(X_train)

# import the performance metrics library
from sklearn import metrics

# Accuracy
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(Y_train, nb_predict_train)))
print()

# import the lib to load / Save the model
from sklearn.externals import joblib

# Save the model
joblib.dump(nb_predict_train, "trained-model.pkl")

Then, i'm loading the model and try to make predictions on a new dataset :

# import the lib to load / Save the model
from sklearn.externals import joblib

import numpy as np

# Load the model
nb_predict_train = joblib.load("trained-model.pkl")

# load the test dataset
df_predict = np.loadtxt("modiftest.csv", delimiter=";")

X_train = df_predict

nb_predict_train.predict(X_train)

print(X_train)

Here comes the error :

  File "predict01.py", line 14, in <module>
    nb_predict_train.predict(X_train)
AttributeError: 'numpy.ndarray' object has no attribute 'predict'


texts: Creating a feature by combining 2 features with different units?

texts:
If I want to create a new feature by combining 2 features with the same unit, I can just subtract them (e.g. buy volume vs sell volume). If the units are different, it doesn't make sense to subtract them (e.g. trade volume vs orderbook size). I can divide them if one of the features is guaranteed to never be 0.

If either features can be 0, what's the best way to compare them? I don't want to use an arbitrary high number to represent infinity if the denominator is 0.


texts: What to do if training loss decreases but validation loss does not decrease?

texts:
I am training a LSTM model to do question answering, i.e. so given an explanation/context and a question, it is supposed to predict the correct answer out of 4 options.

My model architecture is as follows (if not relevant please ignore): I pass the explanation (encoded) and question each through the same lstm to get a vector representation of the explanation/question and add these representations together to get a combined representation for the explanation and question. I then pass the answers through an LSTM to get a representation (50 units) of the same length for answers. In one example, I use 2 answers, one correct answer and one wrong answer. From this I calculate 2 cosine similarities, one for the correct answer and one for the wrong answer, and define my loss to be a hinge loss, i.e. I try to maximize the difference between the cosine similarities for the correct and wrong answers, correct answer representation should have a high similarity with the question/explanation representation while wrong answer should have a low similarity, and minimize this loss.

The problem I find is that the models, for various hyperparameters I try (e.g. number of hidden units, LSTM or GRU) the training loss decreases, but the validation loss stays quite high (I use dropout, the rate I use is 0.5), e.g.

My dataset contains about 1000+ examples. Any advice on what to do, or what is wrong?


texts: How to create a historical timeline using Pandas Dataframe and matplotlib

texts:
So I've seen a few answers on here that helped a bit, but my dataset is larger than the ones that have been answered previously. To give a sense of what I'm working with, here's a link to the full dataset. I imported this into a jupyter notebook using pandas. I've included a picture of one attempted solution, which was found at this link, right here:
Example Picture. The issue is that 1. This is difficult to read and 2. I don't know how to flatten it out so that it looks like a traditional timeline. The issue becomes more apparent when I try and work with larger segments, such as this one: It's basically unreadable. Here's the code I used to produce both of these (I just modified the included code in order to change which section of the overall dataset was used).

event = Xia['EnglishName']
begin = Xia['Start']
end = Xia['Finish']
length = Xia['Length']

plt.figure(figsize=(12,6))
plt.barh(range(len(begin)), (end-begin), .3, left=begin)
plt.tick_params(axis='both', which='major', labelsize=15)
plt.tick_params(axis='both', which='minor', labelsize=20)
plt.title('Xia Dynasty', fontsize = '25')
plt.xlabel('Year', fontsize = '20')
plt.yticks(range(len(begin)), "")
plt.xlim(-2250, -1750)
plt.ylim(-1,18)
for i in range(18):
    plt.text(begin.iloc[i] + length.iloc[i]/2, i+.25, event.iloc[i], ha='center', fontsize = '12')

This code semi-works, but I'd prefer if the bars were either closer together or differently colored and all on the same y-value. I appreciate any and all help. I've been trying to figure this out for about two weeks now and am hitting a brick wall.


texts:
I try to test ResNet approach on cifar10 dataset with the following python code:

# load data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')/256
X_test = X_test.astype('float32')/256
y_train = keras.utils.to_categorical(y_train)
y_test = keras.utils.to_categorical(y_test)

# build a model
def res_unit(x):
    x_shortcut = x

    x = Conv2D(16, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(16, (3, 3), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(32, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Add()([x_shortcut, x])

    x = Activation('relu')(x)

    return x

X_input = Input((X_train.shape[1:]))

X = Conv2D(32, (3, 3), padding='same')(X_input)
X = BatchNormalization()(X)
X = Activation('relu')(X)

X = res_unit(X)
X = res_unit(X)
X = res_unit(X)

X = Flatten()(X)
X = Dense(32)(X)
X = Activation('relu')(X)
X = Dense(10)(X)

X = Activation('softmax')(X)

model = Model(inputs=X_input, outputs=X)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# run model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5)

which outputs the worst result for 10-class classification: 0.1 accuracy on train and validation.

But when I comment all lines with BatchNormalization, everything starts to be OK: I got 0.57 accuracy after first epoch and then it even rises up.

What is the problem with BatchNormalization in my case? Am I using it in a correct way?


texts: In R, can I integrate different classifying algorithms in one bagging model?

texts:
I use R to do data analysis. I have a dataset. When I use different classifying algorithms, such as random forest, SVM, etc, I have the different accuracy. So, I want to integrate all the algorithms into one framework, let's say adaboost.

We know that adaboost framework use multiple "weak" classifying algorithms to combine a strong classifier. So, can I customize the "weak" classifying algorithms as I want? Here is just my current idea: In this framework, I use SVM first. Then give the data that are classified incorrectly more weights. Then, I use random forest. ... In the end, all the classifiers in this framework will work together.

This is just what I think about this issue. If there is other method working such as voting, please let me know too.

Any help is appreciated.


texts: Tableau: Clustering based on value-range for map coloring

texts:
Is there a possibility to cluster coloring for certain statistical ranges?

This is what I have been able to achieve so far.


texts: how to explain the behaviour: linear svm does better than non-linear RBF

texts:
I am working on a binary class classification problem.
Each sample is a vector 1x101, I have a lot of data samples more than 150k
I tried training a linear svm and a non-linear svm (RBF) "zscore normalization is used in both cases". surprisingly, the linear does better than the svm (RBF).
I am trying to explain this by considering the following points:

I beleive that the quality of my feature is not very good.
I think the nonlinear case experinces a kind of overfitting.

my question is how to explain this behaviour?!! does what I am thinking in make sense?!! I am thinking in using Adaboost to perform  the training, is it a good idea or not?


texts: How to Scaling Out Artifical Neural Networks?

texts:
I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as I can see it is quite "primitive" in comparison with TF and Keras (there are only feedforward types and hidden layer/output layer activation functions are hard-wired), there is no wide variety of optimizers, cost functions, architecture types etc.

Is there any competitive alternative for large scale neural networks? Of course, I could use Amazon with very powerful, GPU-driven machines but these are not parallel frameworks either...


texts:
I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having some good results (Been able to remove 65% of the features with an increase in accuracy), but I'm not impressed with how expensive each epoch is to run (in terms of time). Currently, this is a very expensive problem to optimise, 2^79 as there are a maximum of 79 features to choose from, which is 6.0446291e+23 possible feature permutations.

So far, each individual in my population has a binary encoded gene, where 1 = use this feature, and 0 = do not use this feature. Each individual in a population is evaluated by running the boosted regressor with the selected features, and then the RMSLE is calculated. I start to see good results around 3 hours into the optimisation function (I'm not doing any parallel computing).

I've been doing a little bit of research into how I can attempt to "predict" how good my boosted regressor will be without actually needing to evaluate its performance. So far I've found the following techniques:

Problem Approximation. I can see how this would be useful in some specific cases, but as I'm not really able to reduce the complexity of evaluating how accurate my boosted regressor is it seems irrelevant.
Functional Approximation. I think this is quite an interesting technique. They whole premise is to approximate the fitness of an individual without actually needing to evaluate the function
Fitness Inheritance. This also seems rather interesting, and perhaps the most efficient of the methods listed above? Individuals are clustered into specific groups, and then the "represented" individual of each cluster has its fitness evaluated, and then the remaining individual's fitness are approximated by upon a distance measurement.

I could see some issues with these techniques though. If the combination of features used don't have a linear relationship with the output, then surely the function approximation would have a hard time approximating the potential fitness of an individual? I'm a bit stumped on what to further look at.


texts: Using neural networks for classification in Hierarchical data

texts:
I am stuck in a problem wherein I have hierarchical data, say $A->B->C$(smaller to biggest), and the smallest unit of data is a block(A consists of multiple blocks, B consists of multiple A's, and C consists of multiple B's), and I want to classify blocks into labels.

Now the block labels for each group of A is independent of block labels for another group of A, however the "trends or patterns" followed by data could be similar and that is what is to be learnt.

The complexity I am facing is variable input sizes. I cannot possibly train single neural networks for groups of A, since its a large number. So, I am thinking in terms of groups at level B, but how could I create a scheme which could handle these variable input sizes.

Each block is represented by a one dimensional array of the total number of labels in the group of A it belongs to.Also, I have the information for hierarchy for every block(smallest unit) possible. Any help would be appreciated. Thanks!


texts:
I have a big list of spare parts with several parameters (material, weight, size, manufacturing complexity, ...). For some parts in this list, a price has either not been set or has to be adjusted in order to be in line with other parts. There are a few obvious and simple correlations in this dataset, for example:

if material and complexity is the same, bigger parts are more expensive;
if size and material is the same, more complex parts are more expensive;
for equal size and complexity, more expensive material leads to higher price.

Trying to figure out all these rules by hand and sticking them together seems to be an endless endeavour, so I thought about training a neural network with the priced parts' parameters (input) and prices (output) and let it figure out the prices for the parts which don't have a price yet. The decisions of the NN could be supervised by an expert who knows the parts and could manually figure out a price.

Question 1) Is this a good idea in general?
Question 2) If yes, what type of NN would be most suited for such a problem?


texts: 30% accuracy for training set, 80% for test set with a 0.3 split

texts:
I have a time series dataset on which I am training. For some reason, the training accuracy is 30% while the test accuracy is about 88% after about 10 epochs. Is this at all normal?

I should point out that the loss is decreasing for both the training and test data set.

[EDIT]

It's changing. The data does have quite an amount of padding. I'm using LSTM. The padding layers are [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] and the padding target is [0,0,0].

The problem is the time series has between 30 and 170 steps with the median being about 40 steps. I have no easy way of reducing the timesteps.

Also of importance, I don't have any Dropout layers. So this doesn't seem to be the explanation. I'd go with the padding though.

[UPDATE]
Seems OK. I've made a custom callback checking the window that I am interested in. accuracy for that is decent.


texts: Algorithm for classification of words into given categories

texts:
I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like

Medicine_Name
Disease_Name
Body_Part

etc.

Please give me some ideas to implement it, specifically how to prepare the training example and implement the training phase.


texts: How to deal with errors of defining data types in pandas' read_csv ()?

texts:
I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function  pd.read_csv command as:

df = pd.read_csv(filename, header=None, sep='|',
                 usecols=[1,3,4,5,37,40,51,76])

I would like to change the data type of each column inside of read_csv using dtype={'5': np.float, '37': np.float, ....}, but this does not work.

There is a message that column 5 has mixed types. The command print(df.dtypes) shows all columns of the type object. When I examine the column 5, I cannot see any problems. I have to change the data type for each column separately using pd.to_numeric.

My question is: Is there a way of setting data types inside read_csv and what is the problem in this case?


texts: Database for a trie, or other appropriate structure for recommendation engine

texts:
We are storing the information about our users showing interest in our items. Based on this information, we would like to create a simple recommendation engine that will take the items I1, I2, I3 etc of the current user, search for all other users that had shown interest in those items, and then output the items I4, I5, I6 etc of the other users, sorted by their decreasing popularity. So, basically, the standard "other buyer were also interested in..." functionality.

I'm asking myself what kind of a database is suitable for a realtime recommendation engine like this. My current idea is to build a trie of item IDs, then sort the item IDs of the current user (as the order of items is irrelevant) and to go down the trie; the children of the last trie node will build the needed output.

The problem is that we have 2 million items so that according to our estimation the trie will have at least 1E12 nodes, so that we probably need a distributed sharded database to store it. Before we reinvent the wheel, are there any ready-to-use databases or generally, non-cloud solutions for recommendation engines out there?


texts: Modeling time spent on different websites

texts:
Apologies in advance for the poor title. I didn't know exactly how to phrase what I want in a succinct manner so hopefully I can elaborate a bit more.
I have a dataset where I have customer_id, url_visited, and time_spent. So It looks like this (key is the url, value is time_spent):

customer_id
url
time_spent

1
google.com
20 min

1
facebook.com
30 min

2
napster.com
5 min.

2
netscape.com
6 min

So let's say I care less about whether or not they visited a site, (i.e is_google or is_netscape) and more about how long they visited each site (i.e time_spent_google, time_spent_facebook), what would be the best way to model this?
My idea is to do something like this:

customer_id
time_spent_napster
time_spent_facebook
time_spent_google

1
0  min
30 min
20 min

2
6  min
0  min
0 min

So a couple of questions:

Would this be an acceptable way to model this data in most traditional ml models such as RF?
If yes, what would be a technique to encode this with high cardinality data?

Any input is welcomed!


texts: About training and cross validation on a time series problem

texts:
I am new to machine learning.

I'm having a task of predicting whether a user will churn in March, given February feature data and the churn result. However, March data is leaked and now I'm assigned to predict for April.

My strategy is to train the model with February data and perform cross validation with March data. Then I will try to predict April data using model trained with February data.

Here are my questions:

Is my strategy good?
Or should I append both February data and March data to predict April
data?


texts: How to approach semi-supervised binary classification problem with few labels only from one class?

texts:
I confront with a binary classification problem where I do have a few instances with labels (so far this is "semi-supervised" learning as far as I know), but only from the positive class. So I cannot take any negative examples as basis for learning the other class. What is the best practice here? I assume that I should find some examples farthest from the explicit positives and treat these like negatives; but if so, what is a handy way for this in Python (preferably in sklearn)?
Furthermore, following the approach above, I'm a bit confused when to switch to supervised mode (if any time at all) if instances could be separated only with clustering?


texts: Why does least squares linear regression perform so bad when switching from 2D to 3D line?

texts:
The vector of coefficients that minimize least squares can be found like so:

beta = ((X'X)^-1)*X'y

Theoretically this result is true for any number of variables (columns of X).

I have made two python notebooks: one for univariate linear regression and another one for multivariate linear regression.

The code is analogous for both notebooks:

create a random line

# multivariate snippet. A part from the plotting part,
# this is the only part of the code that differs between the two notebooks.
beta = np.random.randint(1,6,(3,))
X = np.linspace([1, 3, 7],[1, 14, 23],100)
y = X @ beta

create noise

mu, sigma = 0, 1
noise = np.random.normal(mu, sigma, 100)

add noise to y

y_noise = y + noise

try to fit a line trough y_noise

beta_ = np.linalg.inv(X.T @ X) @ X.T @ y_noise
y_ = X @ beta_

compare y_ with y_noise

Why do I get such a different result when fitting a 2D line and a 3D one?


texts: How to set hyperparameters in SVM classification

texts:
I am studying image classification using SVMs and it is generally defined as so...

N = number of training examples

W = is the weights

f(x, W) = dot product

λ is explained to be set through cross-validation however no mention is made as to how Δ is set.

I understand that the SVM loss function wants the score of the correct class to be larger than the incorrect class scores by at least by Δ, but they don't explain how Δ is derived.

In most of the examples it is define to be Δ = 1.0, with no mention as to how 1.0 was calculated. Is this value determined through trial-and-error (cross-validation)? How does one determine what should be the value?


texts:
Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers?  In Hinton's paper (which proposed Dropout) he only put Dropout on the Dense layers, but that was because the hidden inner layers were convolutional.

Obviously, I can test for my specific model, but I wondered if there was a consensus on this?


texts: Deep learning: parameter selection and result interpretation

texts:
I have a multiclass(7 labels) classification problem implemented in MLP. I am trying to classify 7 cancers based on some data. The overall accuracy is quite low, around 58%. However, some of the cancers is classified with accuracy around 90% for different parameters. Below cancer 1,2,3 etc means different types of cancer, For example 1= breast cancer, 2 = lung cancer etc. Now, for different parameter settings I get different classification accuracy. For example,
1. hyper parameters
    learning_rate = 0.001
    training_epochs = 10
    batch_size = 100
    hidden_size = 256
    #overall accuracy 53%, cancer 2 accuracy 91%, cancer 5 accuracy 88%,
     #cancer 6 accuracy 89%

2. hyper parameters
    learning_rate = 0.01
    training_epochs = 30
    batch_size = 100
    hidden_size = 128
    #overall accuracy 56%, cancer 2 accuracy 86%, cancer 5 accuracy 93%,
    #caner 6 accuracy 75%

As you can see, for different parameter settings I am getting totally different results. Cancer 1,3,4,7 have very low accuracy, so I excluded them. But cancer 2, 5,6 have comparatively better results. But, for cancer 6, the results vary by great number depending on the parameter settings.
An important note is, here overall accuracy is not important but if I can classify 2-3 cancers with more than 90% accuracy that is more important. So my question is, how do I interpret the results? In my paper how should I show the results? which parameter settings should I show/use? Or should I show different parameter settings for different cancer types? So basically, how to handle this type of situations?


texts: Is there a standard method for choosing features from different feature selection techniques?

texts:
I have four different feature selection techniques, Backwards Elimination, Lasso, feature_importances, and Recursive feature selection. Each technique returns slightly different results. For example,

Backwards Elimination: Spread Direction
Lasso: Spread Move, and Spread
Feature_Importances_: Spread Percentage and Spread Money
Recursive: Spread Money

is there a standard method when choosing features from different models? Should you choose the features that each model returns or is there a preferred method when doing this?


texts: Is there a name for the discipline of creating database structures (tables, columns, etc) to model a domain?

texts:
I develop software, mostly web applications. I generally use a backend framework like Ruby on Rails. I have been relatively successful at modeling business domain structures into code and into database structures with a purely hands-on, intuitive approach for 15 years or more -- i.e. trial and error, and reading articles online (eg how best to store money in a database).
I'm considered a "senior" engineer, and seem to find ways to model business problems in code and data just fine. But as the years go by and I'm exposed to and work with data models that others have created, I realize there are many approaches, and I've never really truly studied this in depth or formally, and "sat at the feet of the masters" so to speak. And I'd like to and I know there's a lot I could learn.
I've read books on programming but never books that focus in depth on the art of architecting database structures to model complex real-world domains. Tonight when I went to look up books on the subject, I searched for "data modeling" thinking that would be what to call it, but all that turned up were results for making diagrams. By complex domains, I mean for example the entire domain of a travel touring company, including, itineraries, guides, tours, equipment, flights, registrations, payments, hotels, fees, etc, etc.
So I'm wondering if what I'm wanting to study is a discipline in its own right, with lots of juicy books and case studies, well known pitfalls, etc, or is it something too general to speak of in those terms.
If this discipline has a name, please let me know, along with any resource/book recommendations.


texts:
I have a time series data about daily usage of a computer program, here is an example

2017-11-10: 0
2017-11-09: 14
2017-11-08: 0
2017-11-07: 6
2017-11-06: 102
2017-11-05: 0
2017-11-04: 0

As you can see 11-06 has a spike at 102. Due to our way of gathering this data, we know that data is probably erroneous and we are sure that 102 is not correct according other values.

So we need to clean these dirty values.

Is there a mathematical way to do this?
Is there a python lib to help us?


texts: Keras retrieve value of node before activation function

texts:
Imagine a fully-connected neural network with its last two layers of the following structure:

[Dense]
    units = 612
    activation = softplus

[Dense]
    units = 1
    activation = sigmoid

The output value of the net is 1, but I'd like to know what the input x to the sigmoidal function was (must be some high number, since sigm(x) is 1 here).

Folllowing indraforyou's answer I managed to retrieve the output and weights of Keras layers:

outputs = [layer.output for layer in model.layers[-2:]]
functors = [K.function( [model.input]+[K.learning_phase()], [out] ) for out in outputs]

test_input = np.array(...)
layer_outs = [func([test_input, 0.]) for func in functors]

print layer_outs[-1][0]  # -> array([[ 1.]])

dense_0_out = layer_outs[-2][0]                           # shape (612, 1)
dense_1_weights = model.layers[-1].weights[0].get_value() # shape (1, 612)
dense_1_bias = model.layers[-1].weights[1].get_value()

x = np.dot(dense_0_out, dense_1_weights) + dense_1_bias
print x # -> -11.7

How can x be a negative number? In that case the last layers output should be a number closer to 0.0 than 1.0. Are dense_0_out or dense_1_weights the wrong outputs or weights?


texts:
Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on)
This is from the online order processing domain. An order consists of a set of variable number of items. Each item can be located in different warehouses, again this is a variable number. The entire order with multiple items and items with multiple warehouses per item, needs to be processed as one training sample. The goal is to learn a function that outputs the warehouses from which the items can be picked under some rules/conditions to minimize processing costs.
The number of items can run in millions and stores in 1000's.
I've been looking at representing these as permutation invariant sets - is there a simpler way or is that the right way to go about it ?


texts: When to use cosine simlarity over Euclidean similarity

texts:
In NLP, people tend to use cosine similarity to measure document/text distances. I want to hear what do people think of the following two scenarios, which to pick, cosine similarity or Euclidean?
Overview of the task set: The task is to compute context similarities of multi-word expressions. For example, suppose we were given an MWE of put up, context refers to the words on the left side of put up and as well as the words on the right side of it in one text. Mathematically speaking, similarity in this task is about calculating
sim(context_of_using_"put_up", context_of_using_"in_short")

Note that context is the feature that built on top of word embeddings, let's assume each word has an embedding dimension of 200:
Two scenarios of representing context_of_an_expression.

concatenate the left and right context words, producing an embedding vector of dimension 200*4=800 if picking two words on each side. In other words, a feature vector of [lc1, lc2, rc1, rc2] is build for context, where lc=left_context and rc=right_context.

get the mean of the sum of left and right context words, producing a vector of 200 dimensions. In other words, a feature vector of [mean(lc1+lc2+rc1+rc2)] is built for context.

[Edited] For both scenarios, I think Euclidean distance is a better fit. Cosine similarity is known for handling scale/length effects because of normalization. But I don't think there's much to be normalized.


texts: How to tune learning rate with HParams Dashboard on Tensorflow?

texts:
In Tensorflow documentation, it is shown how to tune several hyperparameters but not the learning rate.I have searched how to tune learning rate using HParams dashboard but could not find much. The only example is another question on github but it does not work.Can you please give me some suggestions on this?Should I use a callback function?Or provide different learning rates in hp_optimizer as in the question in github? Or something else?

Parts of my code is below:

HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
HP_L_RATE= hp.HParam('learning_rate', hp.Discrete([0.0005, 0.001]))

def train_model(hparams):
    model= tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(None,43), dtype=tf.float64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], activation='relu')),
    tf.keras.layers.Dense(1)])

    model.compile(optimizer=hparams[HP_OPTIMIZER],loss='mae',metrics=['accuracy'])

    callback = tf.keras.callbacks.LearningRateScheduler(hparams[HP_L_RATE])

    model.fit(train_dataset, epochs=100,callbacks=[callback])  #, validation_data=test_dataset
    _,loss=model.evaluate(test_dataset)
    return loss

Problem is I cannot figure out how and where to insert hparams[HP_L_RATE] in the train_model().As you can see, I have tried to use a callback function to implement hparams[HP_L_RATE], but it does not work.

Thank you,


texts:
I am trying to optimize some parameters that used to transform 2d points from a place to another (you may think of that as rotation & translation parameter for simplicity)
The parameters are considered optimal if the transformed points lay inside a pre-defined convex polygon. Otherwise, the parameters should be adjusted till all points lay inside that polygon. I do not care how the points are arranged inside that polygon, my only concern that they are inside..
How can I formulate my loss function in smooth way that the farer the point outside the polygon the more the loss function is and in the same time if they are inside the polygon, the distance does not matter.
P.S. the loss has to be differentiable since I am using derivative-based optimizer.
The first thing came in mind is to test if the point is inside the polygon (traditional point to polygon test), if it is inside, I will return 0, otherwise I will return the distance from the point to the nearest segment of the polygon. Is it valid/good? if yes, is there better?


texts: Drawing new binary classification boundaries given a change in some parameters

texts:
The basic binary classification algorithm determines a fixed decision boundary once it has been trained on some collection of samples. Does there exist a binary classification algorithm that re-draws the decision boundary based on some parameter $z$? The goal is to obtain a plot of this new decision boundary.

Suppose that our samples are points in $\mathbb{R}^2$ and points of class #1 are simply clustered together into a disc in the $xy$-plane, while the points of class #2 lie outside this disc. Now consider another parameter $z$ that causes the points to spread out (and form larger discs) or contract (and form smaller discs). This additional parameter could be viewed as the height – and the collection of samples for different $z$ values happens to form sphere, for example.

Does there exist a binary classification algorithm that takes in two inputs: one would be a vector in $\mathbb{R}^2$ (a point on our disc) and another would be a parameter $z$. The goal would be to have a function that draws the decision boundary for new values of $z$, given some training data for specific values of $z$.


texts: Predicting time series, with few historical samples based on similar series

texts:
I'm trying to build a model with Keras to predict the time series of a sensor, based on its type and historic data of sensors of the same type.

The figure below shows 3 time series, generated from 3 sensors of the same type, the green dashed line is the new sensor data and the vertical line is where the data for the new sensor end.

I've tried writing an LSTM network, that returns the hidden state output for each input time step, while the target was the values for each timestamp. Then trying to predict the new time series giving the model a few points of the sensor history data. With no luck :(

So I'm guessing I'm walking on the wrong path. What are the options of predicting a time series with just a few historical samples based on the history of other time series of the same type?

Any help / reference / video would be appericiated


texts: how to build a lift chart using scikitplot

texts:
I am trying to build a lift chart using scikitplot library.
I am getting the following can anyone guide with the error.
please find the code with the error
CODE:
import scikitplot as skplt
actual = df['Actual']
predicted = df['Pred']

skplt.metrics.plot_lift_curve(act,pdt)

Error:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-38-f14ae9c16143> in <module>
      6 pdt = np.array(predicted)
      7 # plotLiftChart(actual,predicted)
----> 8 skplt.metrics.plot_lift_curve(act,pdt)
      9 # plt.show()

~\Anaconda3\lib\site-packages\scikitplot\metrics.py in plot_lift_curve(y_true, y_probas, title, ax, figsize, title_fontsize, text_fontsize)
   1192
   1193     # Compute Cumulative Gain Curves
-> 1194     percentages, gains1 = cumulative_gain_curve(y_true, y_probas[:, 0],
   1195                                                 classes[0])
   1196     percentages, gains2 = cumulative_gain_curve(y_true, y_probas[:, 1],

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

please guide where I have done the mistake


texts: Imbalanced classes (balance of train, validation, and test)

texts:
1) I am currently trying to set up a feedforward neural network with highly imbalanced classes (binary classification) in which the number of observations of class 1 is very low (and the class of interest to predict).

The dataset is very large, so in order to make the network better to predict class 1, I have downsampled the class 0 observations in the training set (ending up with a training set of approximately 600,000 observations).

Right now, I am estimating the neural network using 'accuracy' as the metric to optimize, while using early stopping to optimize the accuracy on an unbalanced validation set (monitor='val_acc').

My question is, therefore, whether I also should downsample class 1 observations in the validation set? The final test will naturally be unbalanced with a low number of class 1 observations.

Edit:
My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network.
After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved).
My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced

2) Additionally, is there any common order on how to optimize a neural network (etc. first determine the number of hidden layers, number of neurons, and so)? A paper or similar arguing for its order of optimization would be very interesting (haven't managed to find one myself).

Thanks in advance!


texts: Adding an extra dimension in weight matrix in Tensorflow

texts: I was reading styleGAN2 code, in the networks_stylegan2.py file at line 95, they have added an extra dimension in the weight matrix for incorporating mini-batch. What I know is that TensorFlow can handle the mini-batch automatically if we define the first dimension in input as "None" and there is no need to add an extra dimension weight matrix for the mini-batch. So why they have added an extra dimension in the weight matrix?


texts: Does an RL agent still learn if its actions are "blocked"?

texts:
Say we have a game that is a maze environment where there is a character to be controlled through the maze. When the agent (the character) approaches a wall, it may try to execute an action that would lead it into the wall, which is not permitted. Instead of executing this action, we place an if statement in the code that checks if the action is permitted, and if not, does not execute the action, and simply proceeds to the next state.

Another similar example is if an RL agent is being trained in a stock trading environment, where say it tries to sell more stock than it actually owns, or buy less than the minimum amount. Similarly as before, we place an if statement that checks for these conditions and either allows the action (allows the trade) or moves on to the next state.

Does the agent still learn, even if we "override" it and block certain actions? How else would we go about restricting the agent to certain actions in certain states e.g. if a wall is to the left of the agent in the game environment, how would we restrict the agent to only move forward, backward or right, but not left?


texts:
Say I have a training dataset composed by 128 1-Dimensional time series in form of numpy arrays.

They all correspond to a certain action that I label action_1 and I want to recognize.

What would be the most efficient way to accomplish the following taks:

Train a model with the training dataset (the 128 1-D numpy arrays)
Ask the model to predict the action in a new test entry (a 1-D numpy array)

This is my very first attempt at Machine Learning, thank you in advance for any indications.


texts: What are some general tips to improve my MNIST classifier?

texts: I have built a CNN from scratch in python using Numpy, to tackle the MNIST hand-written digit recognition problem. It's composed out of a convolutional layer (3 3x3 filters), a maxpooling layer (2x2 pooling) and the 10-label output layer. I'm using softmax for the output activation function and cross-entropy as loss function. I've tried running it with a couple of different hyperparameters and so far the best accuracy i've gotten is 97%, when training on the whole train dataset (60000 images) for a single epoch and using SGD. The accuracy does vary a bit though, usually around 92-95% under these conditions. I've only tried one epoch when using the whole dataset because it already takes maybe 15 min for my algorithm to train using 60000 images (with cpu on my low grade school laptop). The thing is that I don't have an perception of how good/bad this is, i.e. how much time one would expect such an network to take and how accurate it would be. Is this really slow and inaccurate? I'd love some general tips on how I can improve my network, whether it's through some optimization methods or brute force (increased amount of layers/neurons). I've also tried implementing mini-batches but for some reason (maybe faulty implementation?) this only seems to decrease the accuracy.


texts: Audio signal signal processing for background noise reduction or removal

texts:
I am performing simple audio recognition using tensor-flow to spot key word or hot word detection. The graph takes the microphone input directly and performs "Audio spectrometer --> mfcc's --> and input to the neural network."

Now i would to like to perform some kind of real time back ground noise reduction or cancellation before inputting the mfcc's into the neural network.

I am not sure what kind of algorithms/python packages that are helpful to reduce the back ground noise and If some one can point me to the pieces of code/knowledge base that can help to filter the mfcc's before inputting into neural network.


texts: How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

texts: I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(specifically STALE FILE HANDLEs for now in GPFS file system). I am categorizing the log files and clustering based on their instances per time interval. Since some messages are more predominant over the others in any given time frame than the others, i don’t want the clustering to bias towards the one with maximum variance.


texts: Extract average embedding for class labels

texts:
I am training a neural net to predict class labels for documents. The first layer in my network is an embedding layer.
Once the network is fully trained, I would like to extract an a single embedding which best represents each class label.
A naive and indirect solution would be to feed the documents back in, and calculating an average of each class.
Is there a more direct approach?


texts: Estimating the battery capacity using current power consumption and battery percentage

texts:
I want to estimate the current maximum capacity (in kWh) having the current power consumption (in kWh) and the state of charge of the battery (in %) available in a time series.

I do not have a full battery charge circle recorded but only a snippet with the state of charge going from ~95% to ~35% in a 1 second data recording interval.

At first, i cumulated the current power consumption and noticed that the progression between cumulated power consumption and state of charge was almost the same (see figure 1 and 2).

progression of state of charge vs. cumulated power consumption

So i tried to use a linear regression model to predict two values of cumulated power consumption at 0% and 100% state of charge. In my assumption the delta of these two leaves me with the current maximum capacity of the battery.

Ideally i want to monitor the capacity during the data recording. Which means i only have data of a few percent of state of charge.
In figure 3 and 4 you can see my attempt of trying to create a linear regression model for every 2% state of charge. As you can see in the right plot, the estimated capacity has a very high variance.

LEFT: regression lines (green) of every 2% state of charge; RIGHT: estimated capacity of every 2% state of charge

Since i am not an expert in neither the matter of regression nor batteries/physics, my questions are:

Is there a much simple or more exact way (or both) to estimate the current maximum capacity of the battery?
If yes, is there a good way to estimate an sufficiently exact capacity with an even smaller snippet in an ongoing process, lets say every 2% state of charge? (maybe using a characteristic curve of the discharge process)

Here is a sample of my data: sample file


texts: Theano/Lasagne/Nolearn Neural Network Image Input

texts:
I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype.
All standard examples like MNIST numbers classification run well, but problems appear when I try to work with my own images.

I want to use 3-channel images, not grayscale.
And there is the code where I'm trying to get arrays from images:

 img = Image.open(item)
 img = ImageOps.fit(img, (256, 256), Image.ANTIALIAS)
 img = np.asarray(img, dtype = 'float64') / 255.
 img = img.transpose(2,0,1).reshape(3, 256, 256)
 X.append(img)

Here is the code of NN and its fitting:

X, y = simple_load("new")

X = np.array(X)
y = np.array(y)

net1 = NeuralNet(
    layers=[  # three layers: one hidden layer
        ('input', layers.InputLayer),
        ('hidden', layers.DenseLayer),
        ('output', layers.DenseLayer),
        ],
    # layer parameters:
    input_shape=(None, 65536),  # 96x96 input pixels per batch
    hidden_num_units=100,  # number of units in hidden layer
    output_nonlinearity=None,  # output layer uses identity function
    output_num_units=len(y),  # 30 target values

    # optimization method:
    update=nesterov_momentum,
    update_learning_rate=0.01,
    update_momentum=0.9,

    regression=True,  # flag to indicate we're dealing with regression problem

       max_epochs=400,  # we want to train this many epochs
        verbose=1,
        )

  net1.fit(X, y)

I recieve exceptions like this one:

Traceback (most recent call last):
  File "las_mnist.py", line 39, in <module>
    net1.fit(X[i], y[i])
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 266, in fit
    self.train_loop(X, y)
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 273, in train_loop
    X, y, self.eval_size)
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 377, in train_test_split
    kf = KFold(y.shape[0], round(1. / eval_size))
IndexError: tuple index out of range

So, in which format do you "feed" your networks with image data?
Thanks for answers or any tips!


texts: Loss function when the output is a single probability

texts:
I have a regression problem where the output y is a single probability, i.e. real number that varies in the interval [0, 1]

While using L1 or L2 loss will very likely work well, I feel that they are not the most appropriate options considering that the range [0, 1] is already well defined.

Is Binary Cross Entropy (BCE Loss in pytorch) the most appropriate in this case?


texts:
When I do a linear regression, R²: 0.90, but the estimates are not correct, why is this happening?

(Deep Not : Adjusted R-squared:  -0.3872)


texts: How to know whether machine learning is possible for a given data set. I have been given a data set, I should check whether machine learning is possible or not for that data set. How can I do that. How do you come to conclusion that machine learning can be performed for the given data set or not.


texts:
Is there any good reference for methodologies on quality assessment of machine learning algorithms design. The scope is wide and embraces each steps from data preparation to generalisability testing.

Further, are there any automated "stress-test tools" for ensuring that the algorithms is not biased, that it fits the alledged performances?


texts: Dataset availability for automatic text summarization

texts:
I'm working on an automatic text summarization NLP problem and looking for a dataset with USA legal case reports similar to the Australian legal case reports dataset in UCI repository.

Can you please refer me to any such dataset? I've not been able to find one up until now. It will also be great if you can point me to other industry-relevant datasets that can be used for automatic text summarization.


texts: Similarity Measure of Simulated Time Series vs Observed time Series

texts:
In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters simulate the Light Curve the best.

At the moment I do it with the Cross-Correlation function from numpy. But I am not sure if that is the best option, due to the fact that the Light Curve with the highest Cross-Correlation Coefficient not always looks like the best fit/simulation compared to other simulations with a lower CC-Coefficient. Is there a another way to measure for similarity? I read something about the Chi-Square Statistics, but I am not sure how that works and how this could be applied to my problem.

The observation data I use is not evenly binned, so I used the interpolation function of Scipy. Should I also smooth the observation data or would I lose true features of my data. I thought about using the savitzky golay smoothing.

At the moment I am using a brute force method to try out all possible parameters and simulated the corresponding light curve. The problem is this takes a lot of time with 20 parameters. The parameters are more or less dependent on each other. So I cant use a least-square fit method, because there are multiple possible minimas. Is there a simple method that I overlooked. Or is a restricted brute force fit my best option?

In the picture below you'll see one plot with the simulation and the observaton data.

Thanks for all suggestions.


texts: Re-bucket weekly sales data and calculate descriptive statistics

texts:
I've got sales data in weekly buckets like this:

weekID product SoldQty
1      1       10
2      1       20
3      1       30
4      1       40
5      1       50
6      1       60
7      1       70

1      2       10
2      2       20

Calculating st.dev. of weekly sales per product is pretty straightforward.

Now I'm asking the question: how to calculate the St.dev for the same data, but bi-weekly bucketed? x-weekly bucketed?

Question 2: is there an efficient algorithm for calculating it on weekly data instead of materializing the x-weekly combinations?

From the business side it means that I have various forecast horizons (1wk, 4 wks, 6wks...) per product. And I want to build the confidence intervals for predictions of SoldQty within the forecast horizon.
It all seems very similar to the Safety Stock calculations from logistics but I want to be sure.

UPDATE:

Please consider there are multiple ways to re-bucket (re-combine) this data. I think that if the new bucket contains W weeks, there are W ways to re-bucket data. For an example of 2-weekly buckets.

Way 1

bucket wkid prod sold
1      1    1    10
1      2    1    20
2      3    1    30
2      4    1    40
3      5    1    50
3      6    1    60
4      7    1    70
...

Way 2

bucket wkid prod sold
1      1    1    10
2      2    1    20
2      3    1    30
3      4    1    40
3      5    1    50
4      6    1    60
4      7    1    70
...

Both options make sense business-wise. Essentially you need to calculate st.deviations between sales during fortnights. And, st.deviation in way 2 should be different from way 1.

Thanks in advance for your suggestions!
Constantine.


texts: Sliding window Algorithm and its convolutional implementation

texts:
I want to know why the convolution implementation of the sliding windows is equivalent to the sequential step-by-step sliding window? Why are they the same thing?

I'm following Andrew NG for this:
What I understand is the first row in the picture is supposed to be a Convolution Network which is trained on 4 classes.

The second row is supposed to be 3 channel 16x16 image that is convolved by the trained convolution network from the row one (which is 16 5x5 filters -> 2x2 Max Pool -> 400 5x5 filters -> 400 1x1 filter -> 4 1x1 filter), the tricky part as I understood is that the output is the equivalent to sliding 14x14 window to on 16x16 image which has only 4 positions, they are 2x2x4 outputs, each cell in the 2x2 grid corresponds to a position of the 14x14 window on the 16x16 image.

The Last row is another example but with bigger 28x28 image with the same window size 14x14.


texts: k-means in R, usage of nstart parameter?

texts:
I try to use k-means clusters (using SQLserver + R), and it seems that my model is not stable : each time I run the k-means algorithm, it finds different clusters. But if I set nstart (in R k-means function) high enough (10 or more) it becomes stable.

The default value for this parameter is 1 but it seems that setting it to a higher value (25) is recommended (I think I saw somewhere in the documentation).

So I'm a bit confused...
Any advice ?


texts: How to restrict the values of predicted variable to be positive?

texts: I am using Python Linear Regression to predict the weekly orders for a food deliver company. But some of my orders are coming out as negative. Is there any way to restrict the predicted values to be greater than 0,i.e positive?


texts:
I'm working on an anomaly detection development in Python.
More in details, I need to analysed timeseries in order to check if anomalies are present.
An anomalous value is typically a peak, so a value very high or very low compared to other values.

The main idea is to predict timeseries values and, using thresholds, detect anomalies.

Thresholds are calculated using the error, that is the real values minus the predicted ones.
Then, mean and standard deviation of the error are performed.

The upper threshold is equals to mean + (5 * standard deviation).
The lower threshold is equals to mean - (5 * standard deviation).

If the error exceeds thresholds is marked as anomalous.

What doesn't work with this approach is that if I have more than one anomalous value in a day, they are not detected. This because error, mean and standard deviation are too much influenced by anomalous values.

How can i fix this problem? Is there another method that i can use to identify thresholds without this issue?

Thank you


texts: I am having a problem trying to reshape my numpy array. i have a 2 dimensional array and i want to make it a 3D array. I have looked everywhere but even if i find some answering my question they just make it complicated. I just want to know is there a mathematical way to calculate reshaping a numpy.


texts: How to save a lightGBM model that updates predictions after each fold?

texts:
Hi When I use gradient boosting on Kaggle and large data sets, I run a code like this:

%%time
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)
oof = np.zeros(len(train))
predictions = np.zeros(len(test))
predictions_train = np.zeros(len(train))
feature_importance_df = pd.DataFrame()

start = time.time()

for fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):
    print("fold n°{}".format(fold_))
    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx])# categorical_feature=categorical_feats)
    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx])#, categorical_feature=categorical_feats)

    num_round = 10000
    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 200)
    oof[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)

    fold_importance_df = pd.DataFrame()
    fold_importance_df["feature"] = features
    fold_importance_df["importance"] = clf.feature_importance()
    fold_importance_df["fold"] = fold_ + 1
    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)

    predictions += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits
    predictions_train += clf.predict(train[features], num_iteration=clf.best_iteration) / folds.n_splits
    feature_importance_df.to_csv('feature_importance.csv',index=False)
pd.to_pickle(predictions_train,'X.pkl')
pd.to_pickle(predictions,'predictions.pkl')
print("CV score: {:<8.5f}".format(log_loss(target, oof)))

This produces good results and prevents overfitting, as I understand it the predictions are updated after each fold fitting.
But is there an easy way to save this model for deployment.

As I understand it I could save the model like:

pd.to_pickle('model_fold_{}.pkl'.format(fold_),clf)
pd.to_pickle('model_best_iteration_{}.pkl'.format(fold_),clf.best_iteration)

and then load them all in, and then have a deployment script, concatenating each model on top of each other, so 5 models loaded in.

Is there a simpler way to do this?


texts: Performing a linear regression with Perceptron

texts:
I was wondering about the link between the linear regression and the perceptron!
Perceptrons were used as binary classifiers i.e to classify binary labels ( 0 or 1 ).
My question is How can you perform a linear regression with perceptron and what is the link?
I think the point is hidden in the activation function which we are applying to the perceptron!
I would be appreciated if you can help me with the beginner's perspective on Neural Networks.


texts: Poor performance shown on Rare event modeling

texts:
I am working on a Rare event classification problem. I Have 95% of the data as a majority class and 5% of the data as the minority class. I use classification trees algorithm. I am measuring the goodness of the model using confusion matrix.

As the i have the minority class just 5% of the total data, even though my prediction performance of minority class is close to 70%, the total number of errors are high.

For example, here is my confusion matrix.
             0           1
     0     213812      7008
     1     29083       16877

Though the Minority class(class 1) has predicted 16877 times correctly(70% and the misclassifcation is just 30%, but the absolute value of the misclassifcation is very high(29083) comparing to the correctly predicted minotriy class (16877). Which makes the solution less usable for the business.

Is there any idea on handling these kind of  issues in such rare event modelling.

Kind note: I have  balanced the target variable using the SMOTE algorithm before applying Classification tree.


texts: Cross validation schema for imbalanced dataset

texts:
Based on a previous post, I understand the need to ensure that the validation folds during the CV process have the same imbalanced distribution as the original dataset when training a binary classification model with imbalance dataset. My question is regarding the best training schema.
Let’s assume that I have an imbalanced dataset with 5M samples where 90% are pos class vs 10% neg class, and I am going to use 5-folds CV for model tuning. Also, let’s assume I will hold out a random 100K samples for test (90K samples w/ pos class vs 10K samples w/ neg class). Now I have two options:
Option 1)

Step 1: Pull a randomly selected 200K imbalanced data for training (180K samples pos class vs 20K samples neg class)
Step 2: During each CV iteration:

The training fold will have 160K samples (144K pos vs 16K neg)
and the validation fold will have 40K samples (36K pos vs 4K neg)

Step 3: Apply data balancing for the training fold (e.g., Downsampling, Upsampling, SMOTE, etc.) and fit a model
Step 4: Validate the model on the imbalanced training fold

However, given that I have enough data, I want to avoid using any data balancing algorithm for the training folds.
Option 2)

Step 1: Pull a randomly selected 200K balanced data for training (100K samples pos class vs 100K samples neg class)
Step 2: During each CV iteration:

The training fold will have 160K samples (80K pos vs 80K neg)
and the validation fold will have 40K samples (20K pos vs 20K neg)

Step 3: Fit a model for the already balanced training fold
Step 4: Can I apply down sampling to the balanced validation dataset to restore it to its imbalanced state? If so, how can I do that in sklearn?

I am also clear that I have a 3rd option, which is based on the 1st option above, where the model could be trained on an imbalanced dataset. Therefore, a data balancing algorithm can be avoided.
My questions are:

Is option 2 better than option 1?
How to apply a downsampling to a balanced validation dataset (Option 2-step 4


texts: How to split train/test data 50% by class and grouping by Object ID in R?

texts:
I get pixel values from it using reference polygons. Extracted pixel values are in data frame, but one row represent extracted values for single pixel. In the classification I need to split the dataset into test (50%) and training (50%) by class (tree, meadow e.t.c)

I know how to split a set according to classes. However, I want values extracted for one polygon to be assigned to one of the sets (training OR test ) and they were not mixed

For this purpose I want to use the polygon ID (Object Identification).
I would like to do this using the createDataPartition function. These are just two sample classes (there are many more)

Here is part of table with extracted values:

"band_1"    "band_2"    "band_3"    "CLASS" "Id"
110 134 119 "tree"  1
112 133 118 "tree"  1
105 125 110 "tree"  2
112 132 117 "tree"  2
109 125 115 "meadow"    6
93  110 101 "meadow"    6
86  106 95  "meadow"    7
105 136 116 "meadow"    7
102 128 111 "meadow"    8
108 129 115 "meadow"    8
113 134 119 "meadow"    8

Here is code:

trainIndeks <- caret::createDataPartition(EXTRACTED$CLASS, p = 0.5, list=FALSE, times = 1)

    dataTrain <- EXTRACTED[trainIndeks,]
    dataTest <- EXTRACTED[-trainIndeks,]


texts:
I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the timestamps are not synchronized). In total this means about 20k columns and hundreds of millions of rows. So far we have accumulated about 5TB of data on our local machine.
We would like to be able to train neural networks on this datasets which means we would need to be able to read random/non sequential individual rows or chunks of rows which are between 2 time indexes in to memory very fast.
I have never worked with mySQL but from reading about it, it seems it could be good for this task. I have however seen no examples of using mySQL with pytorch or pytorch Dataset. Is there a reason for this?
What is the conventional way that this task would be accomplished?


texts: Value error in Merging two different models in keras

texts:
I am following the code below with all the libraries imported:

mergedOut = Add()([model1.output,model2.output])

mergedOut = Flatten()(mergedOut)

mergedOut = Dense(256, activation='relu')(mergedOut)

mergedOut = Dropout(.5)(mergedOut)

mergedOut = Dense(128, activation='relu')(mergedOut)

mergedOut = Dropout(.35)(mergedOut)

# output layer

mergedOut = Dense(5, activation='softmax')(mergedOut)

newModel = Model([model1.input,model2.input], mergedOut)

newModel.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
newModel.fit(x_train, y_train, batch_size=100, epochs=5)

However, I am dealing with the following error everytime:

  ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[-0.36227286, -0.0600015 , -0.14906043, ..., -0.28826913,
           -0.16141416, -0.16673401]],
  [[-0.36227286, -0.05597525, -0.14903974, ..., -0.30347877,
            0.11033272,  0.4293...


texts: What does a negative coefficient of determination mean for evaluating ridge regression?

texts:
Judging by the negative result being displayed from my ridge.score() I am guessing that I am doing something wrong. Maybe someone could point me in the right direction?

# Create a practice data set for exploring Ridge Regression

data_2 = np.array([[1, 2, 0], [3, 4, 1], [5, 6, 0], [1, 3, 1],
           [3, 5, 1], [1, 7, 0], [1, 8, 1]], dtype=np.float64)

# Separate X and Y

x_2 = data_2[:, [0, 1]]
y_2 = data_2[:, 2]

# Train Test Split
x_2_train, x_2_test, y_2_train, y_2_test = train_test_split(x_2, y_2, random_state=0)

# Scale the training data
scaler_2 = StandardScaler()
scaler_2.fit(x_2_train)
x_2_transformed = scaler_2.transform(x_2_train)

# Ridge Regression
ridge_2 = Ridge().fit(x_2_transformed, y_2_train)
x_2_test_scaled = scaler_2.transform(x_2_test)
ridge_2.score(x_2_test_scaled, y_2_test)

Output is: -4.47

EDIT: From reading the scikit learn docs this value is the R$^2$ value. I guess the question is though, how do we interpret this?


texts: Guidelines to efficiently train neural networks which have polynomial activations

texts:
I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training time reduces by orders of magnitude. I am aware of the issue of exploding/vanishing gradients on using polynomials as activation functions, but by making the input activations small by using some kind of normalization on the dataset, we can circumvent the problem.
I have come across the following post on stackexchange discussion : https://stats.stackexchange.com/questions/299227/polynomial-approximations-of-nonlinearities-in-neural-networks
which touches this subject briefly. Although it doesn't have anything concrete in it.
I came across few papers like,
https://openreview.net/forum?id=rkxsgkHKvH
but this paper got rejected, so I am confident of their results. I am reaching out to the community for practises/papers in such a scenario.


texts: Are convolutions in deep learning associative?

texts:
Let's denote "convolution in deep learing" as "convolution-deep", and "convolution in math or signal processing" as "convolution-math".
As we all know, convolution-deep are actually cross-correlation, and cross-correlation is not associative while convolution-math is associative.
Therefore, is the formulations listed followed true?
Suppose W1 and W2 are weight tensors of two convolution layers or filters for cross-coorelation in sequence, X and Y are the input and output, respectively.

Y1=conv-deep( conv-deep(X, W1), W2)==conv-deep(X, conv-deep(W1, W2))?
Y2=conv-math( conv-math(X, W1), W2)==conv-math(X, conv-math(W1, W2))?