texts: what is the difference between euclidean distance and RMSE?

texts:
I'm searching for a loss function that fits my Project. Actually I have two question but they are in the same direction. I take a look at the definition of the root mean squared error and the euclidean distance and they look the same to me! that's why I want to know what's the difference between the two. what would be the difference if I use rmse as a loss function or the euclidean distance??

the second question is how to search for a loss function. I mean I know it depends on the problem and commun things are MSE for Regression and Cross entropy for Classification but let's say I have a specific problem, how do I search for a loss function? I also saw that some people use a custom loss function and most of the deep learning frameworks allows us to define a custom loss function but why would I want to use a custom one? how I get the intuition that I need a custom loss function?

now to explain my problem. I'm doing a project where I need to reduce the GPS Error of a vehicle (I have some vehicle data and my neural network will try to predict the longitude and latitude so it's a regression problem) that's why I get the Idea of maybe the euclidean distance would make sense as a loss function, right? now somehow MSE also make sense to me because it is getting the difference between prediction and ground truth. does this make sense to you as a professional ML Engineer or Data scientist ? and if there would be a custom loss function that you can use, what would you suggest and why?


texts: How to professionally spell library names such as "scikit learn"?

texts:
For example for inclusion in a CV. This is mostly a question of whether one should use capital letters or not.
Usually I search the library to find how the library name is spelled on the official page.
For scikit-learn it is spelled without capital letter, while most of the other letter has some capital letter at the beginning (and often inside as well). Still it doesn't feel right next to other capitalised names.
Should I still use a capital "Scikit-Learn" or am I overthinking this?


texts:
Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores. Now I'm looking at scaling up to clusters on EC2 (probably with StarCluster/IPython, but open to other suggestions as well), and have been puzzled by how to reconcile distributing work across cores on an instance vs. instances on a cluster.

Is it even practical to parallelize across instances as well as across cores on each instance? If so, can anyone give a quick rundown of the pros + cons of running many instances with few cores each vs. a few instances with many cores? Is there a rule of thumb for choosing the right ratio of instances to cores per instance?

Bandwidth and RAM are non-trivial concerns in my projects, but it's easy to spot when those are the bottlenecks and readjust. It's much harder, I'd imagine, to benchmark the right mix of cores to instances without repeated testing, and my projects vary too much for any single test to apply to all circumstances. Thanks in advance, and if I've just failed to google this one properly, feel free to point me to the right answer somewhere else!


texts: How do I simultaneously select multiple values for k-means in WEKA?

texts:
I have tried WEKA's Experimenter. However, it's for classification.

I'm looking for a way to apply the k-means algorithm on the same dataset but with multiple 'k' values. Is there any option in WEKA's GUI allow this?


texts:
I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.

The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.


texts: Schedule Sagemaker with boto3 comprehend

texts: I have a jupyter notebook running on SageMaker which I have to run daily to output CSV files. I am using boto3 'comprehend' to analyze the text in my workflow. I want to be able to schedule this daily and get daily CSV files as output. Is there any way around it?


texts: Is providing class weight to neural network enough for imbalanced binary classification?

texts:
I have a highly imbalanced binary classification problem, probably 95:5 for two classes. I don't want to perform resampling as the data is already huge and training it would just take more time. (I'm also aware of down sampling)
But my question is , is providing class weights (let's say computed by scikit-learn's compute class weight) enough?
or there is any other method ?
model.fit(X,y,class_weight=class_weight)


texts: training and Predicting probabilities using logistic regression model

texts: I have these game data where the output variable is continous which indicates probability of winning. How do I train my classification model using this probabilites and predict output probabilites for test data


texts: What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of bedrooms, living rooms and bathrooms. I want to identify outliers in this data, where some of the variables are discrete and some of them are continuous. Any ideas on how to do this?


texts: What algorithms can handle probabilistic targets?

texts:
I have a classification problem where I want to want to use probabilities instead of classes to train my model to learn to output probabilities. In my dataset, I have instances where the probabilities of two classes are almost equal and I would like the model to be able to learn these subtleties instead of me just providing the class for each instance. Is there an ML model that can handle this?
Thanks!


texts: statistics or robust statistics for identifying multivariate outliers

texts: For the single variate data sets, we can use some straightforward methods, such as box plot or [5%, 95%] quantile to identify outliers. For multivariate data sets, are there any statistics that can be used to identify outliers?


texts: Condense matrix of values into one column?

texts:
I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make it easier to manage. I'm familiar with basic R stuff but not an expert by any means so please be patient!

Each R(1-10) corresponds to a response rating for the question (q1-10). The questions are randomised in order for each trial for each participant, so the qnum doesn't refer to the same question every time. I want to condense the matrix of response ratings down so that there is a single column called 'response' which reflects the rating given by the participant for each qnum for each trialNumber, so I don't have unnecessarily repeated tokens! Essentially going down the diagonal and reshaping it into one column. How can I do this?


texts: How to find the ranges in Equal frequency/depth binning?

texts:
I have been looking into the site http://www.saedsayad.com/unsupervised_binning.htm and there it shows range values to the right under equal frequency binning ....

I have so much looked to find how the ranges are formed but has been in vain. So , can somebody explain how those ranges are formed for the given dataset or explain the freq binning procedure in a good manner.

Great thanx for any help on this!


texts: How to use mean IoU for RGB mask (keras implementation)?

texts: I am training pix2pix GAN for converting SAR satellite images to segmentation mask. But I am not aware about how to use the mean IoU to evaluate my model. My output is of (320, 320, 3) shape and classes in ground truth are 5.  I would like to find a Keras implementation.


texts: Implement gaussian mixture model with stochastic variational inference

texts:
I am trying to implement Gaussian Mixture model with stochastic variational inference, following this paper.

This is the pgm of Gaussian Mixture.

According to the paper, the full algorithm of stochastic variational inference is:

And I am still very confused of the method to scale it to GMM.

First, I thought the local variational parameter is just $q_z$ and others are all global parameters. Please correct me if I was wrong. What does the step 6 mean by as though Xi is replicated by N times? What am I supposed to do to achieve this?

Could you please help me with this? Thanks in advance!


texts:
The thing that I am trying to do is the time series shapes classification.

Basically the problem is as following:

Let's say I have some time series and my goal is to have an algorithm that "finds" the similar shapes in the data, that occur throughout the dataset and labels it accordingly (a, b, c...etc.)...The image might be worth than 1000 words, so I have attached the workflow that I have in mind

It would be great if anyone directs me to the algorithms/papers/codes that deal with this kind of problems


texts: Spacy v2.0.1 custom NER: How to improve training of existing model

texts:
I implemented custom NER with bellow trained data first time and it gives me good prediction with Name and PrdName. I mentioned code bellow.

if __name__ == '__main__':
TRAIN_DATA = [
            ('My Name is Rajesh', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Bakul', {'entities': [(11, 16, 'Name')]}),
            ('My Name is Pritam', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Rakesh', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Jayeeta', {'entities': [(11, 18, 'Name')]}),
            ('this is the price of bag', {'entities': [(21, 24, 'PrdName')]}),
            ('what is the price of ball?', {'entities': [(21, 25, 'PrdName')]}),
            ('what is the price of jegging?', {'entities': [(21, 28, 'PrdName')]}),
            ('what is the price of t-shirt?', {'entities': [(21, 28, 'PrdName')]}),
              ]

iterations = 20
try:
    model = 'live_ner_model'
    nlp = spacy.load(model)  # load existing spacy model
except:
    model = None
    print("Exception")
    nlp = spacy.blank('en')  # create blank Language class
    print("Created blank 'en' model")

if 'ner' not in nlp.pipe_names:
    ner = nlp.create_pipe('ner')
    nlp.add_pipe(ner)
    print("Create NER")
else:
    ner = nlp.get_pipe('ner')
    print("Exhisting NER")

# Add new entity labels to entity recognizer
for _, annotations in TRAIN_DATA:
    for ent in annotations.get('entities'):
        ner.add_label(ent[2])

# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
with nlp.disable_pipes(*other_pipes):  # only train NER
    optimizer = nlp.begin_training()
    for itn in range(iterations):
        print("Statring iteration " + str(itn))
        random.shuffle(TRAIN_DATA)
        losses = {}
        for text, annotations in TRAIN_DATA:
            nlp.update(
                [text],  # batch of texts
                [annotations],  # batch of annotations
                drop=0.2,  # dropout - make it harder to memorise data
                sgd=optimizer,  # callable to update weights
                losses=losses)
        print(losses)

# Save model
output_dir = 'live_ner_model'
if output_dir is not None:
    output_dir = Path(output_dir)
    if not output_dir.exists():
        output_dir.mkdir()
    nlp.meta['name'] = model  # rename model
    nlp.to_disk(output_dir)
    print("Saved model to", output_dir)

# Test the saved model
output_dir = 'live_ner_model'
print("Loading from", output_dir)

nlp2 = spacy.load('live_ner_model')
test_text = """
   what is the price of cup. My Name is Rahim
"""
doc2 = nlp2(test_text)
for ent in doc2.ents:
    print(ent.label_, ent.text)

But when I am trying to trained with some new data which has entity with only PrdName or any other new entity excluding Name in existing model.
Then Name entity prediction goes wrong. I think this issue arises as I updated trained data excluding Name entity.

So is there any way we can improve training by not affecting existing training. Can someone share the idea? If possible please share a sample code.

Environment: Anaconda, spacy=v2.0.1, python=3.7


texts: Difference between using RMSE and nDCG to evaluate Recommender Systems

texts: What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would be great as well!


texts: Adding and multiplying higher values based on different columns of a dataframe

texts:
I am trying to calculate scores based on different values in the dataframe. Since these scores are based on different conditions, I am having problems where in the final calculation, which I need in a new column, where I need to choose the higher number in one of the column(Col E) if there are similar values in two columns(Col ID & VID) and unique in one column(Col QID).

ID     VID     QID     A     B     C     D     E
121    212     123     1     2     1     1     1
121    212     435     1     2     1     1     5
223    244     567     2     3     5     1     2
313    232     709     5     1     2     1     3
313    232     887     5     1     2     1     2
454    969     457     1     3     2     2     4
454    969     457     1     2     1     2     4

The last row showcases that Col ID, VID, QID and E can be same but since Col A, B, C and D are different, it will result in different score. The multiplication of Columns A, B, C, D and E (the higher value) should be in the Col Score. The result is expected like below:

ID     VID     QID     A     B     C     D     E     Score
121    212     123     1     2     1     1     1     2
121    212     435     1     2     1     1     5     10
223    244     567     2     3     5     1     2     60
313    232     709     5     1     2     1     3     30
313    232     887     5     1     2     1     2     20
454    969     457     1     3     2     2     4     48
454    969     457     1     2     1     2     4     16

Calculation goes like Columns A * B * C * D * E. For calculating the Score based on the similar ID and VID but unique QID.

The higher value in Col E can be first or last. If it is possible via groupby and then merging them to get this result above then that also solves the purpose.

I have tried .sort in order to bring Col E in a descending or ascending format and then calculating but couldn't write the logic behind the calculation. Just a beginner trying to work this problem for few days now.

Thanks in advance!


texts: convert keras pickle subclassed model to tflite model

texts:
I have generated a model by running this tensorflow project https://github.com/cadia-lvl/punctuation-prediction/tree/master/punctuator2tf2
the generated model is a pickle file. I am able to load the pickle file and run predictions for this model. This is a subclassed keras model. code can be found in models.py in above repository
def punctuate(word_vocabulary, punctuation_vocabulary, reverse_punctuation_vocabulary, text, model):

if len(text) == 0:
    sys.exit("Input text from stdin missing.")

text = [w for w in text.split() if w not in punctuation_vocabulary] + [data.END]

i = 0

while True:

    subsequence = text[i:i+data.MAX_SEQUENCE_LEN]

    if len(subsequence) == 0:
        break

    converted_subsequence = [word_vocabulary.get(w, word_vocabulary[data.UNK]) for w in subsequence]

    y = predict(to_array(converted_subsequence), model)

    print(subsequence[0], end="")

    last_eos_idx = 0
    punctuations = []
    for y_t in y:

        p_i = np.argmax(tf.reshape(y_t, [-1]))
        punctuation = reverse_punctuation_vocabulary[p_i]

        punctuations.append(punctuation)

        if punctuation in data.EOS_TOKENS:
            last_eos_idx = len(punctuations) # we intentionally want the index of next element

    if subsequence[-1] == data.END:
        step = len(subsequence) - 1
    elif last_eos_idx != 0:
        step = last_eos_idx
    else:
        step = len(subsequence) - 1

    for j in range(step):
        print(" " + punctuations[j] + " " if punctuations[j] != data.SPACE else " ", end="")
        if j < step - 1:
            print(subsequence[1+j], end="")

    if subsequence[-1] == data.END:
        break

    i += step

main
if len(sys.argv) > 1:
    model_file = sys.argv[1]
else:
    sys.exit("Model file path argument missing")

vocab_len = len(data.read_vocabulary(data.WORD_VOCAB_FILE))
x_len = vocab_len if vocab_len < data.MAX_WORD_VOCABULARY_SIZE else data.MAX_WORD_VOCABULARY_SIZE + data.MIN_WORD_COUNT_IN_VOCAB
x = np.ones((x_len, main.MINIBATCH_SIZE)).astype(int)

print("Loading model parameters...")
net, _ = models.load(model_file, x)

print("Building model...")

word_vocabulary = net.x_vocabulary
punctuation_vocabulary = net.y_vocabulary
reverse_punctuation_vocabulary = {v:k for k,v in net.y_vocabulary.items()}

while True:
    text = input("\nTEXT: ")
    punctuate(word_vocabulary, punctuation_vocabulary, reverse_punctuation_vocabulary, text, net)

My goal is to use this model on mobile devices. But i cant use pickle model on mobile. How do i save this custom subclassed model into pb or h5 and then convert it into tflite model ?
Any help or nudge towards coreect direction is highly appreciated. I am a beginner and have not much idead regarding this.


texts: Why is backpropagation used for finding the loss gradient?

texts:
I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as little outside resources/help as possible and figure out every step and issue on my own. Now that I'm done, I'm comparing the differences between my approach and the solutions that were created by people who know what they're doing. One of these differences I've seen is the backpropagation algorithm, and I'm in the process of trying understanding it (but am struggling a bit).
While researching the algorithm, most sites describe it as a method of calculating the loss gradient of the network. However, the method that I used for calculating the loss gradient involves considerably less math and complexity - which I don't fully understand - and seems to accomplish the same thing:
$$Loss(w_0,w_1,w_2,...,b_0,b_1,b_2,...)$$
$$\nabla Loss(...)=\begin{bmatrix}{\frac{\partial Loss(...)}{\partial w_0}},{\frac{\partial Loss(...)}{\partial w_1}},...\end{bmatrix}$$
To figure out the gradient of this loss function, you can just tweak each parameter in the loss function and find out how much the loss changes. Dividing by how much you change it, you can get an approximation of the loss gradient at the current network state. Here is a rough outline of what I mean in code:
function LossGradient():
    current_loss = CalculateLoss()
    gradient_vector = []
    i = 0
    delta = 0.00001 # some really small value
    for weight in network:
        weight += delta # tweak weight by delta ( a tiny amount )
        new_loss = CalculateLoss() # calculate the loss after tweaking
        gradient_vector[i] = (new_loss - old_loss) / delta # store ∂Loss/∂W_i in gradient vector
        i += 1

    # do the same for biases

    return gradient_vector

My question is, what's the advantage of the backpropagation over this simple method that I used? I'm in the middle of struggling to understand the algorithm fully, and no resource seems to explain why and how exactly the algorithm is more efficient.
Any insight would be much appreciated.


texts:
I have a data from GPS in the form

1.('latitude', 'longitude','Timestamp').
2.('latitude', 'longitude','Timestamp').
3.('latitude', 'longitude','Timestamp').

I am changing this data into the below form

'latitude_1', 'longitude_1', 'Timestamp_1', 'latitude_2', 'longitude_2', 'Timestamp_2, Timestamp_2-Timestamp_1.

With this format I am training a LinearRegressionWithSGD model of spark where label is Timestamp_2-Timestamp_1 and features are latitude_1, longitude_1, latitude_2, longitude_2.

But when I am giving Origin ( latitude and longitude ) and destination ( latitude and longitude ) the results are very bad.

Kindly guide me whether this approach is the right approach ? and if not then how to build a prediction model from given data to predict Estimated Time of Arrival.


texts: Which performs better in time series forecasting, LSTM or SVR?

texts: I have run LSTM and SVR models on various datasets having sample values in the range of 1-4000 and the MAPE obtained in SVR was consistently lesser than that obtained through LSTM. I was told the reverse is true (that LSTM should perform better) but haven't found much information on this online. I would appreciate any feedback about this and any links to articles or papers (so far, I found grossly varied opinions).


texts: How do I best visualize this voltage data for a science project

texts:
I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the data!
Our original hypothoesis was that the lenght of the wire between the solar panel and the battery would effect the voltage charge the most. We do NOT think that was true!
We just need to get some good charts out of the data to show something that we learned!


texts: How did the phrase "big data" come about? What were some of the new challenges that arose? How does big data differ from a small amount of data and how do we distinguish the two?


texts: Best Method for Data Analysis on a 100 numerical IVs and 200 numerical DVs

texts:
I think I might need the help of this valuable community for a task. I have been given a dataset for 100 numerical independent variables (IVs) that predict output for 200 numerical values (from monte carlo simulation results). Which statistical technique should I start exploring and trying on my dataset? The number of observations can be increased for more points to enhance learning an algorithm. From this, I would like to learn a few insights, such as the multiple collinearity among IVs, their overall contribution to dependent variables.

Thank you very much.


texts: Prediction issue with xgboost custom loss

texts:
I have an issue with xgboost custom objectives: I do not manage to get consistent forecasts. In other words, the scale of my forecasts is not in line with the values I would like to predict. I tried many custom loss, but I always get the same issue.

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.datasets import make_regression

n_samples_train = 500
n_samples_test = 100
n_features = 200

X, y = make_regression(n_samples_train, n_features,noise=10)
X_test, y_test = make_regression(n_samples_test, n_features,noise=10)

param = {'verbosity' : 1,
      'max_depth' : 12,
      'learning_rate' : 0.01,
      'nthread' : 3,
        }

dtrain = xgb.DMatrix(X, y)

best_nrounds = 50

bst_reglinear = xgb.train(param,
                      dtrain,
                      best_nrounds)

def reg_obj(preds,dtrain):
    y = dtrain.get_label()
    N = len(y)
    #residual = (preds-y).astype("float")
    grad = 2*preds-y
    hess = 2*N*np.ones(len(y))
    return grad, hess

bst_custom = xgb.train(param,
                   dtrain,
                   best_nrounds,
                   obj = reg_obj)

dtest = xgb.DMatrix(X_test)

pred = bst_reglinear.predict(dtest)
print(np.abs(pred).mean())

pred_custom = bst_custom.predict(dtest)
print(np.abs(pred_custom).mean())


texts: Class weight adjustment in multi output model error

texts:
I'm trying to make a multi output model. I have 3 regression models (reg_1, reg_2, reg_3) and one classification model (cl_1) that classifies 5 classes (0,1,2,3,4). The classification data is not balanced, which is why I want to adjust the class weights.
My class weight parameters are:
class_weight = {"cl_1": {0: 1.837246564098982,
                         1: 0.6953029770433948,
                         2: 0.967228685544663,
                         3: 1.9681389026921166,
                         4: 0.6777320355514399
                         }
                }

I get an error when starting the model:
ValueError: Expected `class_weight` to be a dict with keys from 0 to one less than the number of classes, found {'cl_1: {0: 1.837246564098982, 1: 0.6953029770433948, 2: 0.967228685544663, 3: 1.9681389026921166, 4: 0.67773} }

I solved the problem by installing tensorflow version 2.1.0, since the same problem also occurred with version 2.2.0.
Is there any other solution using tensorflow 2.3.0?


texts: Training Machine Learning Model - Neural Network - Islands Problem

texts:
I was working on the following leetcode problem:

  Given a 2d grid map of '1's (land) and '0's (water), count the number
  of islands. An island is surrounded by water and is formed by
  connecting adjacent lands horizontally or vertically. You may assume
  all four edges of the grid are all surrounded by water.

  Example 1:

  Input: 11110 11010 11000 00000

  Output: 1

  Example 2:

  Input: 11000 11000 00100 00011

  Output: 3

And wrote up the following explicit solution:

from collections import OrderedDict

class Solution:
    def numIslands(self, grid: List[List[str]]) -> int:
        if len(grid) == 0:
            return 0

        land = ones(grid)
        parcels = len(land)
        islands = 0

        while parcels > 0:

            unexplored = []
            start = land.popitem()[0]
            parcels = parcels - 1

            print(start)

            up = (start[0] + 1, start[1])
            down = (start[0] - 1, start[1])
            left = (start[0], start[1] - 1)
            right = (start[0], start[1] + 1)

            if up in land:
                unexplored.append(up)
                del land[up]
            if down in land:
                unexplored.append(down)
                del land[down]
            if left in land:
                unexplored.append(left)
                del land[left]
            if right in land:
                unexplored.append(right)
                del land[right]

            while len(unexplored) > 0:

                start = unexplored.pop()
                parcels = parcels - 1

                up = (start[0] + 1, start[1])
                down = (start[0] - 1, start[1])
                left = (start[0], start[1] - 1)
                right = (start[0], start[1] + 1)

                if up in land:
                    unexplored.append(up)
                    del land[up]
                if down in land:
                    unexplored.append(down)
                    del land[down]
                if left in land:
                    unexplored.append(left)
                    del land[left]
                if right in land:
                    unexplored.append(right)
                    del land[right]

            islands += 1

        return islands

def ones(grid):
    ones = OrderedDict()
    height = len(grid)
    width = len(grid[0])

    for i in range(0,height):
        for j in range(0,width):
            if grid[i][j] == "1":
                ones[(i,j)] = 1

    return ones

But, my intuition said that finding the number of islands in a grid of ones and zeroes should be a good machine learning problem, associated with image detection.

So, I wrote the following code to generate test cases:

from random import randint

def make_grid():

    width = 30
    height = 30
    grid = []

    for i in range(0,height):
        row = []
        for j in range(0,width):
            if randint(0,1) == 0:
                row.append("0")
            else:
                row.append("1")
        grid.append(row)

    return grid

And, I then generated 60,000 cases and scored them using my explicit solution. I divided them into 50,000 train cases and 10,000 test cases.

I then tried to feed the train cases into a Keras network that was 64 nodes per layer, 10 layers deep using relu and a softmax activation function at the end. I trained the model for 250 iterations.

I got low accuracy statistics on recognizing the number of islands in the "images" on both the train and test cases.

I then tried to feed them into a Keras network using two convolutional layers, the first one 2x2 and the second one 4x4 with 64 nodes each followed by 2 dense layers with 64 nodes per layer. I trained the model for 3 iterations.

I got low accuracy statistics at predicting the number of islands in an "image" on both the train and test cases.

Is this just a bad problem for neural networks? Am I using the wrong kind of networks? Are my networks too small? Do I need to do a larger number of test cases?

Any input appreciated!


texts: Addition and subtraction of values of two columns when using Orange GUI

texts:
I am a new user to the Orange GUI.

I would like to know if there is anyway I can add or subtract two columns and then return the result as a new column?


texts: Why do we use different image processing at train and test time for an image classification tasl

texts:
I would like to ask why we use different image processing at train and test for an image classification task. For example, this pytorch tutorial uses RandomResizedCrop at train time and then Resize + CenterCrop at test time.

I just read this paper called Fixing the train-test resolution discrepancy and it says this has been the best practice.

However, I'm unable to figure out why nor find any documentation online as to why this is the case. Please help me shed some light on this. Thank you.


texts: How do companies like Amazon track what products are most frequently bought together?

texts: What methods could someone use to find out what products are most frequently grouped with each other per order? Are there applications that can make achieving this goal easier?


texts: Lower performace with same script on google cloud vs laptop

texts:
So I want to test a lot of hyperparameters for an xgboost classification model and also do cross validation for all of these. To do this I use a gridsearch. To speed up the process I want to use as many cpu cores as possible, so I set the n_jobs parameter to the number of available cpu cores in the system. This all works perfectly fine, see code below.
    xgb_model = XGBClassifier(use_label_encoder=False, eval_metric='auc')
    njobs = os.cpu_count()
    gsearch = GridSearchCV(estimator=xgb_model,
                           param_grid=param_tuning,
                           scoring='roc_auc',
                           cv=3,
                           n_jobs=njobs,
                           verbose=10)

Now when I run this on my laptop, it takes around 1 second per core to complete a training cycle at a speed of around 2 GHZ. When I run this exact same script on a Google cloud N2 VM with 8 cores it takes around 1 minute per core to do the same, even though these cores are faster than 2GHZ. That's 60 times slower? Does one of you have any idea why it is so much slower on google cloud?
I already tried upgrading the harddrive to an SSD but that seems to have no impact.


texts: Output data from scikit learn logistic regression

texts:
I have run a logistic regression using scikit learn in python.  I know want to output the results to put into a csv and then load into Tableau.  To do that I need to combine the y_test, y_actual, and X_test data.  I was wondering if there's a way to output the y_test, y_actual, and X_test data?  I know I can use to_csv, but when I try that I can extract each of these data and concat together, but I'm afraid its not matching correctly since there's no identifying to join on.

Alternatively, is there a way to keep a unique id (uid) with the logistic regression so then it's easy to see what the regression predicts for a specific person?


texts: Making Inference from a Correlation Heatmap

texts: I have constructed a Heatmap of my dataset for visualization. I have searched on various sites regarding what can I infer from the heatmap and I am unable to get any clear understanding from them. I want to select useful features for Exploratory Data Analysis and Visualization to get some insights. If you can provide any suggestion or advice it would be extremely helpful for me. Thank you


texts: Efficiency of a supercomputer. Alternatives to rent one to process large amount of data

texts:
I want to live from poker as I don't find a job as geologist, and I am making a trainning system to train myself. I need to process some data to calculate the mathematical chances for each 270.000 posible hands -the variety is omaha, permutations of 52 cards in groups of 4- against all the other posible hands of the poker deck, and my computer would take several months to process them -I am gonna say for posible calcles 20 months-.

I have an i5 with 16 GB RAM:

Arquitectura:                        x86_64
modo(s) de operación de las CPUs:    32-bit, 64-bit
Orden de bytes:                      Little Endian
CPU(s):                              4
On-line CPU(s) list:                 0-3
Hilo(s) de procesamiento por núcleo: 1
Núcleo(s) por «socket»:              4
Socket(s):                           1
Modo(s) NUMA:                        1
ID de fabricante:                    GenuineIntel
Familia de CPU:                      6
Modelo:                              158
Model name:                          Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
Revisión:                            9
CPU MHz:                             900.049
CPU max MHz:                         3500,0000
CPU min MHz:                         800,0000
BogoMIPS:                            6000.00
Virtualización:                      VT-x
Caché L1d:                           32K
Caché L1i:                           32K
Caché L2:                            256K
Caché L3:                            6144K
NUMA node0 CPU(s):                   0-3

I am interested in renting a supercomputer for an hour and I have the money to rent it, but it is an expensive cost for me.

  My question is double before deciding what to do:

  i) Would the supercomputer solve the calculations in a short time, being enough an hour of rental?

  ii) Are there better alternatives such as deriving the calculations to a server of a certain capacity paying a month?

If option ii) takes one or two weeks paying eg 100 $, I can wait a week or two, I will skip current step and continue doing calcles that my pc can do. But I wonder if the sc would solve it in a few minuts, so then, my project will continue, and I may prepare my hole trainning system scripts in some months and pay for their execution and generated data later.

I need help from someone that knows how fast are that computers to planify the continuation of my project, or provide a better/cheaper alternative.

Thanks in advance.


texts: Is there a different between one hot encoding and labelling categorical as scalar value and then normalise the batch?

texts:
Let's say I have label = ['Male', 'Female', 'Undefined'].
Is there a difference if I label it as:

[[1, 0, 0], [0, 1, 0], [0, 0, 1]]

compare to labeling it as

[0, 1, 2] and then normalize the data later?

On the topic of this, what if I have another feature age, which of course only has the length of 1. How would I input Age and Gender in the case I choose one hot encoding, since the length of Gender now becomes 3? I'm using Keras.


texts: Sub-sampling so that sample statistics match population statistics

texts:
I want to investigate the impact of various testing strategies on a product. Let's say chairs. I start with 500 random chairs that I've picked up from garage/yard sales. They come in all shapes and sizes, different manufacturers etc, but I've carefully measured each one and record: manufacturer, height, width, depth and fabric type. I calculate my population parameters. I find that some parameters are normal, and others are uniform, fabric types are mostly cotton but some leather.

I want to split my 500 chairs into groups of 100 such that each group has similar sample statistics. This way I can differentiate the impact of the various test on the chairs without worrying that I'm actually observing differences in my input distribution. I.e. I don't want all the leather chairs in one group.

I've tried randomly grouping my dataset, but I always end up with bad bias in one statistic. I thought that it might be possible to start with random grouping and then randomly selecting a pair of chairs to swap groups; recalculate group statistics; if they get closer to population parameter, keep the swap, else revert; repeat. That seems dreadfully slow.

I'm sure that there are many solutions available, but I'm not sure what they're called: what should I be searching for?
If you have a solution handy, I don't really mind what language it is presented in.
I'll add additional tags to help others find this sort of thing in the future.
Thanks!


texts: Hidden Markov Models (HMM) and baum welch algorithm


texts: LSTM sequence prediction: 3d input to 2d output

texts:
I have this LSTM model
model = Sequential()
model.add(Masking(mask_value=0, input_shape=(timesteps, features)))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences=False))
model.add(Dense(features, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

and shapes X_train (21, 11, 5), y_train (21, 5).
Each timestep is represented by 5 features and
return_sequences is set to False because I want to predict one 5D array (the next timestep) for each input sequence of 11 timesteps.
I get the error

ValueError: y_true and y_pred have different number of output (5!=1)

If I reshape the data as X_train (21, 11, 5), y_train (21, 1, 5) instead I get the error

ValueError: Invalid shape for y: (14, 1, 5)

Note: the value 14 is due to the fact that I'm using cross validation.
What should I do?
Edit
I changed the model to
model = Sequential()
model.add(Masking(mask_value=0, input_shape=(timesteps, features)))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(features, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

and used the same shapes as before.
Here model.summary() gives
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
masking_1 (Masking)          (None, 11, 5)             0
_________________________________________________________________
lstm_1 (LSTM)                (None, 100)               42400
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 505
=================================================================

The idea is to produce a multilabel classification. After training the model, I evaluate it on the test data and this is what I get:

X[0] =  [[0 0 0 0 0],[1 0 0 1 0], ...,[0 0 1 0 0],[0 0 1 0 0]]
y_true[0] = [0 0 1 0 0]
y_pred[0] = 2

which is not what I want. How can I get an output of the same shape as y_true, so as to transform it into a multilabel classification?


texts: What are some good methods to forecast future revenue on categorical and value based data?

texts:
I have monthly snapshots (3 years) of all the contract data. It includes following information:

Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc
Contract stages [Categorical]: Prospecting, engaged, tracking, submitted, etc.
Duration of contract [Date/Time] : months and years
Bid Start date [Date/Time]: Date (But this changes when the contracts are delayed)
Contract value [Numerical] : Value of the contract in local currency
Future revenue projection [Numerical]: Currency value breakdown of revenue for next 5 years (this value is available for all the contracts, no matter if it's won or lost)

I also have other information about the contracts like id, name, description, etc.
Answers I am trying to get:

Total value of contracts that are changing status from month to month
Total value of contracts that are changing stages from month to month
Average delay of the start date of the contracts
Future revenue projection (5 years) based on change of status and average delay

Problems I am having with this data:

It's not time series data, it's monthly snapshot, so I can either turn it into monthly time series dataset and accumulate revenues based on each status and stages or count of all the contracts.

Do I accumulate the contracts data or leave it as individual contracts? In the later case, how do I feed it to any model? It won't be a time series data then.

Main problem with finding the right approach:

I am not sure what approaches to use to answer very different questions. Some values are categorical and some are numerical. I am not sure if it is a forecasting problem or 'change in event' prediction problem. Or mix of both?

How do I incorporate, these very different categorical variables with numerical revenue value, into any model.

Methods I looked into:

I have read about forecasting models like ARIMA (mostly sales data). It takes time series data to forecast revenues based on historical data. I am not sure if it is valid here because I have contracts that changes status, and I am not sure how to use it in ARIMA model. Or if it is necessary to do so. I am also not sure if there is a seasonality to the data. Contracts winning or losing is not a seasonal event.

I also looked into  Simple Exponential Smoothing (SES) and Holt Winter's Exponential Smoothing (HWES) examples and found the same issue while calculating average delays or forecasting future revenue. The current data is not univariate.

I looked at following answers: https://stats.stackexchange.com/questions/246151/difference-between-time-series-prediction-vs-point-process-prediction and it made me think that maybe my problem is not time series prediction.

Best Approach to Forecasting Numerical Value Based on time series and categorical data? : This made me think maybe I should look into RNN and LSTM.

How do you predict a continuous variable when all your independent variables are categorical : Or my problem is similar to this one.

I am sorry for the long post. I am trying to make the problem as clear as possible. I have no idea what would be the best approach to solve this problem and what data to feed to the model. I am also lost at how to structure the data to get the best use out of all the variables.
I would be grateful if you can help me suggest any good methods or reading resources, so I can answer the questions. Thank you for your time!


texts: Create Custom Python Model in Azure ML Studio?

texts:
I came across this article on how to create a Custom R Model and use it in Azure ML studio

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/create-r-model

Can we create a Python model instead ? While I have seen articles of python scripts for evaluation or data processing , I haven't not found enough documents on creating / using custom model built using Python ?

All I have to do is , expose my custom model as web / rest service


texts: What is the reason for the speedup of transformer-xl?

texts:
The inference speed of transformer-xl is faster than transformer.

Why?

If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?


texts: Independence of Features assumption in Naive Bayes

texts: How do we know if your features in my dataset are independent before applying Naive Bayes? Basically I want to know is it possible for us to get an idea before training our model if Naive Bayes will give decent results on it.


texts: product recommendation of a single product based on customer similarity?

texts: I have been wondering how you can build a model to recommend only one single product to a bunch of customers. So basically the question that I would like to answer with this model is to have a ranking of which customer is most likely to buy one specific product based on the customers similarity to other customers who have this product. I only have positive customers (so only customers who do have the product) but no data on customers who don't have it. Just to make it clear there exists just one product, I don't have any data on other products. What algorithm could I use?


texts: Generating Image data sets for training CNNs

texts:
I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the Caffe framework.

My next decision is how best to build my data set. According to this book, I need around 5000 data points for each class (so lets say ~500k images).

I've done a little bit of reading on here and other resources, and it seems that the Google Custom Search API is a potential option; but that limits me to at most 100 searches per day (for free). I thought about building a script to scrape sites like Autotrader; but my experience with scraping the web is zero.

Does anyone have experience generating image data sets of this size? Any pearls of wisdom that you could share with me? I am happy to invest time and effort learning for example Beautiful Soup or this Google API, but I don't want to waste time going down the wrong rabbit hole.


texts:
Is it technically possible to find out similar products online based on an given image?

Say the first link is a hoodie for women selling on Amazon. I want to do two things.

Based on the image, I want to generate features of the hoodie in addition to the ones provided from the product specs(color, size). For example, a logo in the middle and large pocket in the front
I want to find out the similar product(second link) based on the image(first link). Is it feasible to find similar images with at least this degree of matching(the more the better)  from a list of images? I will assume it's a failed match if finds me a women's hoodie with a big pocket in front but no logo.

https://www.amazon.com/GAP-Girls-Fleece-Pullover-Hoodie/dp/B075YPTLTQ/ref=sr_1_1_sspa?keywords=gap&qid=1552274273&s=gateway&sr=8-1-spons&psc=1

https://www.amazon.com/dp/B07F65W746/ref=twister_B07F62RWZL?th=1


texts: Training on a single, random domain, per batch vs multiple domains per batch on a common task

texts:
Say I have multiple domains such that d_i is drawn from D=[d_1, d_2, ... d_K]. We have two options to train a CNN which equally represents all domains.

Collect samples from all domains and create a batch B
Generate a batch B from a randomly chosen d_i and keep shuffling domains per iteration

My intuition tells me that option A is better. However, I do not know why. I am looking for research which touches upon this topic. I would also appreciate your intuition, choice of normalization layers and other tricks/techniques that helped you navigate this problem.


texts: Customer Segmentation - Can I use an ML model for this problem?

texts: I have a seller segmentation problem where I need to segment the top and bottom N customers based on three variables: sales, Average selling price and product category. However, the thresholds for the ASP and sales can keep changing as we move along in time and hence the top and bottom N also changes. Can someone explain if there is a more predictive method to segment the top and bottom N?


texts: How to interpret Shapley value plot for a model?

texts:
I was trying to use Shapley value approach for understanding the model predictions. I am trying this on a Xgboost model. My plot looks like as below

Can someone help me interpret this? Or confirm my understanding is correct?

My interpretation

1) High values of Feature 5 (indicated by rose/purple combination) - leads to prediction 1

2) Low values of Feature 5 (indicated by blue) - leads to prediction 0

3) Step 1 and 2 applies for Feature 1 as well

4) Low values of Feature 6 leads to prediction 1 and high values of Feature 6 leads to Prediction 0

5) Low values of Feature 8 leads to prediction 1 and high values of Feature 8 leads to prediction 1 as well. If it's too the extreme of x-axis (meaning from x(1,2) or x(2,3) - it means the impact of low values (in this case) of this feature, has a huge impact on the prediction 1. Am I right?

6) Why don't I see all my 45 features in the plot irrespective of the importance/influence. Shouldn't I be seeing no color when they have no importance. Why is that I only see around 12-14 features?

7) What role does Feature 43,Feature 55, Feature 14 play in prediction output?

8) Why is the SHAP value range from -2,2?

Can someone help me with this?


texts: In natural language processing, why each feature requires an extra dimension?

texts:
I am reading Machine Learning by Example. I am trying to understand natural language processing. The book used Scikit-learn's fetch_20newsgroups data as an example.

The book mentioned that the text data in the 20 newsgroups dataset that we downloaded from fetch_20newsgroups data is highly dimensional. I do not understand this statement.
It is my understanding that dimension is used to describe axies that an array has.
For example,

import numpy as np
np.random.seed(0)
x1 = np.random.randint(10, size=6)
print("x1",x1) # 1 dimensions
np.random.seed(0)
x2 = np.random.randint(10, size=(3,4))
print("x2",x2) # 2 dimensions
np.random.seed(0)
x3 = np.random.randint(10, size=(3,4,5))
print("x3",x3) #3 dimensions

How does no. of axies relates to feature in NLP? Why one feature equals to one dimension? Please explain. Thanks.

Below is the code from the book that used to download the data for your reference.

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()


texts: Mathematical way of identifying wrong suggestions or outliers

texts:
I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person.
Eg.
Input1 -Donald Trump

30/100 classifiers returns politician as the class
20/100 classifiers returns business man as the class
10/100 classifiers returns leader as the class
10/100 classifiers returns american as the class
10/100 classifiers returns republican as the class
10/100 classifiers returns sportsman as the class
3/100 classifiers returns priest as the class
3/100 classifiers returns doctor as the class
2/100 classifiers returns engineer as the class
1/100 classifiers returns indian as the class
1/100 classifiers returns sportsman as the class
In the above case i take 10 votes as a threshold, i can somewhat correctly define Donald Trump, though a definition of sportsman might be wrong. However 10 seems to be a decent threshold
Input2 -Christiano Ronaldo
20/100 classifiers returns sportsman as the class
20/100 classifiers returns foot ball player as the class
13/100 classifiers returns real madrid as the class
13/100 classifiers returns manchesterunited as the class
12/100 classifiers returns juventus as the class
12/100 classifiers returns european as the class
2/100 classifiers returns portugese as the class
2/100 classifiers returns cricketer as the class
2/100 classifiers returns american as the class
2/100 classifiers returns chinese as the class
2/100 classifiers returns korean as the class
In the above example, if i take 12 votes as the threshold, it correctly defines Christiano Ronaldo, though we might be missing portugese tag since its vote is only 2. However we are doing a good job here i guess.
My problem is, if i have an api that returns the votes and class of famous persons this way, what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct


texts: How can give weight to feature before PCA

texts: I wonder how can I give weight to my feature before employing PCA. I mean somehow weighted PCA. Because I know that one of the features is better than others and want to give importance to it in creating components (It is not possible to select only that feature. I should have others impact too)


texts: How to develop custom metric with one class priority Keras

texts:
I tried like this. I get InvalidArgumentError, but I assume I am doing this completely wrong, I just don't know how to implement in in Tensorflow.

def correct_prediction(self, y_true, y_pred):

        y_true = K.get_value(y_true)
        y_pred = K.get_value(y_pred)

        score = 0

        for pred, true in zip(y_pred, y_true):

            if pred == 1 and true == 1:
                score += 100

            elif pred == 1 and true != 1:
                score -= 100

            elif pred == 0 and true == 0:
                score += 1

            elif pred == 0 and true != 0:
                score -= 1

            elif pred == 2 and true == 2:
                score += 1

            elif pred == 2 and true != 2:
                score -= 1

        print('score:', score)
        score = tf.cast(score, tf.float32)
        return K.variable(score)


texts: How to download images and bounding boxes from imageNet?

texts:
I am doing object detection for a specific class, say, chairs .

I want to download images of chairs from imageNet. I also want to download the annotation xml files (bounding boxes) from imageNet.

Both these things are provided on imageNet and I have successfully been able to download them using a tool called ImageNet_Utils

https://github.com/tzutalin/ImageNet_Utils

But the downloaded images and bounding boxes don't have matching names. So it is impossible to tell which xml file is for which image.

How do I download images and bounding boxes from imageNet such that corresponding image and annotation xml files have matching names?


texts:
This may seem to be a silly question. But I just wonder why the MAE doesn't reduce to values close to 0.

It's the result of an MLP with 2 hidden layers and 6 neurons per hidden layer, trying to estimate one outputvalue depending on three input values.
Why is the NN (simple feedforward and backprop, nothing special) not able to maybe even overfit and meet the desired training values?
Costfunction = $0.5 (Target - Modeloutput)^2$
EDIT:
Indeed I found an inconsistency in the inputdata.

Already cheering, I was hoping to see a better result, after fixing the issue with the input data. But what I got is this:

I'm using a Minibatch-SGD and now I think it might get trapped in a local minimum. I read about Levenberg-Marquardt algorithm, which is said to be stable and fast. Is it a better algorithm for global minimum detection?


texts: Theoretical work on validity of restricting movement of Centroid of K-Mean

texts:
I recently received a manuscript for review in which author used ~1000 "fake" data points, so that the final centroid of K-mean stays within the required range. Neither me nor the author seems to have background in data science and the paper is more of application into our research area.

I have tried to find published work related to such method of restricting k-mean centers, but failed to do so. However, on simple logic, it seems like valid way, so maybe author used wrong terminology.

Hence, I would like to ask, is this a valid way to restrict k-mean center and are there any published work on it?


texts: Which of the following is a consequence of Selecting model complexity on test data? (multiple right answers)

texts:
Selecting model complexity on test data (choose all that apply):

A. Allows you to avoid issues of overfitting to training data

B. Provides an overly optimistic assessment of performance of the resulting model

C. Is computationally inefficient

D. Should never be done

This question is part of a machine learning course I am taking on Coursera that I just realized has poor instructor support and hence I am unable to receive any help.

I am confident that I have understood every little detail of the material that this question is based on. But I have no idea why my answer is wrong, i.e.: A and B

Here is my explanation for why I picked A and B:

Analyzing this question's 4 options:

A. Preventing overfitting and under-fitting of training data is the goal of using tuning parameters to pick the right model complexity, hence using test data for this should achieve the goal/purpose.

B. Given that tuning parameter is picked according to model performance on test data as question states, when generalization error is estimated for picked model, it is estimated on the same test data, and hence gives an overly optimistic result.

C. Nothing implies that the process is computationally inefficient (although it is practically inefficient because of double-dipping into test data).

D. Nothing implies it should NEVER be done, because if test data set is containing almost all of the data in the world, even if complexity is fitted to it, it could still serve as good approximation of generalization error (not overly optimistic).

Does this logic above not hold? I am having extreme trouble with this question. Is it because I am assuming that there is no validation set in this case (I don't see how that would change the right answer, though).

Someone please help me.


texts: Is there any way to plot ROC curve for Ensemble hard voting classifier?

texts: I am working on a multi-class text classification problem and performing an Ensemble learning for text classification. I chose hard voting as ensemble technique. I tried to plot ROC curve for my ensemble method but it didn't work by showing the error that ROC curve can't be plotted for Hard voting ensemble metho. So my question is there any possible way by which I can plot ROC curve for my hard voting ensemble classifier in python.


texts: How to create a seq2seq without specifying a fixed decoder length?

texts:
Based on the model presented in this answer:

def create_seq2seq(features_num,latent_dim,decoder_length):

    ##
    encoder_inputs = Input(shape=(None, features_num))
    encoded = LSTM(latent_dim, return_state=False ,return_sequences=True)(encoder_inputs)
    encoded = LSTM(latent_dim, return_state=False ,return_sequences=True)(encoded)
    encoded = LSTM(latent_dim, return_state=False ,return_sequences=True)(encoded)
    encoded = LSTM(latent_dim, return_state=True)(encoded)

    encoder = Model (input=encoder_inputs, output=encoded)
    ##

    encoder_outputs, state_h, state_c = encoder(encoder_inputs)
    encoder_states = [state_h, state_c]

    decoder_inputs=Input(shape=(1, features_num))
    decoder_lstm_1 = LSTM(latent_dim, return_sequences=True, return_state=True)
    decoder_lstm_2 = LSTM(latent_dim, return_sequences=True, return_state=True)
    decoder_lstm_3 = LSTM(latent_dim, return_sequences=True, return_state=True)
    decoder_lstm_4 = LSTM(latent_dim, return_sequences=True, return_state=True)

    decoder_dense = Dense(features_num)

    all_outputs = []
    inputs = decoder_inputs

    states_1=encoder_states
    # Placeholder values:
    states_2=states_1; states_3=states_1; states_4=states_1
    ###

    for _ in range(1):
        # Run the decoder on the first timestep
        outputs_1, state_h_1, state_c_1 = decoder_lstm_1(inputs, initial_state=states_1)
        outputs_2, state_h_2, state_c_2 = decoder_lstm_2(outputs_1)
        outputs_3, state_h_3, state_c_3 = decoder_lstm_3(outputs_2)
        outputs_4, state_h_4, state_c_4 = decoder_lstm_4(outputs_3)

        # Store the current prediction (we will concatenate all predictions later)
        outputs = decoder_dense(outputs_4)
        all_outputs.append(outputs)
        # Reinject the outputs as inputs for the next loop iteration
        # as well as update the states
        inputs = outputs
        states_1 = [state_h_1, state_c_1]
        states_2 = [state_h_2, state_c_2]
        states_3 = [state_h_3, state_c_3]
        states_4 = [state_h_4, state_c_4]

    for _ in range(decoder_length):
        # Run the decoder on each timestep
        outputs_1, state_h_1, state_c_1 = decoder_lstm_1(inputs, initial_state=states_1)
        outputs_2, state_h_2, state_c_2 = decoder_lstm_2(outputs_1, initial_state=states_2)
        outputs_3, state_h_3, state_c_3 = decoder_lstm_3(outputs_2, initial_state=states_3)
        outputs_4, state_h_4, state_c_4 = decoder_lstm_4(outputs_3, initial_state=states_4)

        # Store the current prediction (we will concatenate all predictions later)
        outputs = decoder_dense(outputs_4)
        all_outputs.append(outputs)
        # Reinject the outputs as inputs for the next loop iteration
        # as well as update the states
        inputs = outputs
        states_1 = [state_h_1, state_c_1]
        states_2 = [state_h_2, state_c_2]
        states_3 = [state_h_3, state_c_3]
        states_4 = [state_h_4, state_c_4]

    # Concatenate all predictions
    decoder_outputs = Lambda(lambda x: K.concatenate(x, axis=1))(all_outputs)

    model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

    #model = load_model('pre_model.h5')

    print(model.summary()
    return (model)

It is possible to create a Keras seq2seq model where the prediction of each decoding step is fed as input for the next step.

But while the model defined by this code is unlimited in the length of its input sequence, it is limited to a pre-defined output sequence length.

This may be fine for the toy problem demonstrating the use of this code, that is, just demonstrating the ability to predict the continuation of a given sine-like sequence, but a length agnostic solution would be preferable for an NLP problem I am trying to tackle.

How can I define such a model in Keras?

NB:
This question was previously asked in Stack Overflow


texts: Using z-score for neural network normalization

texts:
I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds.

So, I guess I'm right in saying the feature data then needs to be scaled (using min-max?), after calculating z-score?


texts: Reducing text input size into word2vec without affecting performance too badly?

texts:
So I am implementing Word2Vec for the first time, and I have a set of training data that I would like to train a word2vec model on. Predictably, the problem is the dataset is rather large, and I have more limited computational power than I would like. This is a very common problem of course, but are there any ways to minimize the input text without too horribly affecting the performance?

For example, if I had the example sentences:

  Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. Mr. Dursley was the director of a firm called runnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache.

I could do some thing like take out the stop words... but as I googled, that's a bad idea according to one link and a good idea according to another... so which is it? I also looked into text summarization but that seems harder than implementing word2vec. Another idea I naively had was to randomly take out X% of sentences from the text. But that would obviously be a performance hit in the model, and I'm not sure how big.

So, are there any general methods for trying to do this, and is stop word removal a way to do it?


texts: Does it make sense to interpolate image just before a CNN?

texts: I'm training a CNN with images which have lots of horizontal black lines (due to the nature of the sensor). I'm thinking in removing this artifacts by some kind of preprocessing (interpolation, median filters...). The thing is: does it make sense, since the CNN tries to apply optimal filtering? (if some 2D filtering is intended to be done just before the CNN, is just adding a deterministic layer at the beginning of the net...)


texts: Cp plot in decision tree only showing upward trend

texts:
In Cp plot for decision tree, we expect the error to drop down with size of the tree (or form an elbow). This helps to select right Cp value. But in my graph, the relative error is continuously increasing (please refer to the attached picture).
Am I doing something wrong and how to decide the right Cp here?


texts: What is the distribution of categories in imagenet training set (ILSVRC2012)

texts:
http://arxiv.org/pdf/1409.0575v3.pdf Table 2
says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set.

Ideally I'd like to avoid downloading the 138 GB just for this purpose as I otherwise don't need it.

I was wondering if anyone knew the exact numbers per class in the training set, i.e., how likely is each class in the training set?


texts: Handling dimensions for RGB data with Keras CNN

texts:
I'm trying to make work the code from Keras' documentation getting started. There is something I do not understand about handling RGB data. I made one work with MNIST data (which is greyscale), but I can't seem to figure out the CIFAR10.
HEIGHT = 200
WIDTH = 200
def build_model():

    inputs = keras.Input(shape=(HEIGHT, WIDTH, 3))

    # making things simple, I don't CenterCrop, just set the input to whatever I'm feeding it
    # I do rescale to 0-1 values
    x = Rescaling(scale=1.0 / 255)(inputs)

    # this basically the doc's architecture
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu", )(x)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu",)(x)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu",)(x)
    x = layers.GlobalAveragePooling2D()(x)
    outputs = layers.Dense(10, activation="softmax")(x)
    model = keras.Model(inputs=inputs, outputs=outputs)
    return model

# fake data, easier to play with shapes  than with the actual CIFAR10 data to debug
# shape is (500, 200, 200, 3), cifar10's is (60000, 32, 32, 3)
data = np.random.randint(0, 255, size=(500, HEIGHT, WIDTH, 3)).astype("float32")
# shape is (500,1) cifar10's is (60000,1). Just 10 categories to match the output layer
labels = np.random.randint(0,9, size=(500,1)).astype("int8")

print("got fake data... ")
model = build_model()
print("model built... ")
model.summary()
model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
              loss=keras.losses.CategoricalCrossentropy())
print("model compile...")
model.fit(data, labels)
print("done")

For some reason, I get:
ValueError: Shapes (None, 1) and (None, 10) are incompatible

Strangely, if change the label's fake data to size=(500, 10) then it "works". But obviously that makes no sense, since it would mean I have 10 labels for each sample. The MNIST dataset that works which I refer to can also be found in the linked documentation. The architecture of the network is much simpler (only Dense layers linked together).
What am I missing here? Why would it be so different for a colored image than it is for a greyscale?
I also tried the work with the data from     train, test = cifar10.load_data(), but it yields the same results.


texts:
Is there a way that I can draw a simple CNN architecture in LaTeX if I know that my model has: four convolutions, three blocks (convolution + BN + RELU + dropout), and then pooling after each layer?
Then lastly a normal convolution Conv2d.


texts:
I'm building a binary sound classifier using the ESC-50 dataset. I have taken one class, "dog bark", to be positive and the rest of the 49 classes to be negative. As the dataset is imbalanced I'm running into lot of training issues. I tried building a model but couldn't get a f1-score greater than 0.3.

I'm using mfcc and fft as features. I have tried used LR and SVM to train without much success. Can't use Deep learning models as it's a real-time system and can't have much delay.

How can I approach this problem?


texts: How to solve this problem as Machine Learning?

texts:
I have a time series data. I have to identify the point in red. I know how to solve this using statistics. But if I had to use Machine Learning to detect these points how can I solve this.

How can I modify this data in order to feed it to machine learning model. Data is just one column readings as show in plot. Any idea?


texts: Python Matplotlib Radar not correctly centered

texts:
I'm trying to create radar plots for players's profiling. I got this solution from this previous question, but the graphic is not correctly centered and in fact there is a label that ends outside the plot.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns  # improves plot aesthetics

def _invert(x, limits):
    """inverts a value x on a scale from
    limits[0] to limits[1]"""
    return limits[1] - (x - limits[0])

def _scale_data(data, ranges):
    """scales data[1:] to ranges[0],
    inverts if the scale is reversed"""
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        assert (y1 <= d <= y2) or (y2 <= d <= y1)
    x1, x2 = ranges[0]
    d = data[0]
    if x1 > x2:
        d = _invert(d, (x1, x2))
        x1, x2 = x2, x1
    sdata = [d]
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        if y1 > y2:
            d = _invert(d, (y1, y2))
            y1, y2 = y2, y1
        sdata.append((d-y1) / (y2-y1) * (x2 - x1) + x1)
    return sdata

class ComplexRadar():
    def __init__(self, fig, variables, ranges,
                 n_ordinate_levels=6):
        import ipdb; ipdb.set_trace()
        angles = np.arange(0, 360, 360./len(variables))
        axes = [fig.add_axes([0.1, 0.1, 0.9, 0.9], polar=True,
                label="axes{}".format(i))
                for i in range(len(variables))]
        l, text = axes[0].set_thetagrids(angles,
                                         labels=variables)
        [txt.set_rotation(angle-90) for txt, angle
         in zip(text, angles)]
        for ax in axes[1:]:
            ax.patch.set_visible(False)
            ax.grid("off")
            ax.xaxis.set_visible(False)
        for i, ax in enumerate(axes):
            grid = np.linspace(*ranges[i], num=n_ordinate_levels)
            gridlabel = ["{}".format(round(x, 2))
                         for x in grid]
            if ranges[i][0] > ranges[i][1]:
                grid = grid[::-1]  # hack to invert grid. gridlabels aren't reversed
            gridlabel[0] = ""  # clean up origin
            ax.set_rgrids(grid, labels=gridlabel,
                          angle=angles[i])
            # ax.spines["polar"].set_visible(False)
            ax.set_ylim(*ranges[i])
        # variables for plotting
        self.angle = np.deg2rad(np.r_[angles, angles[0]])
        self.ranges = ranges
        self.ax = axes[0]

    def plot(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

    def fill(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

def data2():
    metrics = ('Shots', 'Goal Conversion %', 'Assists', 'Passing %', 'Int+Tackles', 'Disposessed')
    firmino = (3.4, 15.7, 0.3, 82.3, 2.6, 1.7)
    ranges = [(1, 5), (4.5, 22.5), (0.1, 0.4), (65, 85), (1, 4.5), (1, 3)]
    fig = plt.figure(figsize=(12, 9))
    radar = ComplexRadar(fig, metrics, ranges)
    radar.plot(firmino)
    radar.fill(firmino, alpha=0.2)
    plt.savefig('firmino.png')

if __name__ == '__main__':
    data2()


texts: Vector representation of documents for text classification

texts: I'm looking for proper method of document embeddings. I know that doc2vec will give me the vector representations for given corpus, but how do I embed new documents? I need to train neural network that will classify text, but I have no idea how new documents should be embedded properly.


texts: Getting lower performance metrics when using GridSearchCV

texts:
I have defined an XGBoost model and would like to tune some of its hyperparameters.

I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training dataset, and I have noticed that the 'roc_auc' performance metric is higher than when I used the Grid Search. I was surprised, because I was expecting Grid Search to perform better. I think I am missing the intuition here.

My understanding was that for grid search cross-validation, for say k folds,  given a parameter value from the param_grid, gridsearchcv fits the model on the folds separately and calculates the desired performance metric. Later, for that particular parameter, it takes the 'average' of all the folds' calculated 'roc_auc'. The gridsearch repeats this process for all the other given parameters in the params_grid.
Finally, the '.best_params_' is the one for which the calculated metric is higher.

This is what I tried:

param_test = {'max_depth':[3,5,6,7,9]}
model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)
gsearch = GridSearchCV(estimator=model, param_grid = param_test, scoring='roc_auc', cv=5)
gsearch.fit(X_train, y_train)
print('Best found params: {}'.format(gsearch.best_params_))
print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

This prints:

Best found params: {'max_depth': 6}
Best (Train) AUC Score: 87.2186%

Now, when I use the same model and fit it on the entire training dataset, this is what I get:

model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)

model.fit(X_train, y_train, eval_metric='auc')

# Predict training set:
y_pred_train = model.predict(X_train)
y_pred_proba_train = model.predict_proba(X_train)[:,1]

# Print model report:
auc_score_train = roc_auc_score(y_train, y_pred_proba_train)

print("AUC Score (Train): {:.4f}%".format(auc_score_train*100))

which prints:

AUC Score (Train): 97.0311%

Why is there such a discrepancy? What am I missing here?


texts: best similarity measure for images with different angles

texts:
I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using some libraries like
from image_similarity_measures.quality_metrics import rmse, psnr, fsim
in_img1 = cv2.imread("./IMG_4835.jpg")
in_img2= cv2.imread("./IMG_4836.jpg")
out_rmse = rmse(in_img1, in_img2)
out_fsim = fsim(in_img1, in_img2)
print(out_rmse)
print(out_fsim)

but none of the metrics seem to provide good results in my case. According to the description of different metrics given here, most of them check for contract/signal to error reconstruction ratio etc etc. https://up42.com/blog/tech/image-similarity-measures
What similarity metric or image comparison method in general would suit well in my case?


texts: What do we learn from training a dataset for logistic regression

texts:
What do we learn from training our dataset in Logistic Resgression?
Like in Linear Regression, with the help of training set we are able to generate a best fit line(y = mx+c) where m and c come from training our dataset.
Similarly, once we train our model for logistic regression, what is that the model learns which is then used to predict a class for a particular input?


texts: How to predict orders with a range of items?

texts:
So I do have data like this:

With the help of distinct order IDs, I can figure out how many orders are there and from units shipped, I can get the number of items in the order. Now I want to predict future orders having x items for different ranges (e.g. 1-10 items, 10-20 items, 20+ items). Like tomorrow there might be 6 orders one of them will have 1-10 items, 4 will have 10-20 items and one will have 20+ items. The problem is I can predict them (using LSTM for now) separately using data for only those kind of orders but they obviously do not sum up to the amount predicted by model for totals. So how can I link them up so that total sums up to the values from the individual models?
Approach I am currently using:
I just predict orders for each item range and then manually sum them up. This way they do sum up but error accumulates


texts:
I found in many sources that Hidden Markov Models are linear-chain networks(e.g. in Predicting Structured Data book by MIT). However, as I understand it, HMMs can have any edges in its graph. Even simple example of HMM in wikipedia has non-linear graph:  .

So, the question is: what is the formal definition linear-chain structure and in which case forward-backward and Viterbi algorithms can give precise results.

I have also taken into consideration this picture, taken from CRF tutorial, which says, that linear-chain CRF is "generative-discriminative pair" to HMM.


texts: Transfer Learning for CNNs and Batch Norm Layers

texts:
In some transfer learning models, we set the training argument to False to maintain the pre-trained values of Batch Normalization for example, but the trainable attribute to False to freeze the weights. Then the new "top-layer" is added and we re-train the model. Afterwards for fine-tuning, we can re-train the weights and set the trainable attribute to True. However, what does the argument training=True do for a layer? The stackoveflow answer here does not make sense to me. When the argument training is True, that, to me, implies we are doing some type of learning: i.e. the BN mean and variance are being updated, Dropout is being applied, and the weights are from the backwards pass. What is the difference between training=True and training=False? The Keras FAQ states it just means inference is being performed, but what was being trained when training=True?
Lastly, this is being nit-picky, but in this notebook Google does transfer learning with the MobileNet V2 model. In same from above they use Xception model. Both models have BN, but in the second tutorial, they pass the training=False argument in the base_model implying do not update BN. Whereas in the first they make no mention of training=False. Why might that be? I see the first one is copyrighted in 2019 and the second one in 2020, which might imply the discrepancy.


texts: Dueling DQN - why should we decompose and then combine them back into?

texts:
could anyone who can help me if we decompose them and combine back them into a single Q, what the network can learn?
from my perspective，the V means the total reward when the agent follow the current policy;the Q means if we give a specific action then follow the current policy what the total reward;
and if we get the optimal policy，the V will equal to Q;so we should learn to make the A reach zero;just like the answer ：Dueling DQN - can't understand its mechanism

but in that paper，i cannot understand what is the matter if we cannot identify the sense that given Q we cannot recover V and A uniqueluy.

and it will get this blow finally：

and as well as this：

refer to this blog:https://medium.com/@awjuliani/simple-reinforcement-learning-with-tensorflow-part-4-deep-q-networks-and-beyond-8438a3e2b8df


texts: R: Pam function under cluster package not accepting data set having more than 65536 entries

texts:
I am running pam function for kmedoid under cluster package in R. My dataset has 1 million entries.
It's throwing error  saying

  Have 1 million observation but not more than 65536 observation allowed

Please suggest the solution

Thanks in advance :)


texts: Comparing one small dataset with a big dataset for similar records

texts:
I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-silmilar) values from three different columns. The values are either one word or a sentences. My algorithms is like this:

make a vector of words in each record in both datasets
with a for loop, search for similarities over the big dataset (e.g. with tfidf). That means each record from the small dataset should look for possible similarities over the big dataset.

However, the problem is searching a big data is very slow. Is there any efficeint way to solve this problem?
Thanks


texts:
Is it real to use word's n-grams for Deep Neural Network?

E.g., sentences list contains in X_train dataframe with "sentences" column.
I use tokenizer from Keras in the next manner:

tokenizer = Tokenizer(lower=True, split=' ')
tokenizer.fit_on_texts(X_train.sentences)
X_train_tokenized = tokenizer.texts_to_sequences(X_train.sentences)

And later I use padding:

X_train_sequence = sequence.pad_sequences(X_train_tokenized)

Also I use simple LSTM network:

model = Sequential()
model.add(Embedding(MAX_FEATURES, 128))
model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2,
               activation='tanh', return_sequences=True))
model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, activation='tanh'))
model.add(Dense(number_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer = 'rmsprop',
              metrics=['accuracy'])

In this case, tokenizer execution.
In Keras docs: https://keras.io/preprocessing/text/
I see character processing only, but it is nt apprepriate for my case.

My main question: Can I use n-grams for NLP tasks with deep learning (not necessary Sentiment Analysis, any abstract NLP task).

Indeed, in many tutorials or books I doesn't see any remainder n-grams for text processing, only embeddings.

For clarification: I'd like to consider not just words, but combination of words - I'd like to try it for my task.


texts: Get the average time between first and 2nd call (postgresql)

texts:
I have the following data in table where I want to calculate the average time between 1st and 2nd call.
I know how to get the average, but I have a though time to figure out how to subtract the 2nd from 1st attempt since it is in the same column and I am more familiar to subtracting things between columns.


texts: How to add attention mechanism to my sequence-to-sequence architecture in Keras?

texts:
Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:

model = Sequential()
model.add(LSTM(hidden_nodes, input_shape=(n_timesteps, n_features)))
model.add(RepeatVector(n_timesteps))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(TimeDistributed(Dense(n_features, activation='softmax')))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=30, batch_size=32)

It works reasonably well, but I intend to improve it by applying attention mechanism. The aforementioned blog post includes a variation of the architecture with it by relying on a custom attention code, but it doesn't work my present TensorFlow/Keras versions, and anyway, to my best knowledge, recently a generic attention has been added to Keras -- I was not able add it to my code, however.

Additionally, I tried to complicate my architecture above by adding 2-2 LSTM layers for the encoder and the decoder respectively instead of 1-1 with this:

model = Sequential()
model.add(LSTM(hidden_nodes, return_sequences=True, input_shape=(n_timesteps, n_features)))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(RepeatVector(n_timesteps))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(TimeDistributed(Dense(n_features, activation='softmax')))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=100, validation_split=0.15, batch_size=32)

but I get error message (in the 2nd or 3rd row, I assume):

ValueError: Input 0 of layer repeat_vector_17 is incompatible with the layer: expected ndim=2, found ndim=3. Full shape received: [None, 20, 128]

What could be the reason here?


texts: Prevent overfitting when decreasing model complexity is not possible

texts:
I'm fairly new to machine learning and as an exercise for a more complicated task, I'm trying to do the following what I thought was a trivial task. Suppose as an input I have population density maps. These are 2D images with one layer, in which each pixel is the count of persons living in that area.
From that data, I'd like my model to "estimate" (in fact it would be possible to calculate the exact solution) the total number of persons living on that density map. Essentially, the task consists of just taking the sum of the 2D input.
I have tried many architectures and I found that the simpler the better. In fact a model containing no hidden layers performed best:

from keras.layers.core import Dense
from keras.layers import Flatten, Input

inputs = Input(shape=(225, 350, 1))
x = Flatten()(x)
x = Dense(1)(x)

While this performs very well on the training data, it fares very poorly on the validation data. I know that this is a sign of overfitting, but how can I prevent overfitting given that is not possible to further decrease complexity of the model? Or would another approach / architecture be better altogether?
Note that I have performed the usual data pre-processing (normalising inputs and outputs).

Thanks in advance for any hints.


texts: Derivation of CNN math equations in Matrix format

texts:
I've gone through jefkine's website and Jae Seo's articles to get a hold of math behind the famous CNN architecture. Although I understand it in theory, I'm unable to implement in matrix format or to put it straight...  In numpy format. After digging through internet like a broken web scraper, I could only understand FC ANN after Andrew Ng tutorial and this,  sudeepraja.github.io/Neural/. Can someone please help me understand CNN architecture in similar way?
I'm expecting downvotes and closed as off topic by admins for being naive and dumb. If that's the price I gotta pay to learn...  So be it!


texts: How to show value of a classification model even though it doesn't get the desired performance?

texts:
I developed a classification model for a telecom client. Where we classify between Dual-sim and non-Dual-Sim clients. After many iteration the best precision we can get is 60%. The contract says that the acceptance criteria is 75% precision. The client measures precision based on campaign results. In other words, they call sample data and ask them explicitly are a dual sim or not.

Facts we know:

Learning curve shows that more training data won't do the trick.
We have got all variables that were used in the market before(and everything we could think of, but nothing improved the model).
Training data don't even get 1% of the whole population I am generalizing on(and we can't get more than that).
The random baseline is 35%, so we got a lift of around 1.9 (but they won't approve of it).
We have tried many iterations from the most simple models to the most complex, and this is the best we can get.
The dual sim and non-dual sim distribution across the variables, isn't that different.
The language used were SQL, R.

So the question is, what else can I do to prove that the model has a value but without getting the 75% precision?


texts: Visualizing large number of points as a 3D density map

texts:
The  result of my computational simulation is a (time-dependent) system of large number (~100k) of moving points in a confined space. Each point has its own Cartesian coordinates as well as a weight (w) in the form of $(x_i,y_i,z_i;w_i)$.

I'm looking for a software/app/package to create a snapshot of the 3D spatial density map of these points. (something like this).

Like you see in this figure, the points are not going to be displayed individually, but only a transparent cloud will be drawn whose local intensity is proportional with the local number of points.

The final goal is to create a movie of the changing 3D spatial density with time.

So far, I have tried R, Matlab, Origin, and ImageJ. But no success!


texts: What's the difference between these two custom sparse categorical accuracy functions?

texts:
I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer.

I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head around exactly what they are doing, what the difference are, and whether one is one better than the other?

I'm quite sure the shape of predictions going into both methods are: y_pred = (number of sents, sentence length (sentences are padded to max_len), number of tags), y_true = (number of sents, sentence length, 1)

Method 1: Commented by jerrywind on this page

def custom_sparse_categorical_accuracy(y_true, y_pred):
    flatten_y_true = K.cast( K.reshape(y_true,(-1,1) ), K.floatx())
    flatten_y_pred = K.cast(K.reshape(y_pred, (-1, y_pred.shape[-1])), K.floatx())
    y_pred_labels = K.cast(K.argmax(flatten_y_pred, axis=-1), K.floatx())
    return K.cast(K.equal(flatten_y_true,y_pred_labels), K.floatx())

Method 2: commented by dilshatu on this page:

def custom_sparse_categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.max(y_true, axis=-1),                K.cast(K.argmax(y_pred, axis=-1), K.floatx())), K.floatx())

When I try either method on a tiny sample of my data and view the model history, the train and validation accuracies for that method aren't identical but very close (sometimes the same values).

The sparse_categorical_accuracy scores after model.evaulate are quite different: 89.69% vs 93.49% for method 1 vs method 2.

The F1-scores after model.predict however are close: 19.0% vs 19.4% for method 1 vs method 2.

My model is constructed as follows:

def build_model():
    input = layers.Input(shape=(1,), dtype=tf.string)
    # Custom ELMo embeddings layer
    model = ElmoEmbeddingLayer(name='ElmoEmbeddingLayer')(input)
    model = Bidirectional(LSTM(units=512, return_sequences=True))(model)
    out = TimeDistributed(Dense(num_tags, activation='softmax'))(model)
    model = Model(inputs=input, outputs=out)
    model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=[custom_sparse_categorical_accuracy])
    model.summary()
    return model

model = build_model()
epochs = 10
batch_size = 32
history = model.fit(X_train_sents,
                y_train,
                validation_data=(X_valid_sents, y_valid),
                batch_size=batch_size, epochs=epochs)

My task is semantic slot filling (a kind of named entity recognition). I'm not worried about the bad scores because I'm using a tiny sample and just 10 epochs.


texts: model.predict keeps predicting the same, wrong, class

texts:
I have a simple oxford102 (102 UK flowers) tensorflow model and Android app from here.
I'd like to see if I can add recognition (model.predict) to the python code in the subproject FlowerML which is python code that generates the model in the first place.
Added the following to FlowerML2.py to recognize a single flower:
# sequence of calls adapting image then calling predict:
# https://datascience.stackexchange.com/questions/31167/how-to-predict-an-image-using-saved-model
test_image = image.load_img('d:/ML/images/CallaLily-1.jpg', target_size=(224, 224))
print("test_image PIL size " + str(test_image.size))
test_image_arr = image.img_to_array(test_image)
print(test_image_arr.shape)
# double-check visually
image.save_img('d:/ml/images/WhatIsIt-1.jpg', test_image_arr)
test_image = np.expand_dims(test_image_arr, axis=0)
print(test_image.shape)
result2 = model.predict(tf.convert_to_tensor(test_image), batch_size=1, verbose=2)
print("\nnp.where result2: " + str(np.where(result2 == result2.max())) + " max = " + str(result2.max()) + "\n")

The model correctly identifies Calla Lily in the Android app however when I query the model as above it keeps telling me the wrong class.  'Keeps' refers to many modifications not above - and even with a different flower.  It's like the model, in this context (but not in the Android app), gets stuck with an output to its head layer of n=1.
Since it works in Android, my assumption is that I'm not calling predict properly - although exactly what goes wrong I can't say and I have little experience, so far, digging into how models work which is maybe what you need to do when they don't function correctly.


texts: Can we implement random forest using fitctree in matlab?

texts: There is a function call TreeBagger that can implement random forest. However, if we use this function, we have no control on each individual tree. Can we use the MATLAB function fitctree, which build a decision tree, to implement random forest? Thanks a lot.


texts: Poor Precision-Recall curve for binary classifier trained on balanced data, with imbalanced test data

texts:
I have an very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).

I am performing classification using RandomForest. My test set comprises of unbalanced data (since this is how it is expected to be in the real world), and my ROC curve is pretty good, with AUC=0.873. However, my P-R curve suffers, with AUC=0.547.

If I balance the test data, I see a better AUC; however, this is not how the ground truth has behaved in the past, so quoting performance on balanced test data doesn't seem too rational.

What am I doing wrong? Should I train on imbalanced data? Or are there any ways where I can improve the P-R curve?


texts: Local minima in Stochastic Gradient Descent

texts:
I am running a logisitic regression model using Keras on a dataset. However, I am often running in a problem in which the model settle at an validation accuracy value lower than a theoretical value.

The hypothesis is confirmed by the fact that if I run the model again (without changing anything), the model settles at a different value (often higher) for validation.

I presume such a problem is due to the gradient descent encountering a local minimum and when I run it again, due to random fluctuations, I sometimes end up skipping the local minima.

So any method, to prevent such a problem?

def base():
    model = Sequential()
    model.add(Dense(20,activation = 'linear', input_dim = 12288))
    model.add(Dense(7,activation = 'relu'))
    model.add(Dense(5, activation = 'linear'))
    model.add(Dense(1,activation = 'sigmoid'))
    model.compile(optimizer = optimizers.SGD(lr=0.0075), loss = 'binary_crossentropy', metrics= ['accuracy'])
    return model

model = base()
model.fit(train_x,train_y, epochs = 2400, batch_size =10,validation_data = [test_x, test_y], verbose = 2)


texts: Tree based method are robust against low probability feature space zones when using ML general interpretability methods?

texts:
I have this intuition but I'm not able to verify it.

There are a lot of techniques to understand the effect of single features in ML models. Some take inspiration from counterfactual frameworks, like ceteris paribus, and evaluate the unconditional contribution of feature $X$ by observing the change in prediction when varying $X$ values leaving all other variables fixed.
The most common of such techniques are PDPs https://christophm.github.io/interpretable-ml-book/pdp.html.

The problem is that this methodology are not robust for impossible combinations of the predictors. For example in a model to predict bike-sharing count given weather conditions and period of the year, it's possible to make predictions for a temperature of 40°C in wintertime, even if there's no such data point in the training set.

There are various techniques to accommodate for this bias, like accumulated local estimation plots (ALE). I was wondering though if tree-based methods (simple or ensemble) are naturally more robust than regression-based ones to such bias; I expect this, because tree-based predictions vary only among partitions of the feature space that are present in the data, while regressions allow prediction variation for never observed predictors combinations.

For example, this is the output of a conditional tree trained on the bikes   problem:

[1] root
|   [2] temp <= 12.2
|   |   [3] season in SPRING, SUMMER
|   |   |   [4] temp <= 4: 1663 (n = 64, err = 30258081)
|   |   |   [5] temp > 4: 2852 (n = 133, err = 216353574)
|   |   [6] season in WINTER
|   |   |   [7] hum <= 82.3: 4315 (n = 90, err = 117371810)
|   |   |   [8] hum > 82.3: 2781 (n = 9, err = 26537744)
|   [9] temp > 12.2
|   |   [10] hum <= 84.8
|   |   |   [11] windspeed <= 13.2: 5877 (n = 256, err = 454812206)
|   |   |   [12] windspeed > 13.2: 5285 (n = 149, err = 326330122)
|   |   [13] hum > 84.8: 3382 (n = 30, err = 47251364)

as expected, the temperature and the seasons are correlated, therefore we won't find rules regarding winter for higher (>12.2) temperatures.

So I expect that forcing Winter with a temperature of 14 won't produce a different prediction than Summer. I expect also that this robustness would replicate also for more complex blackbox models like random forests and boosted trees.

Instead, regression-based methods will allow impossible predictions as shown by the following linear model, where the effect of temperature is unbounded.

   (Intercept)           temp   seasonWINTER            hum      windspeed   seasonSUMMER holidayHOLIDAY
        4888.4          152.1         1307.1          -37.6          -64.0          673.2         -621.4

Can someone confirm/dispute this, preferably with a theory-based explanation?


texts: How is it possible to process an image with a few neurons?

texts:
An 1024*1024 pixel image has around one million pixels. If I would like to connect each pixel to an R,G,B input neuron, then more than 3 million neurons are needed.

It would be really hard, to train a neural network, which has millions of inputs. How is it possible, to reduce the number of neurons?