texts: Multitask learning NN only trains on a few tasks

texts:
I am fairly new to the Machine Learning field, but I am working on building a multi-task network for a project I am working on. Unfortunately, despite all my best googling and digging through documentation, blogs and research papers, I am so far unable to get my network training on more than a few of the tasks at a time.

For context, the problem entails a set of images of metallic test samples, all taken under standard lighting and at a standard distance. For confidentiality reasons, I cannot share the dataset itself. The goal of the network is to be able to score 3 different surface defects in 3 distinct regions each, a total of 9 outputs, each with an integer rating score from 0 to 5 (6 possible outcomes for each output).

I have been writing the NN in Python using Keras and visualising the metrics and graph with TensorBoard.

The problem is that, as I attempt to train the network, it only (at best) seems to learn for up to 4 of the 9 tasks, even though each image has a rating for each task. To illustrate my point, here are two screenshots of the TensorBoard metrics for one of the more successful training runs along with the graph showing the network structure:

As you can see, the tasks WCP_3T and WCP_5T are training well, but nothing else really is (the high accuracy on the WCP_F task is due to heavily weighted data, and not much else I'm afraid). The plateau in accuracy for each of the other tasks seems to me to be the network quickly finding the rating that is most common for that task, and remaining there. Thus, the varying accuracies for each task is an artifact of the data for that category.

I have tried deepening (and shallowing) the other two main branches (BS and BD), I have experimented with altering learning rate, batch size, optimizer function (currently using Adam) and activation functions (currently using relu for the WCP branch and LeakyReLU for the other two, but I have also tried ordinary relu for them with no benefit). I have altered the number of filters in the various convolutional layers, and I have increased and decreased the dropout on various layers also. Still, nothing I seem to try can bring the network to train on all the tasks.

It is possible that the size of the dataset is a problem, but I only have a small number of images available and I don't think it is possible to get many more. I am currently using a training set of 506 images and a batch size of 32, which only leaves about 170 images for each testing and validation. Data augmentation to provide more images may also be out of the question as that could conceivably alter the scores placed on the different defects, and I would not know how (they were graded by experts, and the data was then passed to me for this task).

If anyone has had a similar problem before and could suggest something else I could try, or could at least point me towards somewhere that might help to solve my problem, I would be extremely grateful!


texts:
I get the point of a validation and training set, but the importance of a test set doesn't click for me.

Let's say you train a model and you try your best to avoid overfitting by testing your model on the validation set.

After you've decided you have a model you're proud of, you do a final sanity check on the test set, and let's say the performance is trash. Are you really going to start all over? What decision making does it inform? In my workplace, the way timelines are structured, there's no time to start over.


texts: What does the “numDecimalPlaces” in J48 classifier do in WEKA?

texts:
Can anyone explain the purpose of "numDecimalPlaces" parameter in J48 WEKA classifier ? Its default value is 2 and its description is given as follows:

  The number of decimal places to be used for the output of numbers in
  the model.

However, I can't understand how it going to affect the results.
For example, if my feature values are something like this (0.00123 or 0.00234 etc). Does it means that the trained model will only consider these values up to 2 decimal points ? The documentation also says:

In the documentation, its mentioned in the summary fields section:                Field Summary

  Fields inherited from class weka.classifiers.AbstractClassifier
  BATCH_SIZE_DEFAULT, NUM_DECIMAL_PLACES_DEFAULT
  The number of decimal places used when printing numbers in the model.

It means that changing this decimal point should change the decimal points in the printed output of the trained classifier in WEKA explorer. However, when I try to change it (1,2,3,4,5 etc decimal points), it doesn't affect the number of decimals in the decision tree conditional statements in the printed output.
A snapshot:

k1 <= 0.12544
|   k4 <= 4.504
|   |   k22 <= 0.24044: Class1 (179.0)

Cheers


texts: Proof that Gini Impurity in a Decision Tree is Monotone Decreasing?

texts:
I asked this in a reply to an answer to another of my questions; but I think this merits its own question since I couldn't find an answer, and it's a pretty interesting question on its own.
Suppose we construct a decision tree for classification based on the Gini impurity function.  Can we prove that the weighted average of the Gini impurities of children nodes is always <= the Gini impurity of the parent node?
More precisely:
Let G(S)=sum_i p_i(1-p_i), where S is a finite nonempty set of points with known classification, p_i is the proportion of points in S with classification i, and the sum is taken over all classes. In the special case of binary classification, this simplifies to G(S)=2p(1-p), where p is the proportion of one of the classes.
Assume that every point x has a feature f. Denote the value of this feature by x(f). A splitting of S is defined as a partition of S into {S_left, S_right}, where S_left = {x in S: x(f) <= c} and S_right = {x in S: x(f) > c}. We require both of these sets to be nonempty.
A splitting is called good if
|S_left|/|S| G(S_left) + |S_right|/|S| G(S_right) <= G(S)

Must there always exist at least one good splitting?
Must all splittings be good?


texts: Accuracy noise patterns during model training

texts:
I'm training a logistic regression model on a small dataset. I have about 1300 samples that I split into a training and a testing set (70% and 30% respectively).

The training seems ok, however when I plot the accuracy of my model w.r.t. the epoch, some repeating noisy patterns appear at the end well after the accuracy is stabilised (after 800 epochs, see images below).

The training is done with an Adam optimizer. I'm using a learning rate of 0.04 and a weight decay of 0.07 that I found after doing a random search.

Is it something that may happen when training and is without consequence or does it reflect an issue with my data / training / software implementation ?


texts: Where can I find resources and papers regarding Data Science in the area of Public Health

texts:
I'm quite new to Data Science, but I would like to do a project to learn more about it.
My subject will be Data Understanding in Public Health.
So I want to do some introductory research to public health.
I would like to visualize some data with the use of a tool like Tableau.

Which path would you take to develop a good understanding of Data Science? I imagine taking some online courses, eg. Udacity courses on data science, but which courses would you recommend?
Where can I get real data (secondary Dummy Data) to work with?
And are there any good resources on research papers done in Data Science area with the subject of Public Health?

Any suggestions and comments are welcome.


texts: Datasets with only single feature variable

texts:
I mainly have 2 queries:

1) I want to know if there are any standard datasets available who have single feature variable?

2) In these types of datasets, can there be some data points having same values for feature variable but a slight variation in target variable?

For exmaple, following table (X = feature variable and Y = target variable):

X | Y

1 | 2.345

1 | 2.3

2 | 3.76

2 | 3.7

3 | 4

4 | 5.56

4 | 5.57

Kindly let me know a few of them.


texts: In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?

texts:
In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent,  Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics?

By metaheuristics I mean methods such as simulated annealing, ant colony optimization, etc., which were developed to avoid getting stuck in a local minima.


texts:
I am currently training a classifier for detecting resistors using TensorFlow Object Detection API. For that, I downloaded resistor images from ImageNet and I am currently labeling those who will be used for training.

However, I found two different kinds of resistors in my training images :

I want to know if it's safe to put them under the same label or it would be better to classify them under two different labels so that the training will be more accurate ?


texts:
So I have around 600 classes and 17k sentences for text classification.The feature vectors are trained using word2vec.

Vector_size = 100 ;
LSTM hidden units= 100

I am using class weight for imbalance problem.I wanted to know if LSTM is affected by sparsity in training data when max length is kept 18 but minimum length of sentences is 1.

And, how can we use keras train_on_batch to create batches of 5,10,25 for variable length instead of padding.


texts:
Let's assume a vanilla MLP for classification with a given activation function for hidden layers.

I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid is the activation function and -0.5 and 0.5 if tanh is the activation function.

What about ReLu ?

Should I normalise the network input between 0 and 1, -0.5 and 0.5, or -1 and 1

Any known best practices there?

I am not talking about normalisation of the input of the ReLu like using Batch Normalisation just before or just after the ReLu : https://arxiv.org/pdf/1508.00330

But I am talking about normalising the input of the whole network.


texts: What is the meaning of an empty SHAP graph in Explainable AI?

texts:
Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. For all of the instances in this dataset, the visualization outputted by SHAP has an output value of 0 and the higher/lower graph is empty (there are no features listed).

shap.force_plot(k_explainer.expected_value[0], k_shap_values[0], label_test_X.iloc[0])

When I use a different dataset and run the line above, SHAP outputs a graph and lists features.

What does an empty SHAP graph mean, especially in this case where there is an empty graph for each instance of a dataset? I am reading the documentation to help understand this situation, but I appreciate any insight into this.

Thank you in advance for the help!


texts:
I've been trying to implement the backpropagation algorithm using only numpy, I've already done the Keras version, but when implementing the numpy version, the loss is diverging as seen in the image below:

I initialized the weights using the glorot initialization (Keras default), used SGD with the same learning rate as in Keras, using sigmoid as activation function, but still, the algorithm can't converge for a simple XOR.

I think there is something wrong with the backward pass but I can't figure out what it is.

import numpy as np
import matplotlib.pyplot as plt

class MLP:
    @staticmethod
    def glorot_initializer(fan_in, fan_out):
        factor = 4 * np.sqrt(6 / (fan_in + fan_out))
        return np.random.uniform(-factor, factor, (fan_in, fan_out))

    @staticmethod
    def relu(x):
        return np.maximum(x, 0)

    @staticmethod
    def step(x):
        return np.heaviside(x, 1)

    @staticmethod
    def sigmoid(x):
        return 1 / (1 + np.exp(-x))

    @staticmethod
    def sigmoid_diff(x):
        sig = MLP.sigmoid(x)
        return sig * (1 - sig)

    def __init__(self, input_size, output_size, hidden_layers, hidden_size):
        self.input_size = input_size
        self.output_size = output_size
        self.hidden_size = hidden_size
        self.layers = list()
        self.grads = list()

        # input layer
        W = self.glorot_initializer(input_size, hidden_size)
        b = np.zeros((hidden_size, 1))
        self.layers.append(dict(W=W, b=b, type='input'))

        # hidden layers
        for i in range(hidden_layers):
            W = self.glorot_initializer(hidden_size, hidden_size)
            b = np.zeros((hidden_size, 1))
            layer = dict(W=W, b=b, type='hidden')
            self.layers.append(layer)

        # output layer
        W = self.glorot_initializer(hidden_size, output_size)
        b = np.zeros((output_size, 1))
        self.layers.append(dict(W=W, b=b, type='output'))

    def forward_pass(self, x):
        for layer in self.layers:
            x = np.dot(x, layer['W']) + layer['b'].T
            layer['output'] = x
            x = self.sigmoid(x)
            layer['activation_output'] = x
        return x

    def backward_pass(self, x, y, forward_pass):
        m = y.shape[0]
        dlda = 2*(y - forward_pass)  # considering MSE
        dlda = np.sum(dlda, axis=0, keepdims=True) / m
        self.grads = list()

        for i in range(len(self.layers)-1, -1, -1):
            grad = dict()

            z = self.layers[i]['output']
            dadz = np.sum(self.sigmoid_diff(z), axis=0, keepdims=True) / m
            if self.layers[i]['type'] == 'input':
                dzdw = np.sum(x, axis=0, keepdims=True) / m
            else:
                dzdw = np.sum(self.layers[i-1]['activation_output'], axis=0, keepdims=True) / m

            grad['b'] = dlda.dot(dadz).T
            grad['W'] = dlda.dot(dadz).T.dot(dzdw).T
            dzda_prev = self.layers[i]['W'].T
            dlda = np.sum(dlda.dot(dadz).dot(dzda_prev), axis=1, keepdims=True)

            self.grads.append(grad)

        self.grads.reverse()
        return self.grads

    def update_parameters(self, lr=0.2):
        for layer, grad in zip(self.layers, self.grads):
            layer['W'] -= lr * grad['W']
            layer['b'] -= lr * grad['b']

if __name__ == '__main__':
    epochs = 20000
    input = np.array([[0, 0], [1, 1], [0, 1], [1, 0]])
    target = np.array([[0.0], [0.0], [1.0], [1.0]])

    input_size = input.shape[1]
    output_size = target.shape[1]
    mlp = MLP(input_size=input_size, output_size=output_size, hidden_layers=2, hidden_size=2)

    loss = np.zeros(epochs)

    for i in range(epochs):
        fp = mlp.forward_pass(input)
        bp = mlp.backward_pass(input, target, fp)
        mlp.update_parameters(lr=.01)
        loss[i] = np.sum((fp-target)**2/fp.shape[0])

    plt.plot(target)
    plt.plot(fp)
    plt.show()

    plt.plot(loss)
    plt.title('LOSS x EPOCHS')
    plt.show()

Any help, would be very much appreciated!


texts: Modules on Python which are useful for missing Word/Letter prediction in text paragraphs from a coprpus

texts:
Can someone please recommend me some Python packages which I can use for missing word prediction in a paragraph of text. The missing words are a part of existing semantic list of words.( List of missing words are a common set of words like was ,were etc). If there are better packages in R which can help me with this  it would be very useful if you post it here.

Thank you.


texts: What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

texts:
I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what are the correct ordering of these.

AS of now, I think the process and the ordering are ->

Get Data
Delete Duplicates
Find Missing Values and Outliers
Create New Features
Deal with missing values and Outliers
Build a base model
Find the best features to select
Try and find different Models
Select the BEST model
Hypertuning of the Model

Please Do Provide if something is missing and correct the sequence.


texts: Is this correlation between distance matrices?

texts:
I have a set of objects. I have calculated two distance matrices: $X$ defining distance between each objects pair using metric $f1$, and $Y$ -- using metric $f2$. Now, I would like to understand if two objects are similar according to metric $f1$, then they are also similar according to metric $f2$. How can I do it?

For instance, $f1$ could say whether two objects have similar color, and $f2$ --- whether two objects have similar size. But metrics can be anything. For instance, we could talk about articles, $f1$ could be Jaccard distance measuring how many tags both articles share, and $f2$ could be euclidian distance measuring distance between word vectors of two articles. Now I would like to understand if two objects of blueish objects tend to be big, or whether articles tagged with "racism" have similar content.

Am I asking about correlation? How can I calculate it between $X$ and $Y$?


texts: How can I build a simulation environment that assess different risk policies?

texts:
I work in fin-tech and would like to build some sort of simulation program to assess how different inputs will impact net revenue. For example, if we create new policies based on ML scores, how would those have impacted our loss and revenue metrics?
While we can and do run online experiments, it would be desirable to simulate these impacts ahead of time.
Aside from something like reinforcement learning, I was thinking that Monte Carlo simulations might be the best approach. Anyone do something similar before or have suggestions?


texts: Information extraction with reinforcement learning, feasible?

texts:
I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the Google DeepMind & AlphaGo's stuff) to parse and extract information from text.

For example, could it be a competitive approach to structured prediction such as

Named Entity Recognition (NER), i.e. the task of labelling New York by "city", and New York Times by "organization"
Part-of-speech tagging (POS), i.e. classifying words as determinant, noun, etc.
information extraction, i.e. finding and labelling some target information in texts, for instance 12/03 is date given the context meaning 3 December and has the label "expiry date"
What would be a relevant modelling to do these tasks?

Rather naively I would think of a pointer that read the text from start to end and annotate each 'letter' by a label. Maybe it would learn that neighbouring letters in a 'word' share the same label, etc. Would it be able to learn long-term dependencies with this approach?

I am interested by any ideas or references related to this subject.


texts: How should I add reference key column to output after modeling is Complete?

texts:
0
I have created a csv of data with the following columns: (1) app_key (2) churn, (3)tenure
https://i.stack.imgur.com/NAlFF.png
I have performed the following code in order to drop app_key and churn
test_data = pd.read_csv('/Users/lawrence/Downloads/CHS_August_v4.csv')
test_data_ready = test_data.drop(labels = ["Churn", "app_key"],axis = 1)

I dropped the app_key column so that I could perform an XGBoost operation on the data set. Given that the app_key field is just a customer key, I needed to remove that in order to run by XGBoost operation. You can see the XGBoost operaton below.
  test_model = xgb_model.predict_proba(test_data_ready)[:, 1]
explainer = shap.TreeExplainer(xgb_model)
shap_values = explainer.shap_values(test_data_ready)
shap_output = (pd.DataFrame(data=shap_values, columns=test_data_ready.columns))

Shap_output is a dataframe with several columns of data that represent coefficients. I would like to produce a final output that contains app_key with the corresponding coefficients that have been output from shap_output so I can easily lookup which customers have low coefficients.
I attempted to do this by adding the dataframe of my original csv before dropping the values with my new output, but I'm unsure if this is the correct approach. My main concern is that the rows may not align since I'm unsure if there's any unique sorting. I'm unaware of how to check if sorting may be happening as well. below is the code I used to join my shap_output with my app_key
shap_output['app_key'] = test_data['app_key']

Can someone please let me know if this approach is the best way to ensure that I end up with one output that has the app_key and the corresponding outputs from shap_values? Or is there a better way to connect my app_key label with the corresponding coefficient?
Thank you so much!


texts: Cross Validation - Why does more folds increase variation?


texts: When is it useful to measure the Frobenius norm of a matrix?

texts:
In Deep Learning section 2.5 the author review some measures for the size of vectors and matrices.
When in general is it useful for someone to know these?
For instance they give the example of the Frobenius Norm which is analogous to the L2 Norm for matrices. Is this a metric that is more used in academics?


texts: How does Max Pooling (of size 2x2) change the size of receptive field in CNN?

texts:
I am often told that Max Pooling of 2x2 doubles the size of the receptive field from the previous layer. If that is true, I would like to understand how that happens.

I have already checked this article and this one. However, I am not able to understand the effect of max pooling on the receptive field.

Appreciate any help on this.

Thanks!


texts: What is Compatible Function Approximation theorem in reinforcement learning?

texts:
I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of.

What are these two conditions?

What is the logic behind the first derivative equality? Is it just that we assume these two derivatives should be equal since there should be some kind of connection with direction of value function approximation gradient and our policy like-hood?

Then what is the epsilon value? What is this mean square value? Q value with w parameters means the Q values function new approximation. And the other Q value is the value got by following our policy right? So is this OFF policy value approximation or ON policy? I think this is on policy since normally in the Q value update we take the max.


texts: Error loading tidyverse in R Studio - No broom package

texts:
I'm getting numerous errors when trying to install packages, namely tidyverse and ggplot.

The error is of the form are always in the form:

> library(tidyverse)
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
  there is no package called ‘broom’
Error: package or namespace load failed for ‘tidyverse’

I have already tried installing the package broom independently with dependecies = TRUE per the below

Having trouble installing and loading tidyverse- No DIB package

I've also tried restart my R session as well as installing and running an updateR package/ function per:

How can I run tidyverse package?

Thanks,
Marc


texts: Is there an appropriate use of adjusting class weights for a balanced dataset?

texts:
I ask this because I am currently working with a CNN model built for diagnosis of pneumonia. Originally, I followed a notebook on kaggle to build the model and thereby learn what each bit of code is for, etc.
The dataset used was rather imbalanced, with a far greater number of pneumonia cases than normal (healthy) ones. Hence the model.fit class_weight parameter was set to {0:6.0, 1:0.5}.
(0 being normal, 1 being pneumonia)
Since then, whilst working on the model and making adjustments, I acquired a number of new data to add to the model such that now the dataset is fairly balanced. In fact, I ensure that the data is loaded into the model so that it is exactly balanced, the dataframes used are coded to ensure an equal number of pneumonia and normal cases in the training testing and validation dataframes.
So, accordingly, I am now trying to remove the use of the class_weights parameter as (as far as I understand it) it is not necessary and may impart some bias in the results. However, in doing so, the model no longer seems to improve in accuracy. It essentially stalls on 0.5 indefinitely. Whereas, with the weights applied, I achieve 0.90+ accuracy.
Simply put, is there some reason for this? The code is quite long, but I'm happy to post it if it is deemed required, but I feel like this may be due to my lack of understanding than error in code (as it has otherwise been working fine and as expected). Thanks in advance.
EDIT: For the sake of clarity and understanding, I performed a grid search over possible values for applied weight values. It confirmed an appropriate choice as being 0:~4.0, 1:0.4, but also suggests 0:1, 1:5.0.
EDIT 2: For further clarity, a link to a github containing the model code and output files etc. https://github.com/GeeKandaa/ML-Code


texts: data type (int vs float) with sklearn models

texts:
I have a rather general question.
I am trying to create a classification model and my training data consists of various variables. I have figured out that it would be most efficient if I convert all my data into numeric types. However, as I am using 2D numpy array and I have to use float for values that are originally stored as integers. I wanted to ask if having such conversion would effect the efficiency of my prediction,
or simply:
Is there a difference between having 1.00-0.00's and 1-0's in a training set.


texts: How to train the model with for loop instead of the built-in epochs

texts:
i want to use a for loop for epochs instead of the built-in ones. Does these parts are similar.
1)
epoch=0;
while epoch in range(0, 30):
        print('Epoch', epoch)
        results = model.fit(X_train, Y_train,  batch_size=16, epochs=1, callbacks=callbacks)
        epoch +=1

results = model.fit(X_train, Y_train,  batch_size=16, epochs=30,  callbacks=callbacks)


texts: Beautifulsoup find_all returns TypeError: list indices must be integers or slices, not str

texts:
This should have an easy solution but I can't understand how to avoid it.
    content1 = soup.find('div', class_='fusion-text fusion-text-6')
    content2 = soup.find('div', class_='fusion-text fusion-text-7')

    for para in content1:
        comp_name = para.find_all('a')['href']
        print(comp_name)

The error comes because I have a list at ['href'] together with find_all.
        comp_name = para.find('a')['href']

doesn't return an error and gives the right output (an URL), but just the first one. Since I want to scrape all of them inside my content1 I wanted to use find_all but inevitably I get the error.
How can I avoid this?
Thank you.


texts: Is feature scaling at all needed for a feature set with a single feature?

texts:
I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a single feature in a feature set. For example, I have a quantity (events over time) on a time series and I'm forecasting for the future. In this case, do I need feature scaling (normalization or standardization?) or will it be unnecessary?
I searched online but couldn't find resources talking about the case where there's only a single feature in the feature set.


texts:
I have a CNN network in keras. I do the training on a cloud GPU. I get completely different accuracy and loss graphs when I run the training twice. I set random seeds as below, still no luck. Anything missing? Or is this normal behaviour on external GPU? I read that sometimes that they induce randomness because of certain libraries they might be using? I see %2-4 difference in accuracy everytime I run. So makes it difficult to judge my hyperparameter tuning.

import numpy as np
np.random.seed(3)
import tensorflow as tf
tf.set_random_seed(4)
import keras
keras.backend.clear_session()

from keras.layers import LeakyReLU
from keras.models import Sequential
from keras.layers import Activation
from keras.layers import Convolution2D, MaxPooling2D, Dropout, Reshape
from keras.layers.pooling import GlobalAveragePooling2D
from keras.optimizers import Adamax, Adadelta, Adagrad, Adam, RMSprop
from keras.callbacks import EarlyStopping, ModelCheckpoint,
 ReduceLROnPlateau
from keras.regularizers import l2,l1,l1_l2
from sklearn.metrics import precision_recall_fscore_support,
 roc_auc_score
from keras.models import model_from_json
from keras.layers.normalization import BatchNormalization
from sklearn.metrics import confusion_matrix, f1_score,
 precision_score, recall_score
import keras.backend as K
from keras.layers import Dense, Dropout, Flatten
from sklearn.preprocessing import Normalizer


texts: Why the performance of VGG-16 is better than Inception V3?

texts: I’ve initialised VGG16 and InceptionV3 with ImageNet weights and fine tuned using very small learning rate from the first layer on a chest X-ray dataset of 20K images (since the medical images are different from ImageNet classes). I reportedly find the VGG16 shows no overfitting and is more accurate than InceptionV3 that overfits and gives less accurate results. What is the reason?


texts: How to identify similar words as input from a dictionary

texts:
Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below

Input Terms

Subaaaash
Paruuru
Mettromin
Paracetamol
Crocedinin
Vehiclehcle
Buildding

Dict terms #this dict has around million records and I have to find a closest match for input term from this dictionary

Metformin 250 MG
Metformin
.....
Crocin
Vehcile
Paru
Subash

Now, I expect my output to be like as shown below

As you can see that red colored Paru is not a correct match for paracetamol, but that's the closest match we can get for the input term paracetamol from the dictionary. So, I would like to do matching based on

1) Word sounds (when pronounced). Phenotics

2) Spelling mistake corrections

3) Find the best matching word from the dictionary for input terms

Can you let me know how can I do the above?


texts: Recognition of objects in almost plain background

texts:
I want to recognize climbing holds in a climbing wall. I thought about implementing my own heuristic algorithm for background detection because it's almost plain, but I was wondering if this kind of task is well known in the community.

Does anybody know the right keywords associated with this? Any research papers or tools that are relevant?

I'm mostly interested in non-supervised learning to avoid having to create a training set for what I think it should be relatively easy but I might be very wrong.

This is a sample image I'd like to classify:

You can see 4 big holds, 2 small and a number of holes. Ideally I'd detect all of those with bounding boxes but having only the big holds would be good enough.
Any hint on how to approach this would be highly appreciated!
Thanks


texts:
I'm a newbie of Deep Q Learning. After read some papers and tutorials on the web, I tried to train a DQN to control a game using TensorFlow.

The input is the screenshoot of the game. The output is an float array [x, y](where 0<=x<=1, 0<=y<=1), which contorls the tap location of the screen.

But I found that almost all the DQN I saw before only has a discrete, and they can use the tf.one_hot function to generate all the state table easily.
And I don't know how to deal with my float output and how to perform my backpropagation with this float output.

How can train the to DQN output such a float result and perform a backpropagation?


texts: Does K - Means clustering on data reduced using PCA and the original data make any difference?

texts:
I am working on clustering and I have 90 features with 13500 data points and after removing the correlated variables which had pearson correlation more than 90% my feature space reduced to 70. Also, almost all my original 90 features has lot of values as zeros (more than 70-80% of data points). What I did in terms of algorithmic implementation is :

Ran K-Means on original data with 70 features (all numerical) by selecting number of clusters based on Silhouette index.
Ran K-Means by reducing the dimensions to 2 by selecting the number of clusters based on silhouette index.

What I observed and my corresponding question is :

K-Means on pca reduced data gave better clusters. Is there any way I can use this clusters that would make sense ? Like assigning the cluster labels from pca reduced data to the original data
How is the K-Means on original data and K-Means on pca reduced data different ? I understand that the pca would have reduced the data to two dims that I chose and has preserved the components with max variance. But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not.

Also, there are many implementation of K-Means like Lloyds (Python), Elkan(Python), Hartigan-Wong(R), Forgy(R), MacQueen(R). Which of these can be used for numerical vars and which one for categorical ? I think, wong is used for categorical variables not sure though. Also, which of these Implementations can I simply rule out ?


texts:
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable like location:

Location
----------
Californian
NY
Florida

We have to convert them like:

1  0  0
0  1  0
0  0  1

However, it was suggested that we have to discard one dummy variable, no matter how many dummy variables are there.

Why do we need to discard one dummy variable?


texts: Building Recommendation engine with Python

texts:
Which all are the equivalent or advanced libraries in Python for building recommendation systems like Mahout for  Collaborative Filtering and Content Based Filtering ?

Also is there a way to integrate Mahout with Python?


texts: What is the best way to classify data not belonging to set of classes?

texts:
I am building a multi-class support vector machine (8 classes to be precise) on an image dataset of pre-defined classes. And then I thought of a question:

  What if I have an image that doesn't belong to the set of predefined classes, what would be the outcome?

So I decided to experiment with it and the result was very bad. I got a higher accuracy for images that don't belong to any of the classes. Some images gave 98% accuracy, that they belonged to a particular class, even though my expectation should be that they should should have a very low accuracy.

I also tried using OnceClass SVM to first predict if it's part of the class or not. If yes, then what's the label? (Meaning I have 2 models).

But this doesn't seem to work as the OneClass SVM couldn't classify the "other" images well. Now I am running out of ideas of how to go about it.

How can I approach this problem?


texts: How to add confidence to model's prediction?

texts:
I am a newbie in ML working on a time series prediction project. The objective is to predict the future outcome of a time series (int valued, with different upper bound, think of it as different sized parking lot availability prediction) based on its historical value.

I'm currently using a regression approach using slide windows algorithm. I tried different ML models and they seem to be working okay(better than my baseline at least).

Now I'm trying to add confidence to my prediction, something like "I have 95% confidence that the outcome would be 2". I'm thinking about using the prediction mean squared error as a metric.

The problem is,

Is it feasible to assume the prediction error follows gaussian distribution and add confidence based on that?
What distribution should I use for highly discrete state space? For example, when there is only 4 possible states {0,1,2,3}, and I predicted 2.5 with mse 1, how can I distribute the possibility over those states?

Any advice on the general model architecture and confidence set up will be appreciated!


texts: Regression of discrete continuous averaged data

texts: If I have data that is continuous and discrete that is the average of a timestep. For example average power usage over the past 1 hour that is continually sampled every hour. How could I turn such data into data with a 1 minute timestep?


texts: Grouping/clustering similar words python

texts:
I have a question regarding grouping of similar words for example I have list of words give below:

artificialintelligence
Artificial Intelligence
AI
Machine Learning
ML
Data Analytics
Data & Analytics

I want to group these words into [Artificial intelligence, machine Learning, Data Analytics]

I have used difflib.get_close_matches() but that does not give me desired result
For example this is how difflib group:
Information Technology': ['Information Technology','Mobile Technology',
  'newtechnology']

I have also used fuzz.token_set_ratio() but that also does not provide me the desired result.  Neither Levenshtein does.

If there is any Machine learning algorithm or any python library please let me know.

Thank you


texts: Do I need to correct predict_proba by training fraction?

texts:
Many algorithms provide a predict_proba function indicating probability of a case to belong to that class (e.g. https://scikit-learn.org/stable/modules/generated/sklearn.svm.libsvm.predict_proba.html ).

Quoting from the answer by @Media at Explain Binary Classification with output 0.5 (True)

  Suppose that you have a car classifier for distinguishing between
  white and blue cars. during training you had 100 images of blue car
  and 20 images of white car. During recall phase, if for an arbitrary
  image you have 50 percent for each class...

If blue cars accounted for 83% of training cases, and I get predict_proba for a car to be blue to be 0.5, do I take the probability to be 0.5 or do I need to correct it by a factor of 0.83?

If I do need to correct, do I multiply the factor (0.5*0.83) or divide it (0.5/0.83) to get the correct probability?


texts: transition to data scientist for more money

texts:
Not sure if this is a right forum. Please advise if not.

I am currently employed as a Software Engineer and am fairly happy. Except, I'd like to make more money. I hear data science is the hottest field these days but wasn't sure if it paid more. I was contemplating to get a certificate like this one.

I understand in the beginning it might not be more than what I currently make (~100K) as I am new to the field. However, in few years (let's say about 3 years) is it common to make 150K+?

I have a MS in Computer Science, and was good at math classes. So I feel confident in being able to transition.


texts: F1 maximization with convolutional neural net. for an imbalanced dataset

texts:
I'm dealing with an imbalanced dataset for binary classification (about 70% to 30%). I was wondering what is the best way to optimize the F1 score for such a task when using a convolutional neural net.

As of now, I'm sampling the dataset in order to create a balanced training set and am using the mean of softmax_cross_entropy_with_logits (from tf) with a regularization term as my loss.

How can I optimize for the F1 score? As it isn't convex I can't just plug it in as my loss, right? Most papers I found mentioned finding the best threshold. However, these were relatively old, is there something better which can be done for convolutional neural nets ?


texts: can't get correct plot from alpha vantage and plotly

texts:
I am trying to plot a week stock data from alpha vantage with plotly but I get weird plots compared to other source's plot like google's chart for example.

data, meta_data = ts.get_intraday(symbol='MSFT',interval='1min', outputsize='full')
data = data.sort_index()

If I plot without the x axis labels, I get a correct plot but the x labels are replaced with integer ranges.

close_graph = go.Scatter(y=data['4. close'], hoverinfo=['y'])
close_layout = go.Layout(title='Intraday Times Series for the MSFT stock (1 min)')
close_fig = go.Figure(data=close_graph, layout=close_layout)

else if I do something like

close_graph = go.Scatter(y=data['4. close'], x=data.index, hoverinfo=['y'])

I get a weird plot


texts: Do I need to engineer lagged features when creating an LSTM for time series forecasting?

texts:
Long short-term memory networks are fairly complicated and I haven't completely wrapped my head around them.
It seems to me like the big gain in LSTMs for time series forecasting is the lacking necessity for lagged features: it determines on its own which lagged information is actually significant and remembers it for the next tiestep(s).
Should one still create lagged features as inputs for timesteps when training an LSTM? Like the output of the last timestep, or means and medians of the foregoing timesteps, means and medians for a specific class, distances, differences, etc.?


texts: How to use the fillna method in a for loop

texts:
I am working on a housing dataset.  In a list of columns (Garage, Fireplace, etc), I have values called NA which just means that the particular house in question does not have that feature (Garage, Fireplace).  It doesn't mean that the value is missing/unknown.  However, Python interprets this as NaN, which is wrong.  To come across this, I want to replace this value NA with XX to help Python distinguish it from NaN values.  Because there is a whole list of them, I want use a for loop to accomplish this in a few lines of code:

na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu', 'GarageType',
           'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

for i in range(len(na_data)):
    train[i] = train[i].fillna('XX')

I know this isn't the correct way of doing it as it is giving me a KeyError: 0.  This is kinda like a pseudocode way of doing it to visualize what I'm trying to accomplish.  What is the way to automate fillna('XX') on this list of columns?


texts: What makes a Tree-Structured Parzen Estimator "tree-structured?"

texts:
From what I understand the Tree-Structured Parzen Estimator (TPE) creates two probability models based on hyperparameters that exceed the performance of some threshold and hyperparameters that don't.

What I don't fully understand is why TPE is "tree-structured." Is the simple clustering of y > thresh and y < thresh sufficient to call it a "tree?" When I envision a "tree structure" I usually think of many different "forks," perhaps some that are not binary and some that are forks of forks. Although a tree with just one fork is still technically a tree, the name "Tree-Structured Parzen Estimator" seems like it would describe something much more complex.

My question is whether or not I am missing some other deeper conceptual "forks" inherent in TPE that would turn it into a "tree" when you look at the bigger picture.


texts:
I have around 72,000 Multivariate time series (MTS) with four Dimensions and of Length around 3000 milliseconds(not constant). It is (4*3000*72000) DLN. This MTS has tow possible outcomes either Pass or Fail.

How to select a better classifier that suits above criteria?

I have read some implementations of MTS classification. I cannot apply distance measure techniques for classification as the data is too lengthy and is not constant when compare to other MTS.

I am considering to apply below techniques for classification

1) Extract global statistical features from data and apply any classifier(KNN or Random Forest or neural networks)

what would be the best approach. Please guide me better technique and I am also open for other techniques to classify MTS


texts: How should we use a sparse matrix in classification?

texts:
I have a sparse matrix beside my data. I think these data could be beneficial but because it is sparse in usual usage is useless. Is there any method to transform it? for example, PCA, do you have any experience?

additional detail:

I working on titanic competition from Kaggle.com, I add nationality (base Name feature) to data. my nationality is something like: "Nordic,Scandinavian,Norway" so I have converted it to dummy variable, but then I have a sparse matrix.

I think usually sparse matrix could not be helpful for classification and we should transform it. (??? Is it true?)

I think in some way I should merge my features to reduce sparsity.


texts: What is the most efficient data indexing technique

texts:
As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented Database (which I try to write/play a little around with C#), which technique you suggest?

I read about MurMurhash-2 and specially v3 comments say Murmur is very fast. Also Lucene.Net has good comments on it. But what about their memory footprints in general? Is there any efficient solution which uses less footprint (and of course if faster is preferable) than Lucene or Murmur? Or should I write a special index structure to get the best results?

If I try to write my own, then is there any accepted scale for a good indexing, something like 1% of data-node, or 5% of data-node? Any useful hint will be appreciated.


texts:
We got several models with predictions. How can we compare scores of different models with each other?

We assume that we got xgboost models and scores distribution can be different for each model, so how can we compare scores?


texts: Using time series data from a sensor for ML

texts:
I have the following data for a little side project. It's from an accelerometer sitting on top of a washer/dryer and I'd like it to tell me when the machine has finished.

x is the input data (x/y/z movement as one value), y is the label on/off

Because the x values overlap for y=1 and y=0, I was thinking of using x and a rolling 3 minute window as inputs for a SVM:

xyz60=res.xyz.resample("60S").max()
X["x"]=xyz60
X["max3"]=xyz60.rolling(window=3, min_periods=1).max()

Is this a good approach for this kind of problem? Are there alternatives that might produce better results?


texts: When should you use deterministic classification rather than probabilistic

texts:
Probabilistic classifiers look really good because they give you more information than deterministic ones i.e. estimated probabilities of class memberships rather than just which class the model thinks an individual datum should belong to.
So in what circumstances would you choose a deterministic classifier rather than a probabilistic one?


texts: Should your learning rate get smaller as your neural net gets larger?

texts:
Is it correct to say that as you add more layers and more neurons, your learning rate should then decrease?
So, generally speaking, the larger the net, the smaller the learning rate?


texts: analysing stability of a measurement over time

texts:
I have some sensor data (actually many different sensors about 90 of them) and I have their readings over some time course. I was wondering on a couple of things:

1: What metrics can I use to assess if the measurements are stable across the time period. Is there something more I can do besides calculating the standard deviation which might give me an indication of the stability of the measurement?

2: What is the best way to visualise the data. So, I have about 90 sensors and I can compute something like box-plot for the various sensor measurements to visualise the distributions but my issue is that the sensors have different scale and I was wondering what might be an effective visualisation tool.

For example, I could have one sensor measuring temperature and one measuring pressure and the reading could be:

temp        pressure
51          9.3
49          9.2
50          9.3

So the scales are different. How can one visualise these sort of different  data types in one graphic as this could be much more efficient in communicating the information. I wonder if there is a way to use this graphic and the measurement stability metrics to indicate quickly if one or more sensors are misbehaving.


texts: Random forest multivariate forecast in Python

texts:
I am working with a multivariate time-series dataset and have put together a Random Forest code (see below) to forecast the variable TM at a future time (by training the model using data pertaining to two variables FL and TM). I know that the two parameters are closely correlated.
I was not sure if I got the code right initially, but after the training, when I tested it on the test data, I got an R$^2$ value of 0.98! This is my first attempt at multivariate forecasting and I was pleasantly surprised and hoped that I did not make any mistakes along the way.
So, before I proceed with this, I was wondering if someone could help double check my code below and advise if anything is obviously wrong that I might have overlooked. Thanks!
Here is a link to the data.
Also, any help with how I could make this multi-step, so I can forecast TM at any time $t+1, t+2,....t+n$, would be greatly appreciated!
Python code:
import seaborn as sns
from sklearn.preprocessing import StandardScaler,MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score

import numpy as np
import pandas as pd

import datetime

def table2lags(table, max_lag, min_lag=0, separator='_'):
    """ Given a dataframe, return a dataframe with different lags of all its columns """
    values=[]
    for i in range(min_lag, max_lag + 1):
        values.append(table.shift(i).copy())
        values[-1].columns = [c + separator + str(i) for c in table.columns]
    return pd.concat(values, axis=1)

data = pd.read_csv('Data_UW.csv')
data['DATE_TIME'] = pd.to_datetime(data.Date + ' ' +data['Time'])

for obs in range(1,6):
    data["T_"+str(obs)] = data.TM.shift(obs)

data.fillna(0.00,inplace=True)

training_data = data[data.DATE_TIME<pd.to_datetime('03/06/2019')]
test_data = data[data.DATE_TIME>=pd.to_datetime('03/06/2019')]
val_mask = (data.DATE_TIME>=pd.to_datetime('03/01/2019')) & (data.DATE_TIME<pd.to_datetime('03/02/2019'))
val_data = data.loc[val_mask]

clean_train=training_data[['DATE_TIME', 'TM', 'FL']]
clean_test=test_data[['DATE_TIME', 'TM', 'FL']]
clean_val=val_data[['DATE_TIME', 'TM', 'FL']]

X_train = table2lags(clean_train[['TM', 'FL']], 2)
X_train.TM_1[0] = 0
X_train.FL_1[0] = 0
X_train.TM_2[0] = 0
X_train.TM_2[1] = 0
X_train.FL_2[0] = 0
X_train.FL_2[1] = 0

X_train,y_train=X_train,training_data.TM

rfr = RandomForestRegressor(n_estimators=200,max_depth=20,criterion='mse',verbose=2,n_jobs=5)

rfr.fit(X_train,y_train)

X_test = table2lags(clean_test[['TM', 'FL']], 2)

X_test.TM_1[26803] = 0
X_test.FL_1[26803] = 0
X_test.TM_2[26803] = 0
X_test.TM_2[26804] = 0
X_test.FL_2[26803] = 0
X_test.FL_2[26804] = 0

X_test,y_test=X_test,test_data.TM

rfr.score(X_test,y_test)

test_data["TM_Pred"] = rfr.predict(X_test)


texts: Good accuracy on train dataset with cross validation, but low accuracy on test dataset

texts:
I'm using the adult dataset from UCI and trying to predict the native country of a person, based on workclass, education.num, race, gender and income. I'm using sklearn and KNeighborsClassifier.

With this, predicting the test split I get 0.917252 accuracy,  with K = 15. But, when I use the test dataset (given in the UCI link), that accuracy goes to less than 1%.

Where am I going wrong?

train = pd.read_csv('datasets/adult_data/adult.csv')
test = pd.read_csv('datasets/adult_data/adulttest.csv')

# Cleaning null values
train = train[train["workclass"] != " ?"]
train = train[train["occupation"] != " ?"]
train = train[train["native.country"] != " ?"]
test = test[test["workclass"] != " ?"]
test = test[test["occupation"] != " ?"]
test = test[test["native.country"] != " ?"]

category_col =['workclass', 'race', 'education','marital.status', 'occupation',
               'relationship', 'gender', 'native.country', 'income']
for col in category_col:
    b, c = np.unique(train[col], return_inverse=True)
    train[col] = c

for col in category_col:
    b, c = np.unique(test[col], return_inverse=True)
    test[col] = c

features = train.drop('native.country',axis=1)
target = train["native.country"]
features_test = test.drop('native.country', axis=1)
features_target = test["native.country"]

features_name = ['workclass', 'education.num', 'race', 'gender', 'income']

features = features[features_name]
features_test = features_test[features_name]

#spliting data into train and test data
X_train,X_test,y_train,y_test = train_test_split(features,target,random_state = 12)

k_values = np.arange(1,26)
scores = []

for i in k_values:
    clf = KNeighborsClassifier(n_neighbors=i)
    clf.fit(X_train,y_train)

    y_predict = clf.predict(features_test)
    # y_predict = clf.predict(X_train)

    scores.append(metrics.accuracy_score(features_target, y_predict))
    # scores.append(metrics.accuracy_score(y_train, y_predict))

print("Accuracy for {} is {}".format(np.argmax(scores),max(scores)))

plt.plot(np.arange(1,26),scores)
plt.title('Variation of accuracy with K value, with all features')
plt.xlabel('K values')
plt.ylabel('Accuracy')

plt.show()


texts: Calculate rating based on different parameters

texts:
I want to predict the rating of an area based on different parameter like crime rate, number of malls, number of hospitals etc. There is no historical data available, so I don't have any training data for this.

There are two scenarios:

Each parameter has same weighting.
Each parameter has different weighting.

How can I calculate this?


texts: What's the difference between Haar-like Features and Haar Cascades?

texts: Are they the same in terms of their algorithm? Or do they differ in their respective detection methods?


texts: Minimizing cross entropy vs minimizing negative probabilities

texts:
The cross entropy loss can be written as $L_1 = -\sum_i\sum_c y_{ic}\log P_{ic}$, where $i$ represents images and $c$ are the classes. $y_{ic}=1$ for the correct class.
Instead of L_1 I can minimize the following $L_2 = -\sum_i\sum_c y_{ic}P_{ic}$

When I use these as the loss functions to CNN I found that for a particular problem $L_2$ performs significantly better than $L_1$. However, I am unable to explain it. What could be the reason for this?


texts: Which design pattern is better for data pipelines: batches or one at a time?

texts:
I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes.

One thing I've noticed is that I often write code like this:

mediumInputArray => someFilterFn => someMapFn => someOtherMapFn => someOtherFilterFn => someFinalFn

All of those functions in that pipeline accept arrays, so method 2 doesn't start on the batch until method 1 finishes with all the array elements, and so one.

It seems often mediumInputArray evolves into hugeInputArray and I need to shard things to run on many machines. At that point dealing with batches becomes brittle and cumbersome.

Is it a better design pattern to just write a pipeline that operates on one atomic entity, and then have 1 job run from start to end?

What would be some design patterns that are related to this?

Is there a good book on advanced data science design patterns?

Thanks!


texts: Any useful tips on transfer learning for a text classification task

texts:
I am doing a supervised binary text classification task.
I want to classify the texts from site A, site B, and site C.
The in-domain performance looks OK for texts of each site. (92%-94% accuracy).
However, if I applied the model trained on texts of one site directly onto texts of another site(without fine-tuning), the performance downgrades a lot. (7%-16% downgrade for accuracy).
Approaches I already tried:

Doc2vec embedding(trained on texts from one site) + logistic regression.

Bert embedding + logistic regression. (Using bert-as-a-service to generate the embeddings based on google pre-trained bert models).

TF-IDF + logistic regression.

Pre-trained Word2vec embedding(average word embedding for text) + logistic regression.

All of those approaches don't work very well.
I knew that the performance downgrade is unavoidable, but I would like to get a maybe 3% - 5% downgrade.


texts: Classifying potential triggers for events (many-to-many relationship): Which algorithm to use?

texts:
I want to use Data Mining/Machine Learning for a problem and I'm not sure if there is a standard algorithm for my problem.

The problem is as follows: There is a set of Events and a set of Potential Triggers. Each trigger can give rise to none, one or several Events. I want to classify Potential Triggers based on their features into ones that do not cause any event and ones that do. So far this is a standard classification task. Now comes the twist: For some of the Events, there are multiple Potential Triggers. This usually means that one is the real trigger and the others have been labeled accidentally. If the algorithm generates a rule (assuming rule learning for the moment) that covers one of the potential triggers for an event, this event shall be considered "explained" so that the other potential triggers for this event do not have to be classified as triggers for this event any more.

The twist in the problem smells a bit like set cover (https://en.wikipedia.org/wiki/Set_cover_problem), so I'm wondering if there is a good algorithm at all.

Is there a published algorithm? And does my problem have a standard name under which I can search for further information?


texts: Interpreting confusion matrix and validation results in convolutional networks

texts:
I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:

  Architecture: InceptionV3
  Pre-trained InceptionV3 with weights from image net
  replaced last layer, retrained the second half of the network
  Classification: Softmax with cross entropy ploss
  Adam optimizer with keras default parameters
  Hyper parameters: Learning rate 0.0001, batch size 64
  Goal: Multi-label classification with exclusive labels
  Data: 17 classes ranging from 0-16 with 48.600 images in total
  Train, Test, Validation: 80%, 10%, 10%
  Training: 2 epochs training only last layer + 2 epochs training second half of network

Results

Based on my validation set I created a confusion matrix with the following results.

Here are also some more per-class performance metrics:

As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?


texts: Does it make sense to train a CNN as an autoencoder?

texts:
I work with analyzing EEG data, which will eventually need to be classified. However, obtaining labels for the recordings is somewhat expensive, which has led me to consider unsupervised approaches, to better utilize our quite large amounts of unlabeled data.

This naturally leads to considering stacked autoencoders, which may be a good idea. However, it would also make sense to use convolutional neural networks, since some sort of filtering is generally a very useful approach to EEG, and it is likely that the epochs considered should be analyzed locally, and not as a whole.

Is there a good way to combine the two approaches? It seems that when people use CNN's they generally use supervised training, or what? The two main benefits of exploring neural networks for my problem seem to be the unsupervised aspect, and the fine-tuning (it would be interesting to create a network on population data, and then fine tune for an individual, for instance).

So, does anyone know if I could just pretrain a CNN as if it was a "crippled" autoencoder, or would that be pointless?

Should I be considering some other architecture, like a deep belief network, for instance?


texts: Neural network is getting partially trained

texts:
So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am printing the output of the last layer it is definitely not the optimal one! Let me explain it a little further...

My training data is a pretty simple one

Input: {0,0,1},{0,1,1},{1,0,1},{1,1,1},{1,0,0},{1,1,0}
Expected output: {0},{1},{0},{1},{0},{1}

I have tried many variations, even some logical gates (AND, OR, XOR, etc.)! I have tried increasing the number of the neurons on each layer or increasing the layers. I tried both ReLu as the activation function and the common sigmoid one, even some combinations! I have played with the learning rate value but I am still getting the same results. On the above given example of the training set my network's output using ReLu is

0.001585
0.999990
0.000713
0.000109
0.000000
0.000000

and the error is

Error: 2.02248e-06

using the sigmoid function the output is

0.098032
0.840373
0.046706
0.036155
0.036184
0.059911

and the error

Error: 0.0289854

As you can see the expected output of the fourth and sixth element were supposed to be values close to 1 like the second element. I am facing similar problems with different data sets.

So how am I back-propagating the network?
I am calculating the output error like this

error = 2.0 * (expected_output - actual_output)

and I am updating the weights like this

w += learning_rate * transposed_layer_output * layer_delta

Am I missing something?


texts: Building a time-series demand forecasting model

texts:
I am forecasting demand for certain types of goods and services, which I expect to be correlated to a sub-set of a basket of macroeconomic indicators (considering 15-20 indicators)

I do not know which indicators influence the demand more, whether they have a simple correlation influence or whether a derivative of change influences the demand (i.e. GDP or GDP change for example) or whether there’s a delayed effect on demand (e.g. increased government spend in last year may better predict this year’s demand?). Some macro indicators may be correlated to each other.

I have some basic hypotheses on likely indicators - that may be right or wrong.

Questions 1. What are good time-series forecasting models? What can be considered, apart from just a multivariate regression? 2. Is there a tool whereby I can input the historical demand, historical macro indicators, which will then output which set of indicators best predict the demand and which model works best?

I know how to do regressions in excel, but that’s just one set of indicators at a time. 20 indicators (plus derivatives, plus lag) throws up so many possibilities I cannot manually simulate.

Any help appreciated.


texts: (RL Curiosity) - "Exploration by Random Network Distillation" - what's the benefit?

texts:
Curiosity-Driven learning motivates the agent to explore unseen states. It does it by rewarding the agent more when its expectation differs from the actual next state. (orange color on the diagram). This extra reward is called "intrinsic reward" $r^i$ which is added to the usual reward (we call it with a new name "extrinsic reward" $r^e$).  Here is a video, explaining this architecture

But this causes issues in states that change unpredictably ("noisy TV screens" or tree leaves that move because of the wind). The agent will then get stuck, contemplating them forever, because the ExpectedState - observedState is very high, resulting in high reward.

As I understod, the authors tackle the problem by intruducing the Inverse-Model (blue on the diagram).
It tries to predict an action that was taken to get from currentState to the observedState.

This teaches the agent to only consider features that are relevant to the transition from currentState to nextState, and ignore stuff like noisy TV-screens.

Question:

Why is there a paper called Exploration by Random Network Distillation that builds on top of the original paper? To me, it seems to solve this same issue.

  Namely we
  predict the output of a fixed randomly initialized neural network on the current observation

Wasn't the problem already tackled by the Inverse Model (blue on the diagram) in the original paper?

When using Random network distillation, the agent tries to predict the embedding
of a current state, not the next state as I thought initially.

If the agent sees a random frame on TV, initially it will be receiving high “intrinsic” rewards, because it doesn’t match the output of the fixed random target network. An embedding produced by the target network is always unpredictable (although repeatable), meaning the online network will need to overfit, which is what we want in this case. This approximates the state-visit count. So RND is an interesting form of curiosity. Is more useful than vanilla?


texts: Can we define a data partitioning in K clusters, by cutting the branches of the tree at some levels in the tree below the root node?

texts: Assume we have a dendogram (hierarchical clusterisation tree), can we define a data partitioning  in K clusters, by cutting the branches of the tree at some levels in the tree below the root node?


texts: Detecting if a sentence contains a numeric series

texts:
I have many different strings of text. These strings of text are labels for particular things. But these labels are sloppy, sometimes one label is used for many different things. For example:

"Brown foxes edition 1999 series 1-6 EDI"

"Light [old] seasons 1,2,3,4 other gibberish"

I would like to answer the question: "If the label contains a series, does that serie contain the value N?" For the examples above 6 would be included in the first one, but not the second.

Initially I thought of using regexes but that quickly grew out of hand. Digits appear everywhere, people can get very creative with separators and the location of the series in the label is not fixed.
There are many different ways the labels denote series.

What I can do, however, generate labels with series. I'll just grab a bunch of separators, a start and an end digit and iterate. This gives me a nice labeled training set.

Naive Bayes comes to mind for this problem but I'm not sure what good features would be.

update

Let me try to clarify.
Given a label and given a number determine if that number is contained within that label.

For example: Given I'm looking for season 2 (the number). Does "Pioneer One 2011 seasons 1-3" contain season 2?


texts: Accuracy is lower than f1-score for imbalanced data

texts:
For a binary classification, I have a dataset with 55% negative label and 45% positive labels.
The results of the classifier shows that the accuracy is lower than the f1-score.
Does that mean that the model is learning the negative instances much better than the positive ones?
Does that even make sense, to have accuracy less than the f1-score?


texts: What is the difference between causal discovery and inverse modeling?

texts: I do not see these words used interchangeably, but they seem to be similar. In inverse modeling we are trying to find causal factors given an effect. In causal discovery, we are also looking for causal factors, right? How would you use these terms in different situations?


texts:
I want to convert the input data available as C4.5 file format into a R data frame.

The data set can be downloaded from the following link
https://archive.ics.uci.edu/ml/machine-learning-databases/communities/

and is avaialble as .NAMES and .DATA files.

My approach was to convert it to a csv file using WEKA and then import the csv file into R. Could not do it as WEKA detected a error while parsing the C4.5 file.

Can some one point me in the right direction.

Thanks


texts: The Why Behind Sum of Squared Errors in a Linear Regression

texts: I'm just starting to learn about linear regressions and was wondering why it is that we opt to minimize the sum of squared errors. I understand the squaring helps us balance positive and negative individual errors (so say e1 = -2 and e2 = 4, we'd consider them as both regular distances of 2 and 4 respectively before squaring them), however, I wonder why we don't deal with minimizing the absolute value rather than the squares. If you square it, e2 has a relatively higher individual contribution to minimize than e1 compared to just the absolutely values (and do we want that?). I also wonder about decimal values. For instance, say we have e1 = 0.5 and e2 = 1.05, e1 will be weighted less when squared because 0.25 is less than 0.5 and e2 will be weighted more.   Lastly, there is the case of e1 = 0.5 and e2 = 0.2. E1 is further away to start, but when you square it 0.25 is compared with 0.4. Anyway, just wondering why we do sum of squares Erie minimization rather than absolute value.


texts: Modelling if condition of multiple estimators in a pipeline

texts:
How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline.

if feature_x < constant:
    result = pipeline1.predict(feature_vector)
else:
    result = pipeline2.predict(feature_vector)

Other than modelling it as custom transformer/predictor, is there a alternate way to model it in a pipeline


texts: Expected behaviour of loss and accuracy when using data augmentation

texts:
I have implemented a convolutional neural network in Keras, and I use off-line data augmentation in the training set. The way I do this is that I create batches of training data in separate files (because they are not generated in Python), I load one batch/file at a time and run a single epoch of training on it, so during training the exact same data is never seen twice:

for current_batch in range(1, number_of_batches + 1):
    # Load new (X_training, Y_training) pair.

    # Fit for a single epoch.
    model.fit(X_training, Y_training, epochs = 1, batch_size = 32, validation_data = (X_validation, Y_validation))

My understanding is that this is what Keras would do internally if I used it's native (image) data augmentation tools, namely, for each epoch, generate a brand new set of data. I don't use validation split as can be seen above, I rather supply (non-augmented) validation data manually, which is currently the same for each training batch.

My question is this: What is the expected behaviour of (binary cross-entropy) loss and accuracy as a function of the number of batches, in a setting like this? If I performed training for $N >> 1$ epochs on the same training set, there would be a chance of obtaining a very small training loss, whereas in this case, my guess is that I cannot expect that? Indeed, from what I've seen so far, the training loss and accuracy are a bit noisy and how to use stopping criteria etc seems not so easy.


texts: Neural network with multiple inputs from various angles, contradictory features

texts:
Is there a way for the neural network to communicate with each other. I.E. An image of a rectangular prism can look like a cube from one perspective, but as soon as it's rotated, we know it's a rectangular prism.

Is there a way for the neural network to communicate with itself such that when it sees "oh it's a rectangular prism" it disregards it's first thought that it's a cube?


texts: Problem encountered while updating a particular cell of a pandas data frame in Python

texts:
I am trying to read in an excel file into a pandas data frame using pandas.read_excel() function. The first column is 'Date'. The first entry into that column has asterisk mark which I want to remove. Here are the first 2 rows of the data frame DF.
0   1/4/1996*   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN
1   1996-01-11 00:00:00 13881   11735   2532    4188    32064   21989   3819    2586    0   NaN NaN`

In order to remove that *, I am doing this:DF['Date'][0] = '1/4/1996'. However, once I do that, I am getting a warning message
/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  app.launch_new_instance()

and it changes the cell value, but, it also attaches another 'Date' column at the end of the df like this:
0   1/4/1996   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN 1/4/1996

I don't understand why this is happening.
I would greatly appreciate it if someone can explain this or point out the correct way of changing the cell value.


texts: Implementing simple linear regression using a neural network

texts:
I have been trying to implement simple linear regression using neural networks in Keras in hope of understanding how to work in the Keras library. Unfortunately, I am ending up with a very bad model.

Here is the implementation:

from pylab import *
from keras.models import Sequential
from keras.layers import Dense

#Generate dummy data
data = data = linspace(1,2,100).reshape(-1,1)
y = data*5

#Define the model
def baseline_model():
   model = Sequential()
   model.add(Dense(1, activation = 'linear', input_dim = 1))
   model.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['accuracy'])
   return model

#Use the model
regr = baseline_model()
regr.fit(data,y,epochs =200,batch_size = 32)
plot(data, regr.predict(data), 'b', data,y, 'k.')

The generated plot is as follows:

Can somebody point out the flaw in the above definition of the model (which could ensure a better fit)?


texts:
I am performing Named Entity Recognition using Stanford NER. I have successfully trained and tested my model. Now I want to know:

1) What is the general way of measuring accuracy of NER model ?? For example what techniques or approaches are used ??

2) Is there any built-in method in STANFORD NER for evaluating the accuracy ??


texts: When should the bias b be updated with weights w and when should it be updated seperately?

texts:
It seems in some Machine Learning models, the bias term $b$ is updated just like other weights $w_i, i=1...n$. For example, in Logistic Regression, using SGD, $b \ \text{or} \ w_0$ is updated with:
$$b \ += LearningRate *(target - w^T*x) $$

However in SVM, $b$ is updated sperately from $w$ by using support vectors $y_s$ (i.e., $y_s(w^Tx_s+b)=1$):
$$b=y_s-\sum_{i=1}^{n}\alpha^*_iy_ix_i^Tx_s$$
For the ML models with the linear model pattern $y=w^Tx+b$, is it only SVM updates bias $b$ seperately from $w$?


texts:
I observed that my GPU's memory is being consumed but the Utilisation stays 0. Because of this, my model is taking forever to load. I have tweaked this code to handle multilabel data. The only changes being-using y as a matrix instead of vector and different methods of calculation of cost and errors(Sorry that I am not able to provide the actual code). We can observe that the theano function train_model is taking a lot of time. I am a beginnner at Theano and also this method internally seems to having cascading variables and methods as well which is making me heard to debug.

What might be the possible reasons for such a scenario?

What debug methods should I use for this?


texts: Example of a problem with structured output labels

texts:
I'm studying SSVM (Structured SVMs).
On my book is stated that Structured SVM is an extension of the SVM, in which

  Each sample is assigned to a structured output label z ∈ K, e.g. partitions,
  trees, lists, etc.

It's not clear to me what a structured output label is.

Could you provide me with an explanation of this term along with some tangible examples?


texts: Pandas dataframe with multiple observations per model

texts:
I currently have a pandas dataframe with the following format

    Model      Metric                     Value
--------------------------------------------------------------
0     Ours     Accuracy  [0.79, 0.79, 0.82]
1   Theirs     Accuracy  [0.68, 0.56, 0.64]
2     Ours  Sensitivity  [0.64, 0.55, 0.55]
3   Theirs  Sensitivity  [0.82, 0.82, 0.78]
4     Ours  Specificity  [0.68, 0.48, 0.6]
5   Theirs  Specificity  [0.68, 0.48, 0.6]

In the evaluation script I am writing I want to be able to take into account situations where the training of a model is repeated multiple times, the results are stored in an numpy array (Value column). For visualization with seaborn, i believe i need a long form where it looks something like this:

    Model      Metric                     Value
--------------------------------------------------------------
0     Ours     Accuracy                   0.79
1     Ours     Accuracy                   0.79
2     Ours     Accuracy                   0.82
3     Theirs   Accuracy                   0.68
4     Theirs   Accuracy                   0.56
5     Theirs   Accuracy                   0.64
6     Ours     Sensitivity                0.64
...   ....     ...                        ...

I cannot figure out how to do this.


texts: Hyperparameter tuning does not improve accuracy?

texts: I am working on titanic dataset, I achieved 92% accuracy using random forest. However, the accuracy score dropped to 89% after I tuned it using Gridsearch. Now, I was wondering if it caused by imbalanced dataset, since it has only 342 out of 891 passenger survived the disaster. Would appreciate the clarification.


texts: If there are no missing values in our training set, should we accommodate missing values in an unseen test set?

texts: My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test set is 'locked away' during training time. Should I peek at the test data to see if there are missing values, or should I fit imputation for all features on a complete training set?


texts: Reason behind choosing Neural Network for classification

texts: Given a two class multi dimensional classification problem, what reason would you give to choose Artificial Neural Network for carrying out the classification instead of Support Vector Machine or other classification methods?


texts: Implementation of Tree Kernels in Python

texts: I'm looking to use a tree kernel (e.g. Zelenko et al., 2003, Moschitti 2006, etc.) for text classification in Python. However, I'm still a little unclear on the implementation. Is there an existing implementation (in any language) that I could use as a reference, or is there a 3rd-party tree kernel library I could use?


texts: Variable selection methods in survival analysis

texts:
I am using two models for my research. One is the coxph regression model and other is survival trees. I have found the scikit-survival package in python which is able to identify contributing variables and also rank them, but there is no reference to the papers.

I am looking for latest(published after 2010) research papers which implement variable selection methods specifically for survival analysis and if possible their GitHub links. I tried looking up papers but all I can find are papers which are published way before and there are not many citations for those papers. Can someone please provide me with the papers.

Thanks in advance!


texts: Topic Similarity Measure | Multi-class Text Classification Model

texts:
I am trying to build a multi class text-classifier that classifies whether the tweet belongs to one of the categories ( Advise or Science or others )
let the input be any tweet like this ,
Input :

The goal of teaching should not be to help the students learn how to
memorize and spit out information under academic pressure. Brain

The purpose of teaching is to inspire the desire for learning in them and
make them able to think, understand, and question.

#maths #richardfeyman

Output :
( Learning : 98% , others : 1 % , Science : 1 % )
For now i came up with this
Idea 1 :
Maybe i can use LDA to get the topics of the tweet and perform semantic matching between these labels ( learning , science , others ) so one with the highest score can be choosen as right one (argMax(...) ).
What do you think about the above idea ?
Can anyone please enlighten me or point me in the right direction .


texts: Tips & Tricks on training DCGAN on small dataset

texts:
I have made a DCGAN which I am trying to train on custom dataset of only 1200 images. I have tried to gather more, but even gathering these 1200 was hard enough. If you are wondering I used Google Chromes extension "Fakun Batch Download Image" to gather my dataset.

TRAINING DETAILS:
In training procedure I am simultaneously updating parameters of both, Generator, and Discriminator network. I've read that it works much better then training only one player ( Discriminator ) for K steps and then other ( Generator ) for one.

QUESTION: Should I perform maybe some kind of transformation on all of those images and then merge transformed images with the initial ones, or something similar?


texts: Is it possible to customize the activation function in scikit-learn's MLPClassifier?

texts:
Scikit-learn lists these as the implemented activation functions for it's multi-layer perceptron classifier:

‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x
‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).
‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).
‘relu’, the rectified linear unit function, returns f(x) = max(0, x)

Does Anyone know if it is possible to implement a custom activation function? If not, can someone point me to a library where this is possible?

Thanks!


texts: What is the preferred approach for this problem?

texts:
I have the Data of 10,000 users Time Session in a website/App, The Login time, logout time, the person activity,

The Data is available for 60 days ( per user )

Using this 60 days data for 10k users, can I predict the active time of the respective user on 61st day? if yes what is the best approach and please suggest which type of problems I can refer to solve this

Thank you


texts: Predicting survival on Haberman's dataset

texts:
I am currently involved in analyzing a particular dataset called Haberman Survival Dataset.
The dataset contains cases from a study that was conducted between 1958 and 1970 at the University of Chicago's Billings Hospital on the survival of patients who had undergone surgery for breast cancer.

Attribute Information:

Age of patient at time of operation (numerical)
Patient's year of operation (year - 1900, numerical)
Number of positive axillary nodes detected (numerical)
Survival status (class attribute),
 1 = the patient survived 5 years or longer,
 2 = the patient died within 5 year

For this dataset I have first processed/cleaned the dataset using R, and now I am planning to collect a sample from this population and then train the sample data (training set) with some suitable machine learning algorithm and build a suitable model (1-3 are predictor variables, 4 is the response variable), and then again collect the sample (test set) from the population and then apply the model that I have built to test set and predict the outcome.

What machine learning models can be used to achieve this objective?


texts:
Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run.

What are some of the best ways to find the most influential columns/features? I know it's important to know what models to run first, but is there a general way to find columns that's more important than others?

So far I've already removed the columns that are 80%+ empty, but I still have over 600 columns and I have no idea where to start. Please let me know your thoughts! I'm kind of new so any pointers will help.


texts: Python using collection's counter and simple count on dataframe are returning different results for products that are most often sold together

texts:
I am following a real data science project on youtube and learning some new scripts and techniques to go around the data and extract insights. The teacher wants to check what are the products that are most often sold together, and he used the following script:
from itertools import combinations
from collections import Counter
counter = Counter()
for row in df['Products Sold']:
    row_list = row.split(',')
    counter.update(Counter(combinations(row_list, 2)))

counter

And the result was:

('iPhone', 'Lightning Charging Cable'): 1005,
   ('Google Phone', 'USB-C Charging Cable'): 987,

   ('iPhone', 'Wired Headphones'): 447,

   ('Google Phone', 'Wired Headphones'): 414,

   ('Vareebadd Phone', 'USB-C Charging Cable'): 361,

   .....

We can change 2 to 3 or more to check more combinations. but for me with a simple line:
x = Counter(df['Products Sold'])
x.most_common()

Or even more simpler:
products = df.groupby('Products Sold').count().sort_values(by='Order ID', ascending=False)
products

The result was:

iPhone,Lightning Charging Cable   882
Google Phone,USB-C Charging Cable 856
iPhone,Wired Headphones   361
Vareebadd Phone,USB-C Charging Cable  312
Google Phone,Wired Headphones 303

You can clearly see that there is huge difference between both methods. If we take 'Google Phone', 'Wired Headphones' you will see 2 numbers (414) and 303.
What is the correct method in that case?
here is the data link.


texts: Ways to simulate weather data over several periods (Python or R)?

texts:
I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding precipitation and temperature variables would be a great idea. I tried finding some relevant external data but most of it is abstract and spread out. How would one simulate dynamic data in Python with varying means, highs and lows for say six months on a daily basis, taking into account average temperatures/precipitation for each month?

So if I have the mean temperatures (C) as below for state A:

year    Jan   Feb   Mar   Apr   May   Jun
2017    5.5   6.0   12.0  15.0  20.0  25.0

I would like data to be simulated as below without really doing for each month since that would make the whole task very tedious:

Duration     Temp
2017-01-01   5.0
2017-01-02   5.1
2017-01-03   4.9
.
2017-03-01   7.8
2017-03-02   9.0
2017-03-03   9.5
.
2017-06-30   26.7

Are there ways to achieve this in Python (or R)?


texts:
I'm using mean squared error as reconstruction error for my autoencoder. The dataset is ECG (time series) and model is conv1d.

I assumed MSE as the best option for reconstruction error, but it's failing sometimes. What can be the plausible reasons for it not working.

In general when does MSE doesn't works or what are the disadvantages of MSE.

Please answer in general scenario as well as my specific case.


texts: Choosing the right model for predicting demand

texts:
We have a data set of 300,000+ records that looks something like the following:

Item ID, Quantity 2017, Quantity 2016, Quantity 2015, Quantity 2014, Quantity 2013
1111, 100, 50, 25, 10, 0
2222, 0, 10, 100, 500, 1000
3333, 10, 0, 5, 2, 4

We are currently attempting to find the best model to predict Quantity 2017 based on the previous quantities for each record. We have tried Decision Tree Regression, Multiple Linear Regression, and Random Forest Regression (10, 100, 1000 trees) but our results were a bit too far off for approval. We are using 80% of our data for training and 20% for testing.

Are there any models that are better suited for this type of calculation?

We are also concerned that perhaps our data structure is part of the problem and we should re-evaluate. Is there another structure that might be better?


texts: How does scikit-learn decision function method work?

texts:
The scikit-learn docs say it is the signed distance of that sample to the hyperplane.

I've taken the sum of the weights and their corresponding coefficient and added the intercept to that sum but this does not return the value given by the decision_function method.

When doing classification tasks how is the signed distance different from taking the linear summation of the weights, feature values, and intercept?

For example, If I have the following feature values, intercept, and weights I would calculate the distance from the point to the hyperplane as...

$x_1 = 3, x_2 = 2$

$w_1 = -0.28, w_2 = -0.20$

$b = -0.68$

$$\theta^T \cdot \textbf{X} + b$$

When sklearn calculates the decision_function it returns

return safe_sparse_dot(X, self.coef_.T,
                               dense_output=True) + self.intercept_

Where self is our fitted classifier and X is the example for which we are calculating the decision function.

How can I replicate what decision_function is doing?


texts: What does these points mean in Naive Bayes?

texts:
I have two concept related questions related to Naïve Bayes.

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

Regards, Akshit Bhatia


texts: Automatically categorize parts of a piece of writing

texts:
Suppose I had a piece of writing. The document contains aspects like questions, assertions, examples, and explanations. Is it possible to use Natural Language Processing to categorize each sentence of the writing into such aspects, thus creating a code like 1.1:assertion, 1.2:explanation, for instance, meaning paragraph 1, sentence 1 assertion, and paragraph 1, sentence 2, explanation?

If that is possible, I would appreciate if someone would share their knowledge about known algorithms for performing this task.


texts: Recommendations to generate a training set of images

texts:
We are about to start a series of experiments in physics which will generate 8.5 thousands of images per month. This is a study on microscopic material reactions so the images are not 'normal' objects commonly found in sets available online (cats, cars, bicycles, etc.). The objective of the study is to generate those images to document our other measurements, and the the 2ry objective is that those images can be used to train an algorithm and later use that algorithm to classify objects in the rest of the images (including future studies, this is very important). So, given that we still did not start to acquire the images, we have the opportunity to "make it right" (if this even exist) from the beginning. We intend to generate and use those images in the following few years to come.

I am new to the ML world, although I have some experience in image analysis. We still did not decide which algorithm/approach we will use. Could be YOLO, DeepLearn, TensorFlow, OpenCV, etc. In principle it would be the easiest possible for non specialists (we will rather apply ready-made solutions instead of writing our own algorithms).
Since both image acquisition and object detection/recognition solutions change so fast all the time, and assuming that the train set should be ideally very similar to the images I plan to analyze with the trained algorithm, I have a few concerns (below) for a long-term projects. Could you give me your opinion and suggestions about the weight of each one (maybe I’m worrying too much) and how to circumvent (if possible) some of them?

1) What happens if I start the project with a given camera (eg Nikon DSLR D3500, 24.2 MP, 6000x4000 px, Full HD ) and then I switch to another one quite different (eg Raspberry Pi camera V2, 8MP, 3280x2464 px, Full HD)? If the 1st images are aquired by camera 1, then I assume I will lose accuracy of the object recognition algorithm due to different image resolution, is that so? How bad is this? What should I do to minimize this? Can I eg mix images from both cameras into the training set to avoid large deviations of the accuracy? I suspect this must be an old and very commented problem in object recognition. If I generate a training set with camera A, then can the algorithm work well in images acquired with camera B? For example, would it help if I take images from camera A and lower the resolution to match that of camera B (or just lower the resolution, since I not necessarily always know which camera we will use in the future)?

2) I read that one can make substantial changes to the training set (posterize, flip, decrease color depth, blur, etc.) to broaden the application of the algorithm. Would this enhance the accuracy of the ML algorithm to make it robust against camera changes?

3) As mentioned above, we still did not decide which ML technique we'll be using. Is this a decision we need to make before starting the training set generation?

4) Any other suggestions of ‘good practices’ on generating the training set and prepare for future changes in image acquisition technology?


texts: Finding image based on closest camera pose

texts:
I have a camera matrix world of a 3D scene (let's call it A):
[[ 6.86118087e-01 -7.27490186e-01  1.11022302e-16  0.00000000e+00]
 [ 3.82193007e-01  3.60457832e-01  8.50881106e-01  0.00000000e+00]
 [-6.19007654e-01 -5.83804917e-01  5.25358300e-01  0.00000000e+00]
 [-2.89658661e+00  5.19178303e-01 -3.50478367e-02  1.00000000e+00]]

And, I have 1000 2D images of the scene. For each image, I have pose (camera to world) as follows:
[[-9.55421e-01  1.19616e-01 -2.69932e-01  2.65583e+00]
 [ 2.95248e-01  3.88339e-01 -8.72939e-01  2.98160e+00]
 [ 4.07581e-04 -9.13720e-01 -4.06343e-01  1.36865e+00]
 [ 0.00000e+00  0.00000e+00  0.00000e+00  1.00000e+00]]

I want to find the image that is closest to A based on their pose matrix. How do I compare A with the camera pose of each image?
What is closest?
Just a little background, we asked annotators to write descriptions of an object in a 3D scene. Along with that, we also captured camera parameters, such as matrix world (see A above), center, lookat, and dof, so that we can estimate where in 3D scene were they looking at when they wrote the description. So, now I am trying to create a training set of images (one image for each scene from 1000 available) and would like to find the image that best matches the recorded camera parameters.
Please let me know if more information is needed for clarity.


texts: How to handle missing IP addresses for a ml model?

texts:
I am using netflow data for a ml project that includes IP addresses. But it seems there are many missing values in IP address column. One way is to discard those, but I am trying to retain as many data points as possible since the dataset is bit small. I am wondering if there is a way to handle missing IP addresses (other than dropping those values) that anyone has used previously or that I can try out.
Thanks for your suggestions.


texts: Determine model hyper-parameter values for grid search

texts:
I built machine learning model for Ridge,lasso, elastic net and linear regression, for that I used gridsearch for the parameter tuning, i want to know how give value range for **params Ridge ** below code? example consider alpha parameter there i uses for alpha 1,0.1,0.01,0.001,0.0001,0 but i haven't idea how this values determine each models.(ridge/lasso/elastic) can some one explain these things?

 from sklearn.linear_model import Ridge
    ridge_reg = Ridge()
    from sklearn.model_selection import GridSearchCV
    params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
    Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
    Ridge_GS.fit(x_train,y_train)
    Ridge_GS.best_params_


texts: Minimal example: Keras functional API & multi-input/multi-output regression

texts:
Problem:

I have a regression problem, where I want to predict two or more numerical outcomes $y_i$ based on a number of numerical features $X_i$. The model would look like: $$y_{1,i}, y_{2,i} = \beta X_i + u_i.$$

I understand that the Keras functional API could be used to solve a problem like this. In an NN setting I would use ordinary densely connected layers to solve the problem. However, I have no experiance with the functional API and I'm not sure if the problem can be solved (at all) using the functional API.

Question:

Can someone point me to a full-fledged minimal example on how to use the functional API for multi-input/multi-output regression as outlined above? Ideally, I'm looking for something like the code examples provided for more basic problems as described in "Deep learning with R" (code can be Python or R as I work with both).

Any additional comments on model/method choice are highly appreciated as well. However, please note that I don't want to estimate/train individual models for each outcome $y_i$, I really aim at predicting all outcomes in one model.


texts: Are Denoising Autoencoders for anomaly detection on structured data?

texts:
Can denoising autoencoders be used for anomaly detection on structured data?

I know I can use denoising autoencoders for anomaly detection on images, but I don't know if they can do it for structured data as well? If not, why?


texts: What is different between R2 and mean of R2 in multiclassification probelm? Which one is correct?

texts:
I have a question. I have a big dataset (unfortunately confidential).
What I did?
I have trained my model with Naive-Bayes.
BRNBReg=BernoulliNB(alpha=0.01, binarize=0.0, fit_prior=True, class_prior=None)
BRNBReg.fit(x_train,y_train)

#CrossValidation
cv_BRNBReg_score=cross_val_score(BRNBReg,x_train,y_train,cv=9)
cv_BRNBReg_pred=cross_val_predict(BRNBReg,x_train,y_train,cv=9)

print("R2-Socre-Mean: ",cv_BRNBReg_score.mean())
print("Score: ",r2_score(y_train,cv_BRNBReg_pred))

## Prediction
BRNBReg_pred=BRNBReg.predict(x_test)
print("Score: ",r2_score(y_test,BRNBReg_pred))

My Score with CV is:
R2-Socre-Mean:  0.908198797087686
Score:  0.8673184920974637

My problem:
My Score with test-data is:
Score:  0.6244917483855538

the difference is huge! But I don't know why?

Is my model overfit\underfit?
Shall i tune my hyperparams?