texts: Can features negatively correlated with the target be used?

texts:
In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target?
I don't think negative correlation means the predictor does not provide any information about the target.
Some feature selection methods (like Filter method) are based on using only those predictors that have high correlation to the target variable, and dropping those with low correlation.
My question is - shouldn't negative correlated features also be considered? I think the problem of feature selection should be whether a feature is "simply correlate " with the target or not, rather than whether it is a Positive/Negative correlation. Am I right? Can someone please clear my confusion?


texts: Should estimated probabilities from multi class classification sum to 1

texts:
I am using a neural network with sigmoid activation function $h(z) = 1 / {(1+e^{-z})} $ in order to classify image data into 6 categories.  When running the trained neural network over new image data, I noticed that the sums of the estimated probability from the hypothesis output for all 6 classes do not always sum to 1.  For example given an input image, the hypothesis output for each class might be:

  Class 1  --- 0.10

  Class 2  --- 0.11

  Class 3  --- 0.12

  Class 4  --- 0.13

  Class 5  --- 0.14

  Class 6  --- 0.15

I interpret this image as having a $13%$ probability of being classified into class 6.  However, the sum across all classes is < 1.

My intuition says the probability of each class should sum to 1 but again, I am very new to the machine learning world.

Could there be a bug in my code or is this a 'normal' output?


texts:
What is spectral clustering?
I have little background in statistics. I have tried to search for notes online but they assume quite a lot of knowledge.

Would be good if you are able to find some notes online which teach the basics and the math foundations for spectral clustering

I have found notes like this one which require a lot of background knowledge which are not suitable for me


texts: How to transform many columns of TRUE/FALSE/NA to just TRUE/FALSE?

texts:
I have a dataframe that consists of a few columns of text, and then a bunch of columns that are TRUE/FALSE or NA (they were TRUE/FALSE but I left-joined them with merge and that added NAs).

Eg:

issue # | title | body  | x     | y     | z    | lbl1 | lbl2  | lbl3  | lbl4 | lbl5
1       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE
2       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE
3       | blah  | blah  | blah  | blah  | blah | NA   | NA    | NA    | NA   | NA
4       | blah  | blah  | blah  | blah  | blah | NA   | NA    | NA    | NA   | NA
5       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE

I know how many columns need to not converted (and also their names), though I don't know how many label columns there are (or their names - they don't share any prefix).

I tried doing:

data[,-7] <- as.logical(isTRUE(data[,-7]))

Since this seemed to work with -1 for the same elsewhere, however my first columns all ended up as TRUE/FALSE too.

I also tried:

data[8:ncol(data)] <- sapply(data[8:ncol(data)], isTRUE)

But that resulted in everything being FALSE!

I also tried:

data[data==NA] <- FALSE

But that didn't seem to do anything (still has NAs).

I'm completely new to ML and R so please bear that in mind when answering. I don't know hardly any of the functions (or even completely understand all the syntax for selecting/replacing subsets of the dataframe as I'm trying to do here!).


texts: Hodrick–Prescott filter on train/test split

texts:
I have a multivariate time series $X$ which has been time-based split to $X_{train}$ and $X_{test}$. If I wanted to do standard scaling without data leak, it is possible to learn the scaling parameters on $X_{train}$ and apply the same to $X_{test}$
X_train_scaled = StandardScaler.fit_transform(X_train)
X_test_scaled = StandardScaler.transform(X_test)

Similarly, is it possible to apply the  Hodrick-Prescott filter without creating data leakage in regards to $X_{train}$ and $X_{test}$.? I am asking about both the mathematical and technical feasibility.
I assume that a working solution could exploit the current statmodels implementation of HP filter,  but  unfortunately there is no API resembling fit_transform / transform.


texts: Looking for feedback on a semi-supervised learning approach for multi-class classification

texts:
Problem: Currently only have 1200 labeled (3-classes) customers with an entire customer base of 4.7M. Just leveraging the 1200 to train the model isn’t generating sufficient results so I’m now looking to semi-supervised techniques to generate more data.
Here is how I’m planning to add more labels:

Leverage simple, business-driven filters to come up with more data

Take the average of the features in each class. Compare those averages to the rest of the universe using cosine similarity. Only include strong matches (>0.9) in the labeled set.

After these steps, I would then train my model and classify the remaining users.

Let me know what you think!


texts:
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to use UI (easily scalable, pannable, easily zoomable/fit to screen), but cannot handle the large sets of data I'm collecting. Does anyone know of any alternatives?

I have MATLAB, but don't have enough licenses to simultaneously run this and do development at the same time. I know that LabVIEW would be a great option, but it is currently cost-prohibitive.

Thanks in advance!


texts: Can we use weka for multivariate data analysis? When we have more than one variable as the dependent variable... ( without using factor analysis to reduce the number of variables associated with the dependent variable). Thank you


texts: how to find probability of one or more events to happen from an incomplete data set

texts:
I have a dataset that gives information of a population. For instance, I know the fraction of people that are males (M) and that are within a certain age range (A), P(M & A), and then I know the fraction of males that live in a certain area (L), P(M & L).

What I'm interested in computing is P(M & A & L), which is the fraction of people that are males, are within a certain age range and live in a certain area.

Using Baye's formula I can say that

P(M & A & L) = P(M & A | L) P(L)

But my dataset only gives P(L) and not P(M & A | L). However, if I assumed that M & A and L are independent I have

P(M & A | L) = P(M & A) P(L)

How large is the error on P(M & A | L) if I make this assumption. Do you know of any other method I could use to estimate P(M & A | L) without assuming independence?


texts:
I am trying to understand some simple neural net case using theano.
The deeplearning.net site gives the following simple code for implementing a logistic regression application to a simple case:

import numpy
import theano
import theano.tensor as T
rng = numpy.random

N = 400
feats = 784
D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=2))
training_steps = 10000

# Declare Theano symbolic variables
x = T.matrix("x")
y = T.vector("y")
w = theano.shared(rng.randn(feats), name="w")
b = theano.shared(0., name="b")
print("Initial model:")
print(w.get_value())
print(b.get_value())

# Construct Theano expression graph
p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b))   # Probability that target = 1
prediction = p_1 > 0.5                    # The prediction thresholded
xent = -y * T.log(p_1) - (1-y) * T.log(1-p_1) # Cross-entropy loss function
cost = xent.mean() + 0.01 * (w ** 2).sum()# The cost to minimize
gw, gb = T.grad(cost, [w, b])             # Compute the gradient of the cost
                                          # (we shall return to this in a
                                          # following section of this tutorial)

# Compile
train = theano.function(
          inputs=[x,y],
          outputs=[prediction, xent],
          updates=((w, w - 0.1 * gw), (b, b - 0.1 * gb)))
predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    pred, err = train(D[0], D[1])

print("Final model:")
print(w.get_value())
print(b.get_value())
print("target values for D:")
print(D[1])
print("prediction on D:")
print(predict(D[0]))

I understand most of it, p_1 is the logistic regression function, the prediction is whether the value will be in the 0 class or 1 class, xent is the loss function, i.e. how far from correct is our prediction. I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss?
What is the cost function representing here?
Also, why is the bias initially set to 0 and not a random number like the weights?


texts: How to reconstruct a scikit-learn predictor for Gradient Boosting Regressor?

texts:
I would like to train my datasets in scikit-learn but export the final Gradient Boosting Regressor elsewhere so that I can make predictions directly on another platform.
I am aware that we can obtain the individual decision trees used by the regressor by accessing regressor.estimators[].tree_. What I would like to know is how to fit these decision trees together to make the final regression predictor.


texts: How to train a model to predict if 2 samples refer to the same thing?

texts:
I have 2 ddbb with around 60.000 samples each. Both have the same features (same column names) that represent particular things with text or categories (turned into numbers). Each sample in a ddbb is assumed to refer to a different particular thing. But there are some objects that are represented in both ddbb, yet with somewhat different values in the same-name column (like different open descriptions, or classified as another category).
The aim is to train a ML model that recognizes when the two “descriptions” refer to the same thing.
We have manually recognized a few thousand "duplicated" cases that we have labelled correspondingly. This label is the target variable to learn. Yet, the supervised classification examples I've seen, work with a single data frame and/or predict something about a single row of features.
How does it work when we are not trying to predict what a sample represents, but the relation between 2 samples (specifically now if they refer to the same object or not)?
I don't even know how to feed 2 dataframes to sk-learn or even auto-sklearn (or similars) so that it can handle the task of recognizing if a sample from one represents the same thing than another sample in the other one... Maybe I should concat them so that there's just 1 df, but then it would have to compare all with all the rests... Or doesn't make it a lot of difference?
Any idea or hint about how to proceed here or how to frame the problem better?


texts: NN with 1 hidden layer: Number of half planes visualised

texts:
Using the tensorflow playground I get the result below for the

circle shaped sample data
1 hidden layer with relu
3 hidden units

From the 3 hidden units and relu 3 splitting lines are generated. From these I expect ${3\choose 0} + {3\choose 1} + {3\choose 2} = 7$ segments, of which the finite segment should be a triangle. However, the plot (below) on the right shows a hexagon.

I then manually extracted the weights that the tensorflow app optimized and generate my own plots below in R. Note that the 3 half planes generated by the straight lines (of each hidden layer) look the same as in the screenshot of the web application. However the final output (top left of the plot below) is a triangle not a hexagon. See the code below.
What am I a doing wrong here?

library(tidyverse)
library(gridExtra )
#Weights as taken from the optimal solution in the tensorflow playground
W <- data.frame(
    W11 = c(1.2, 0.31,-1.4),
    W12 = c(1,-1.5,0.56),
    b = c(-0.51,-0.68,-0.51),
    W2 = c(-1.6,-1.7,-1.6 )
)

#Grid
G <- expand.grid(
    x = seq(-6,6,0.05),
    y = seq(-6,6,0.05)
) %>%
mutate(
    Node1 = NA,
    Node2 = NA,
    Node3 = NA,
    output = NA
)

#fill the Grid
for (i in 1:nrow(G))
{
    for (j in 1:nrow(W))
    {
        G[i,paste0("Node",j)] <- max(G$x[i] * W[j,1] +  G$y[i] * W[j,2] + W[j,3],0)
    }
    G$output[i] <- G$Node1[i] * W$W2[1] + G$Node2[i] * W$W2[2] + G$Node3[i] * W$W2[3]
}

p1 <- G %>% mutate( Node1 = ifelse( Node1<=0,-1,1)) %>% ggplot(aes(x = x, y = y, color =  Node1) ) + geom_point()
p2 <- G %>% mutate( Node2= ifelse( Node2<=0,-1,1)) %>%     ggplot(aes(x = x, y = y, color =  Node2) ) + geom_point()
p3 <- G %>% mutate( Node3= ifelse( Node3<=0,-1,1)) %>%  ggplot(aes(x = x, y = y, color =  Node3) ) + geom_point()
p4 <- G %>% mutate( output = ifelse( output<0,-1,1)) %>%     ggplot(aes(x = x, y = y, color =  output) ) + geom_point()
grid.arrange(p4, p1, p2,p3, nrow = 2)


texts: Detect the presence/absense of simple components from a circuit board using object detection

texts:
In this case I have a challenge of inspecting a recently mounted product, using computer vision, to detect the absence of any component that I have to check.

For this task, I tried the concept of detection of regions of interest, which in this case are the components I want to check. I moved to an approach using tensorflow's object detection API, which I would train a model to detect the components I wanted to check using annotated images. In this case, all images contained just the presence of the component.

The results were very good using validation images, but unfortunately produces a lot of false positives (cases that the component is not present but it still writes a box on its area), sometimes with 99%+ confidence (so discarding my chance to filter these cases) and, on other side, there's cases that it can't find at all a, for an human view, very clear component.

With these results, I tried to train the model with both positive and negative cases (presence/absence), hoping that it would learn the difference between them.
Again, the false positives appeared, this time with a smaller frequency, but considering that my validation set is very small (about 10 images), in a global situation this error frequency can be very high, thus discarding the viability to use the model in production.

After several iterations of experiments, I'm starting to notice that the model is very unstable on its results to use in production. Is this approach the best way to solve this type of problem?

Unfortunately I don't have many examples to show the problem I'm facing at the moment, but just to illustrate my problem, here's two results of the approaches I listed:

I already tried using all models supported by the api, which includes several types of faster_rcnn and ssd. The one I stick with, with the best trade of training time and results, is the faster_rcnn_resnet101 with COCO training as the pre-training.


texts: What are some standard ways of computing the distance between individual search queries?

texts:
I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.).  I made this a separate question because search queries are considerably smaller than documents and are considerably noisier.  I hence don't know (and doubt) if the same distance metrics would be used here.

Either vanilla lexical distance metrics or state-of-the-art semantic distance metrics are preferred, with stronger preference for the latter.


texts: At what stage are ROC curves used when building machine learning model?

texts:
When developing a machine learning model, at what stage are ROC curve with AUC used?
Typically I have three data sets
train - validation - final test
I do K-Fold cross validation using the combined train + validation set
During that phase we can calculate the metrics including true positives, false positives as well as other metrics and average them to create a plot like the ROC curve.  Similar to this example from scikit-learn
However we can also get the metrics at the end by training the final model using all the data from train + validation and testing on the test set This can also give us all the metrics, classification report and ROC curve etc.
My question is, do people generally do the ROC curves twice, once during cross validation and then a second time for the final testing?  OR is it something that is used only during validation phase / hyper parameter tuning when selecting the algorithm?


texts: I need help on how I would go about solving this problem, my apologies for the vague title

texts:
I'm a beginner in the Data Science field and I'm trying to find a way to solve this problem, but I'm not sure where to start.
Here's the situation:
I have a battery that stores and discharges energy into my house from the power grid. Energy is produced by different generation techniques throughout the day and so I have data on the emission impact of using electricity at different hours. I also have data on the energy I've used in the past year. My goal is to charge and discharge this battery in the most effective way for reducing the emission impact of my home.
If anyone can point me in the right direction for what I should know and use to solve this problem I would be greatly appreciative!


texts:
I've been experimenting with Random Forests on Python after trying Naive Bayes which gave me lower accuracy than I expected, 62%. My csv file has around 14,000 records, I use 80% for the training set and 20% for testing set. I tried with different parameters like 100 trees, 500 and 1000, -1 for n_jobs and so on but during all those tests the accuracy never changed too much, it was always around 74% or 75%, almost 76% at times. I checked that the online tutorial which teaches how to implement got the accuracy of 73%, as my tests got higher than that, I wasn't worried, thought it's normal for this kind of algorithm.

However, my surprise came during my latest test, the only difference is I saved the model and vectorizer to file so I don't have to train the model each time. The new script loaded those files, applied the loaded (trained) model to the testing set and wow, I got the highest accuracy I've seen so far, 93.989%, almost 94%.

Anyone got a similar experience? Is this big increase related to having saved the model to a file and loaded it? The rest of the code is all the same. Or was I just too lucky and if I try more times I will go back to the usual around 75% accuracy?


texts: Any "rules of thumb" on number of features versus number of instances? (small data sets)

texts:
I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By using sparse methods (LASSO, elastic net) we can remove several features to reduce the model.

My question is (theoretically): before we use metrics to assess the model selection are there any empirical observations which relate the optimal number of features to the number of observations?

For example: for a binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?


texts: Implementing a custom hard sigmoid function

texts:
I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values:

0 if x < 0
1 if x > 1
x if 0 <= x <= 1

How do I implement it with a tensorflow backend Keras?


texts: Implication of Central Limit theorem and its application in machine Learning

texts:
According to central limit theorem, irrespective of the distribution of the original/population dataset, the sampling distribution will follow normal distribution.
My Question is if the means of n samples of the population data follow normal distribution, what is this helping us in ? I tried to find examples in the context of Ml but found no good example. It would be of great help if you could help me understand the significance of CLT ( basically application of knowing that sampling distribution will have bell curve) in the context of ML with an example.


texts: How to incorporate an attribute that only exists in some observations?

texts:
In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition is that the class is related to the number of days since that event occurred. I'd like to somehow include the number of days since the event occurred in my model ("days_since_event").

Example python data:

import pandas as pd
df = pd.DataFrame({'event_date':
                   pd.Series(['2019-02-25','','2019-01-31','','2019-03-03']),
                   'event_occurred': pd.Series([1,0,1,0,1]),
                   'days_since_event': pd.Series([42, '', 67, '', 36]),
                   'class': pd.Series([1,2,2,1,1])})

    event_date  event_occurred  days_since_event  class
0   2019-02-25               1                42      1
1                            0                        2
2   2019-01-31               1                67      2
3                            0                        1
4   2019-03-03               1                36      1

Is this a standard missing data problem or is there a way to better represent this data in a model-friendly format? Is this a situation where I can fill the missing observations with a global value and trust that the model will learn to ignore that value if "event_occurred" is 0?


texts: Do 3D bar charts have advantages over 2D bar charts?

texts: I vaguely remember that there was a study / blog post which made a strong point against 3D bar charts. Do you have a source at hand which compares the two - 2D bar charts and 3D bar charts?


texts: Backprop with relu and softmax in matlab

texts:
I am trying to implement a network for a classification task and I am kinda struggling with backpropagation.
The network should classify MNIST dataset (0 - 9).
As a training set, I have 4500 images of size 20x20px, which are serialized in a matrix X = 4500x400.
The network has 3 layers: 400 - 25 - 10 neurons
According to the architecture, the weights are randomly inicialized matrices w1 = 25x401 and w2 = 10x26 (both include a column with bias).
L1 hidden layer uses the relu as an activation function and L2 output layer uses the softmax for multiclass classification.
Forward propagation is calculated as
function [y, A1, A2] = forward(w1, w2, X)
    y = zeros(size(X, 1), 1);

    X = [ones(size(X, 1), 1), X]; % add bias column for L1
    A1 = relu(X*w1'); % compute L1
    A1 = [ones(size(A1, 1), 1), A1]; % add bias column for L2
    A2 = softmax(A1*w2'); compute L2

    [~, y] = max(A2, [], 2); % get column numbers with the highest values
    y = y - 1; % numbers in MNIST are in range 0 - 9, but matlab indexes columns from 1, so I have to subtract the 1 to get 0 - 9 range
end

The cost function is the cross-entropy, defined as
% y_score is the output from softmax
function cost = eval_cost(y_score, y, lambda)
    y_encoded = double(y == 1:max(y)); % one-hot encoding

    m = size(y, 1);
    cost = -sum(sum(y_encoded.*log(y_score)))./m;
end

Up to this point everything looks to be ok, but I make a mistake in the backprop implementation.
function [w1, w2] = backward(w1, w2, X, y, alpha, lambda)
    y_encoded = y == 1:max(y);
    A0 = [ones(size(X, 1), 1), X]; % add bias column to the input data

    [~, A1, A2] = forward(w1, w2, X);

    dz2 = -y_encoded - A2; % this dJ/dZ2 gradient value for the output layer, when there is the softmax and cross-entropy
    dw2 = dz2'*A1; % gradient dJ/dw2

    % gradient for the relu
    dz1 = zeros(size(A1)); % dw2 / drelu
    dz1(A1 > 0) = 1;
    dz1 = dz1(:,2:end); % remove bias for L2
    % according to my calculations, there should be dz1 = dz1(:,2:end).*(dz2), but the size of dz2 is 4500x10 and the size of dz1 is 4500x25, so I cannot do it.

    dw1 = dz1'*A0; % gradient dJ/dw1

    % update weights
    w1 = w1*(1 - alpha) - alpha*dw1;
    w2 = w2*(1 - alpha) - alpha*dw2;
end

The problem is, that I have no clue how to propagate the error through the relu function.
Could I ask anyone for a hint? :)


texts: Normalizing impact of outnumbered positive reviews to extract sentiment of each term

texts:
I'm trying to extract sentiment of Italian words using reviews that users wrote in Italian amazon. After doing some cleaning (remove punctuation, stop-words, etc.) I used this method to get the sentiment:

s = (tf_1/tf)*(-2) + (tf_2/tf)*(-1) + (tf_4/tf)*(1) + (tf_5/tf)*(2)

where tf is term frequency in whole corpus of reviews, tf_1 is the term frequency in one-star reviews and the like. But the problem is that, overall, majority of reviews are 4 and 5 stars (more than 80%). So a lot of neutral words end up with positive sentiment. How can I normalize this effect? I tried the following method, but result get worse:

s = ((tf_1/n1)*(-2) + (tf_2/n2)*(-1) + (tf_4/n4)*(1) + (tf_5/n5)*(2)) * (n/tf)

where n is number of total reviews, n1 is number of reviews with one star and the like.


texts: Conventions for dimensions of input and weight matrices in neural networks?

texts:
Im currently learning neural networks and I see conflicting decsriptions of the dimensions of weight and input matrices on the internet. I just wanted to know if there is some convention which more people use than the other.
I currently define my input matrix X with the dimensions of:
(m x n)
Where m is the number of samples and n is the number of features.
And I define my weight matrices with the dimensions:
(a x b)
Where a is the number of neurons in the layer and b is the number of neurons in the last layer.
Is that conventional or should I change something?


texts:
I keep on hearing this term "shot" used in machine learning.
Is a "shot" well-defined?
From what I can tell, "shot" is a synonym for "example". Most machine learning systems seem to be "multi-shot" meaning you have a huge dataset that has many different examples of different categories. However, for a system to have "one-shot" capabilities means that it is able to predict the category of something given exactly one example. Similarly, "few-shot" applications seem to only need a few examples in order to perform some function with the input. And "zero-shot" learning seems to be making predictions without any examples during training.
Is a shot just an example?
Given the evidence above, it seems like this is the case, but it also seems like it's a little bit more nuanced, something like a shot is a post-training example when 0 examples were given during training. But I'm not sure if this is right, thus the question.


texts: Error in keras in getting class labels in imagedata test generator for creating confusion Matix

texts:
I want to create confusion matrix for an Image segmentation.  My image data consists of MRI Image of Left Atria, Ground truth or mask image of binary segmentation of Left Atria.
I have created the following code for test and train data generators
SEED = 999
BATCH_SIZE_TRAIN = 16
BATCH_SIZE_TEST = 16

IMAGE_HEIGHT = 200
IMAGE_WIDTH = 200
IMG_SIZE = (IMAGE_HEIGHT, IMAGE_WIDTH)

NUM_TRAIN = 1320
NUM_TEST = 440

NUM_OF_EPOCHS = 75

data_dir_train = 'data/Train/'
data_dir_train_image = os.path.join(data_dir_train, 'img')
# The images should be stored under: "data/Train/img/img"
data_dir_train_mask = os.path.join(data_dir_train, 'mask')
# The images should be stored under: "data/Train/mask/img"

data_dir_test = 'data/Test/'
data_dir_test_image = os.path.join(data_dir_test, 'img')
# The images should be stored under: "data/Test/img/img"
data_dir_test_mask = os.path.join(data_dir_test, 'mask')
# The images should be stored under: "data/Test/mask/img"

def create_segmentation_generator_train(img_path, msk_path, BATCH_SIZE):
    data_gen_args = dict(rescale=1./255,
                         featurewise_center=False,
                         featurewise_std_normalization=False,
                         rotation_range=0,
                         width_shift_range=0.0,
                         height_shift_range=0.0,
                         zoom_range=0.0
                         )

    datagen = ImageDataGenerator(**data_gen_args)
    datagen1 = ImageDataGenerator(rescale=1./255)

    img_generator = datagen.flow_from_directory(img_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED,shuffle=False)
    msk_generator = datagen1.flow_from_directory(msk_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED,shuffle=False)
    return zip(img_generator, msk_generator)

def create_segmentation_generator_test(img_path, msk_path, BATCH_SIZE):
    data_gen_args = dict(rescale=1./255,
                         featurewise_center=False,
                         featurewise_std_normalization=False,
                         rotation_range=0,
                         width_shift_range=0.0,
                         height_shift_range=0.0,
                         zoom_range=0.0
                        )

    datagen = ImageDataGenerator(**data_gen_args)
    datagen1 = ImageDataGenerator(rescale=1./255)

    img_generator = datagen.flow_from_directory(img_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED,shuffle=False)
    msk_generator = datagen1.flow_from_directory(msk_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED,shuffle=False)
    return zip(img_generator, msk_generator)

train_generator = create_segmentation_generator_train(data_dir_train_image,data_dir_train_mask,BATCH_SIZE_TRAIN)
test_generator  = create_segmentation_generator_test(data_dir_test_image,data_dir_test_mask, BATCH_SIZE_TEST)

Now when I try to get classes from test_generator
true_classes = test_generator.classes

I get following error as
AttributeError                            Traceback (most recent call last)
 in
----> 1 true_classes = test_generator.classes
AttributeError: 'zip' object has no attribute 'classes'
What am I doing wrong?  Is there a specific code modification for Zip object since I have created image and mask generators as zip objects?  Please help.


texts:
I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.

$$
\begin{array}
\\
p &= \frac{1}{1+e^{-x}}
\\
\hat{y} &= min(max(p, 10^{-7}, 1-10^{-7})
\\
FN &= y \times log(\hat{y})
\\
FP &= (1-y) \times log(1-\hat{y})
\\
Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP
\end{array}
$$
I have calculated the gradient and hessian for this loss function:
$$
\begin{array}
\\
\frac{dLoss}{dx} &= 4py + p - 5y
\\
\frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p)
\end{array}
$$

And my code:

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = 1.0 / (1.0 + np.exp(-y_hat))
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)

xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic',
      'max_depth': 6, 'tree_method': 'hist', 'seed': 42}

model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'),
            (d_val, 'vld')], obj=logistic_obj, feval=err_rate)

Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective directive, it apparently defaults to RMSE which makes matters worse.

[0] trn-error:0.065108  vld-error:0.056749  trn-error:0.782048  vld-error:0.755389
[1] trn-error:0.064876  vld-error:0.056645  trn-error:0.727871  vld-error:0.695685
[2] trn-error:0.064487  vld-error:0.05651   trn-error:0.699920  vld-error:0.662203
[3] trn-error:0.064573  vld-error:0.056553  trn-error:0.691798  vld-error:0.64864
[4] trn-error:0.064484  vld-error:0.056514  trn-error:0.698498  vld-error:0.649974
[5] trn-error:0.064483  vld-error:0.056514  trn-error:0.716450  vld-error:0.662659
[6] trn-error:0.064470  vld-error:0.056507  trn-error:0.742848  vld-error:0.683847
[7] trn-error:0.064466  vld-error:0.056506  trn-error:0.775665  vld-error:0.71153
[8] trn-error:0.064435  vld-error:0.056497  trn-error:0.813440  vld-error:0.744165
[9] trn-error:0.064164  vld-error:0.056393  trn-error:0.854973  vld-error:0.780628


texts: online detection of plateaus in time series

texts:
I need to detect plateaus in time series data online. The data I am working with represents the magnitude of acceleration of a tri-axis accelerometer. I want to find a reference time window that I can use for calibration purposes. Because of that, the system must not move and hence only gravity should influence the system.

How can I find such plateaus or is there even a more principled approach that I can take?


texts:
I want to make the following Python data-processing code more efficient by replacing for loops. Is there any way to vectorize code like this?

I have a DataFrame object df that looks somewhat like:

names number
bob 5
sara 10
bob 8
foo 12
moo 16

I want to subset the DataFrame to find out all the rows associated with each name, and then perform an operation on number. This is what I am doing now:

for myName in set(df['names']):
    nameSubset = df.loc[df['names']==myName]
    operation(nameSubset['number'], **args)

'''Basically,perform an operation on the `number` column of nameSubset.'''

Is there any way to make this code run faster? Theoretically, this could be made faster if, instead of running through each myName at a time, the computer could process several myNames at any given moment. I'm not sure how to vectorize/parallelize my code to make this happen though.


texts: Having Trouble Using Orange Data Mining in Linear Progression in Continuous Data

texts:
I'm having trouble with a simple task. I just imported an Excel sheet (CSV) into Orange.

It recognized the features as continuous, but it couldn't run Linear Progression or Discretized on it.

How can I debug/fix this?


texts: finding start and end time of each session id in r


texts: Need help to understand backpropagation for gated recurrent units (GRU)

texts:
I'm stuck regarding the implementation of backpropagation in a binary classification task using a GRU. I wanted to know if someone could tell me how to proceed. I was able to understand how BP works in a simple RNN because it is very similar to an MLP. However, I don't know how to do it in the context of a GRU.

Let's say we want to use a GRU RNN for our binary classification task, i.e. $y_{true} \in \{0, 1\}$.
We have the following Gated recurrent unit
\begin{equation*}
\begin{aligned}
\Gamma_{r}^{<t>} & = \sigma\left(\gamma_{r}^{<t>}\right) = \sigma\left(U_r \; a^{<t-1>} + W_r \; x^{<t>} + b_r\right) \\ \\
\Gamma_{u}^{<t>} & = \sigma\left(\gamma_{u}^{<t>}\right) = \sigma\left(U_u \; a^{<t-1>} + W_u \; x^{<t>} + b_u\right) \\ \\
C^{<t>} & = tanh\left(c^{<t>}\right) = tanh\left(U_c \left(\Gamma_{r}^{<t>} \odot a^{<t-1>} \right) + W_c \; x^{<t>} + b_c\right) \\ \\
a^{<t>} & = \left(1-\Gamma_{u}^{<t>}\right) \odot a^{<t-1>} + \Gamma_{u}^{<t>} \odot C^{<t>}
\end{aligned}
\end{equation*}

Only at the final time $T$, we compute
\begin{equation*}
\begin{split}
v^{<T>} & = W_{ya} a^{<T>}+b_{y}, \\ \\
y^{<T>} & = \sigma\left(v^{<T>}\right).
\end{split}
\end{equation*}
Hence, the loss function is only computed at the final time $T$ over a single training example as
$$
\mathcal{L} = - \left[y_{true} \times ln \left(y^{<T>}\right) + \left(1-y_{true}\right) \times ln \left(1-y^{<T>}\right)\right].
$$

The activation functions are
$$
\sigma(x) = \frac{1}{1+e^{-x}}
$$
$$
tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}
$$

The question is, how can I compute the derivatives w.r.t. all the parameters?


texts: Overfit model after introduce dropout=0.5

texts:
I am facing a problem where my validation loss stagnates after 20 epochs. The training loss keep reducing which makes my model overfit. I have tried dropout with a value of 0.5 but there is no change in validation loss. Besides that, I have done activity regularization on the dense layer and add number of hidden layer but validation loss remain unchanged. How do I reduce my validation loss? I am looking forward for opinion and suggestion.

Thanks.


texts: Is there a parallel to record linkage/entity resolution where ML can be applied to the data for schema matching?

texts:
In the context of collecting disparate data sets holding similar information, are their examples of algorithms being able to resolve attributes of records being similar (while their values are different) based on their relation to other attributes? (So fuzzy matching would not necessarily apply)

Simplistic example: 2 Datasets regarding vehicle test data (perhaps with headings in different languages)

Source A: Metric information: Car, Distance tested (km), Fuel Used (l), Consumption (km/l)
Source B: Imperial information: Car, Distance tested (miles), Fuel Used (gallons), Consumption (mpg)

Goal:

Mapping between A & B, which recognises that the consumption data in different units, is actually the same because they are simply "column 2 / column 3" ?
Could it identify that the distance columns were holding similar information with a 1.6x scaling factor applied etc?
In this hypothetical example, matching could apply on column 1 to identify cars for record linkage, however I'm curious to see if there are methods which can identify the relationships between attributes, and recognise where attributes are similarly linked in other sets.

The closest material I have found seems to be this paper however I'm under the impression that this is probably a well investigated problem where I am not aware of the correct avenue to search!

Thanks for your input!


texts: How can I deal with circular features like hours?

texts:
Assume I want to predict if I'm fit in the morning. One feature is the last time I was online. Now this feature is tricky: If I take the hour, then a classifier might have a difficult time with it because 23 is numerically closer to 20 than to 0, but actually the time 23 o'clock is closer to 0 o'clock.

Is there a transformation to make this more linear? Probably into multiple features? (Well, hopefully not 60 features if I do the same for minutes)


texts: how to do the imputation for categorical feature with a missing rate?

texts: I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?


texts: Open-source interactive dashboard in Python

texts:
I am trying to find a package to construct a dashboard with interactive graphs (including widgets such as sliders) in python (mainly IPython notebook). I know there is plotly but I would like a fully open-source solution without constraints (i.e. having a public repository as with plotly without the subscription fee).

I have looked at IPython Dashboard package but it is not compatible with python 3 (because of MySQL-python).

Has anyone had any luck with any other package?


texts: Is the search space of Hyperparameters Continuous or Discrete?

texts:
I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete?
My understanding of both those cases:
1. Continuous would make it 'easier' to look for hyperparameters combinations that are more fine-tuned to the problem.
2. Discrete would imply that a more brute-force (or random) approach would be helpful.

Subsidiary question: Am I mistaking on my understanding of either of those cases?

Thanks


texts: different outcome of feature importance and coefficient from same data

texts:
I built a regression model to predict profit based on client, sales person, product category, client industry and client region.

After trying several models with tuning hyperparameters, I found that Gradient Boosting Regressor and Ridge Regression were performing best (they got similar RMSE).

However, when I ran feature importance and coefficient of each model to analyze which features were important for the prediction of profit, it showed that GB Regressor showed client & salesperson as top features while ridge regression had client region and industry as top coefficient.

I realized that I've been thinking coefficient and feature importance as the almost same thing.
Can someone help me to figure out how should I interpret this result - which one is more meaningful when I want to find the relationship among columns and target?


texts: Can a machine learning model be used as some kind of compression?

texts:
I'm trying to understand how machine learning is working. I read a lot and now came into my mind that it could be missuses in a practical way. I also hope that this question is on topic here.
Please correct me if I have some wrong assumptions:

All models require sample data
The learning process is some kind of optimization of its functions aka neutrons
The learning process is iterative to find the best values of the parameters to maximize the expected output of the neurons
To make this process faster you do some kind of reduction of the data, so that in the end the neutrons do not need to look at every part of the input
You split up the training data in two buckets to find out of the result became better for the optimization (I would call this verification, not sure how this is really called)
If your verification data are bad or equal to the input data than the results can just be as good as your input data

If this is true or at least most parts I got this idea:
When I set the input data to the same as the output data, then the model is doing nothing, but for equal data I get always the same data right?
So when I train the model as in the table:

input
output

0
3

1
1

2
4

3
1

4
5

5
9

6
2

7
6

8
5

Then the model would return PI based on the correct count of digits (of cause for non learned indexes that would be random)
I absolutely know that this makes no sense at all, but I'm just curious if I understand it correctly that you could "compress" data in a ML-Model.


texts: How to segment old digitized newspapers into articles

texts:
I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of size, one year of issues is around 350 000 words long.

What I'm trying to achieve is to detect the different articles that form a newspaper issue. Knowing that an article can be two or thee lines long or very much longer, that there is no systematic typographic division and that there are a lot of OCR errors in each file.
I should also mention that I don't have access to others OCR data like document layout in XML or so.

I've tried the TexTiling algorithm (the nltk implementation) but the results were not really conclusive.

Before diving deeper by myself I was wondering if maybe some of you would have some hint about a task like this one : train a machine learning model, try others algorithms ?


texts: Data visualization of frequencies of state transitions (possibly in R?)

texts:
I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc.
I see such transitions for a number of independent data points, and I have the cumulative frequencies from all data.

For example, I have:
$$
  Period  A   B  C           \newline
  1       4   4  2
  2       1   2  7
  3       0   1  9
  4      10  0  0
  5       8   1  1
$$

I DO know the transitions from one state to another, for example from A->B, B->C so on and so forth. For example I know that from Period 1, (all A's went to C. Among the missing B's one went to A, and rest to C.)
I was thinking of what would be the best way to visually represent this time wise transitions from one state to another. I was thinking that there might be some better way than just having a transition matrix, maybe something that looks like a Markov Chain but which could accommodate all the 5 periods of transitions in a succinct way?
I myself work on a statistical software called STATA, which has limited graphical applications. IS there something on other software packages (R maybe?) which can help me in this?

Sorry for the hack representation of the data matrix.


texts:
I am trying to make my inline plots in jupyterlab interactive. So far, I have tried a suggestion as pointed out here, among others:

https://stackoverflow.com/questions/50149562/jupyterlab-interactive-plot

# %matplotlib notebook - does not work : Javascript Error: IPython is not defined
# %matplotlib widget - works, but plots are overwritten

The widget magic works in making the plots interactive, but unfortunately, my plots are overwritten. Subsequent cells render plots on top of the output of cell 1 as below:

plt.scatter(trainData['x'], trainData['y'], color='Red', s=0.5)
plt.show()

plt.hist(trainData['x'])
plt.xlabel("Values of X")
plt.ylabel("Frequency")
plt.show()

plt.hist(trainData['y'])
plt.xlabel("Values of Y")
plt.ylabel("Frequency")

This does not look right, have I missed an option? I am looking for an IPython magic that can help with having several inline plots (after each cell)?

I am using jupyterlab v 0.35.4, run from Anaconda navigator.


texts: Transformation of categorical variables (binary vs numerical)

texts:
When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have  a feature vector with values A, B and c. The first method will transom A,B and C to numeric values such 1,2 and 3 respectively, other researches use (1,0,0), (0,1,0) and (0,0,1).

What is the difference between the first method and the second one?

The only difference I can think of is, if you use binary values, the size of the training/testing data will increase linearly according to how many values you have, which may slow down the performance, while the first one will keep the size unchanged.

Does either of these methods will effect the accuracy of your machine learning model (or classifier)?


texts: Looking for a machine's breakdown dataset

texts:
I am searching for a dataset that contains a machine's (any mechanical machine) operational variables (e.g. vibration, temprature, etc. ) versus it's operational status (e.g. functioning, malfunction, breakdown).
I have failed to find such dataset with googling.
Is there such a dataset?


texts: Inverse_Transform and multistep data - how to scale and when

texts:
I tried to ask this question before, but I think it was difficult without revealing the full context.  Right now I'm tinkering with what I find online to informally learn about tensorflow before I formally learn it in the spring.
The model I have created takes normal open, high low, close, volume data and calculates a bunch of other features.  My intent is to use the prior 15 days of OHLCV+Technicals to predict the future 5 Close.
My issue is at the very end when I need to invert the model.predict.  Currently I am using the following workaround:
sc = MinMaxScaler()
test_set_tomorrow = dataset_train.iloc[-time_step:,:].values
scaled_test_set = sc.fit_transform(test_set_tomorrow)
scaled_test_set = test_set_tomorrow
X_test = []
for i in range (0,len(test_set_tomorrow)):
  X_test.append(scaled_test_set[i,:])
X_test = np.array(X_test)
X_test=np.reshape(X_test,(1,time_step,-1))
predicted_price = model.predict(X_test)
predicted_price_fix = np.zeros(shape=(len(predicted_price), X_test.shape[2]))
predicted_price_fix[:,0:5]=predicted_price[:,0:5]
print(predicted_price_fix)
predicted_price=sc.inverse_transform(predicted_price_fix)
print(predicted_price)

Which outputs the following:
[[-0.04685311 -0.04455814 -0.04243669 -0.04118608 -0.03964855  0.
   0.          0.          0.          0.          0.          0.
   0.          0.          0.          0.          0.          0.
   0.          0.          0.          0.          0.          0.
   0.        ]]
[[ 1.56266435e+00  1.59385461e+00  1.49811773e+00  1.56682045e+00
   8.50910050e+05  1.65916667e+00  2.13096538e+00  1.60200000e+00
   1.63600000e+00  1.81628246e+00  2.08335540e+00 -3.99787353e-01
   6.99000000e-02  1.37914286e-01 -3.75699907e-01  1.00000000e+00
   2.61479727e+08  0.00000000e+00  2.01494297e+07  1.58000000e+00
   1.90910833e+00  1.76660253e-01 -4.35308886e+02 -2.06430382e-16
  -4.73584558e-18]]

There are two things which I find out.  The first is the values quite a bit off.  The second, is the fifth value.  It appears to be wildly out of whack.


texts:
I am wondering, do we really need <unk> tokens?
Why do we limit our vocabulary?

Is it for speed? Accuracy?

If we disable all limitations, what do you predict happens?


texts: How to include the other variables at t=t to predict the target variable with time lags also in LSTM?

texts:
I am having a training data set for a time-series dataset like below:
import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]

So as you see for training I am inputting the network with past three timesteps of the all 4 variables and the current timestep of the three remaining variables which makes up to 15 variables.
I want to train this with an LSTM with functional API in Keras because I cannot use sequential API in my case due to different input size. So I tried the below:
import tensorflow
from tensorflow.keras.utils import plot_model
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
visible = Input(shape=(100,15))
hidden1 = LSTM(10)(visible)
hidden2 = Dense(10, activation='relu')(hidden1)
output = Dense(1, activation='sigmoid')(hidden2)
model = Model(inputs=visible, outputs=output)
model.compile(loss='mae', optimizer='adam')

history = model.fit(train_X, train_y, batch_size=64, epochs=2, validation_split=0.2)

But this throws the error like below:

ValueError: Input 0 of layer lstm_2 is incompatible with the layer:
expected ndim=3, found ndim=2. Full shape received: [None, 15]

Then I reshape the train and test like below:
train_X=train_X.reshape(train_X.shape[0],1,train_X.shape[1])
train_y = train_y.reshape(train_y.shape[0],1)

train_X shape is (100,15)
train_y shape is (100,1)

But this gave a very huge MAPE and RMSE.So I think my reshaping is wrong here.
Any help is appreciated.


texts: How can I find out what class each of the columns in the probabilities output correspond to using Keras for a multi-class classification problem?

texts:
I'm using transfer learning to build an image recognition model using a pre-trained VGG network in Keras and excluding the final fully-connected layer to get the output weights.  I'm then using these output weights to feed into my new model which has a few layers along with a new fully-connected layer of my own that I'm training. The fully-connected layer maps to the number of output classes that I'm trying to predict.

Everything is working fine.  However, when I run:

results = model.predict(img_tensor)

I get output probabilities corresponding to each class, similar to the below:

print(results)

[[0.1426621  0.6193871  0.23795079]
 [0.11187755 0.6208466  0.2672758 ]
 [0.10050113 0.3768951  0.52260375]
 [0.1338948  0.59470254 0.27140263]
 [0.06612041 0.69726    0.2366195 ]
 [0.12080433 0.495977   0.38321865]]

My question is: How can I find out what class each of the columns in the probabilities output correspond to?

Does Keras have anything built-in to identify which column of the output probabilities corresponds to which class?  I would be shocked if nothing is provided...

What have others done to create a work-around?


texts:
I want to perform image classification using Keras and a dataset made of 50 classes. At the moment, I have only 7 images per class and I need to perform data augmentation in order to train the model and obtain acceptable accuracy values.

I am using the ImageDataGenerator class from keras which is recommended for image augmentation on the fly (during training). Since the classification is performing badly, I was wondering if it would be necessary to perform offline augmentation, i.e, enlarge the dataset before the training, because I honestly think that 7 is far from being a reasonable number of images per class.

Is it a common practice to perform both types of augmentation (before and during the training)? I am planning to use some 3rd party software or tools like imgaug to enlarge the dataset first and save the augmented images to disk and only then perform real-time augmentation with ImageDataGenerator class.

In conclusion, the flow would be similar to this:

Image pre-processing and offline data augmentation => enlarge the original dataset
Training with real-time augmentation => Load the dataset and use ImageDataGenerator

What do you think?

Thank you.


texts: Random Forest - ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

texts:
I'm trying to apply the RandomForest method to a dataset and I get this error:

ValueError: Input contains NaN, infinity or a value too large for dtype ('float32')

Could someone tell me what I can modify in the function for the code to work:

def ranks_RF(x_train, y_train, features_train, RESULT_PATH='Results'):
    """Get ranks from Random Forest"""

    print("\nMétodo_Random_Forest")

    random_forest = RandomForestRegressor(n_estimators=10)
    np.nan_to_num(x_train)
    np.nan_to_num(y_train)
    random_forest.fit(x_train, y_train)

    # Get rank by doing two times a sort.
    imp_array = np.array(random_forest.feature_importances_)
    imp_order = imp_array.argsort()
    ranks = imp_order.argsort()

    # Plot Random Forest
    imp = pd.Series(random_forest.feature_importances_, index=x_train.columns)
    imp = imp.sort_values()

    imp.plot(kind="barh")
    plt.xlabel("Importance")
    plt.ylabel("Features")
    plt.title("Feature importance using Random Forest")
    # plt.show()
    plt.savefig(RESULT_PATH + '/ranks_RF.png', bbox_inches='tight')

    return ranks


texts: how to optimize the weights of a neural net when feeding it with multiple training samples?

texts:
My question is:

Let's say we have a 3 by 1 neural net similar to that one in the image (an input layer with 3 neurons and an output layer with one neuron and no hidden layer), I find no problem calculating the outputs of the forward and back propagation when feeding the neural net with one training sample (namely feature1, feature2, feature3 inputs) and I know exactly how my initial weights get optimized, the problem I find is when feeding the NN with multiple training inputs each time, here, I don't know exactly how the initial weights get optimized.

For example we have training inputs of 3 × 3 Matrix.

[[195, 90, 41],
[140, 50, 30],
[180, 85, 43]]

the first column is the height, 2nd: the weight, 3rd: shoe size, we feed the NN with the first row then the second and the third row.
We know that to calculate the new weights when feeding the NN with one training sample we rely on this formula:
New_weights = Initial_weights - learning_rate × (derivative of the loss function wrt the weights).

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what?
I'm a bit confused here.

I would be grateful if any of you could explain how the initial weights get modified when feeding the NN with multiple training inputs.


texts: should we include or exclude a variable in a logistic regression based on the description below?

texts: should we include or exclude a variable in a logit regr. model which will only obtain values if a certain event takes place otherwise will show N/A? this variable tells whether or not a product will be bought based on calls made by the company. the variable tells: number of days that passed by after the client was last contacted from a previous campaign. the values (-1) are for users who


texts: Does it make sense to parallelize machine learning algorithms as part of PhD research?

texts:
I'm developing machine learning algorithms to aid in the diagnosis and prognosis of various cancers for my PhD. My lab is an Nvidia teaching center (CUDA).

My supervisor thinks that I need to also optimize ML by parallelizing it in CUDA. However, as I see it, a model is trained once and there is no need to train again. Testing a model is also not time consuming. My interests are in ML, not Parallel Processing.

1) Should I spend a large chunk of my time parallelizing with CUDA?
2) Is CUDA still a viable framework for research?
3) In the world outside of research, will this make it easier to get a ML job?


texts:
I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know that higher values are better but I do not know about their weights. I am looking for an approach to find the best candidates (there exist some training data, the suitable candidate of the entity is available).
Could you please advise me which one should be used: classification, regression or otherwise?


texts: How to treat sparse categorical features in a Neural Network for multiclass classification with Tensorflow?

texts:
I am building a Neural Network for multiclass classification.

My dataset has 3 millions of observations. My features are 7 unordered categorical values. My problem is sparse as 4 features among the 7 can take more than 1500 different values. The label is a categorical variable that takes 2100 different values.

My model outputs a vector of 2100 probabilities through the softmax function.

I am using Tensorflow with Python and I wanted to use one-hot encoding to encode each of the features.

Do this method suffer from the sparsity of my data?

For instance, do the One-Hot encoding transform my inputs into a vector of more than 2000*4 = 8000 features? In that case, if I use a mini-batch gradient descent with a batchsize of 50, is the fact that I have a lot more features than samples (8000>>>50) a problem ("curse of dimensionality")?

Would you have a solution, and a way to implement it on Tensorflow with Python?


texts: I am trying to make logical sense about why DCG is metric is formulated as such but I am not able to understand what's the need of taking the log of (1 + rank) term. Can you help me undestand the significance of log here?


texts: Siamese network using VGG16 to verify the similarity images

texts:
I want to create a deep learning model which verify the similarity of the images. So, I will use Siamese network. My dataset is images dataset not CSV file.

How can I create a Siamese network using VGG16 in Keras?


texts:
Dataset

**Col1**                                 **Col2**      **Col3**
dog                                        Z             st02
dog,cat                                    Z             st02
dog,bat,cat                                Z             st02
bat,cat,elephant                           Y             st02
dog,bat,cat,elephant                       Y             st02
tiger                                      Z             st01
pigeon                                     Z             st01
pigeon,parrot                              Z             st01
dove,parrot                                Z             st01
pigeon,parrot                              Z             st01
pigeon,parrot,dove                         Z             st01
lion,leopard,cheetah                       Z             st01
tiger,lion,leopard,cheetah                 Z             st01
dog,tiger,cheetah                          Y             st01
dog,tiger,leopard,cheetah                  Y             st01
eagle,jaguar,Kangaroo,zebra                Z             st02
cheetah,eagle,jaguar,Kangaroo,zebra        Z             st02

The expected output is:

**Col1**                                 **Col2**       **Col3**
dog,bat,cat                                Z              st02
dog,bat,cat,elephant                       Y              st02
tiger,lion,leopard,cheetah                 Z              st01
dog,tiger,leopard,cheetah                  Y              st01
cheetah,eagle,jaguar,Kangaroo,zebra        Z              st02
pigeon,parrot,dove                         Z              st01

In order to extract the above rows as output, I tried tracing the patterns and using the below logic:

data = pd.read_excel("data.xlsx")
data['Col4'] = data['Col1'].str.count(',')
v1 = []
v2 = []
v1.append(0)
v2.append(0)
for i in range(0,data.shape[0]-1):
    x = data['Col2'][i]
    y = data['Col2'][i+1]
    t1 = data['Col3'][i]
    t2 = data['Col3'][i+1]
    g1 = (x == y) & (t1==t2)
    d1 = data['Col1'][i]
    d2 = data['Col1'][i+1]
    c1 = data['Col4'][i]
    c2 = data['Col4'][i+1]
    flag = 0
    if(all(x in d2 for x in d1)):
      flag = 1
    g2 = (flag == 1)&(c2>c1)
    v1.append(g1)
    v2.append(g2)
    data['new_cond1'] = v1
    data['new_cond2'] = v2
    data['Final_flag'] = (data['new_cond1']==True)&(data['new_cond2']==True)
    data_output = data[data['Final_flag']==True]

But I didn't end up getting the expected output, rather few additional rows are also present in output. Could someone please help me extracting the rows mentioned in expected output.

From the dataset, I am trying to extract 1) Rows which has maximum number of animals separated by commas (or consider birds wherever pigeon/parrot/dove is mentioned). 2) Need not be the case that there should be only one maximum number of animals per Col2 or Col3, there might be even more than one Example as in case of row no. 1 and row no. 5 with same value in Col 2 and Col 3. This is because category of animals is different in row no.1 and row no. 5. Hope it's clear.

Thanks in advance!


texts: Attention to multiple areas of same sentence

texts:
Lets consider some sentences below:
"Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too"
"Can company1 buy company2? What will be their total turnover then?"
"Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians."

In all above sentences you can see there are multiple questions or utternaces. Sometimes separated by and sometimes by question mark and sometimes by just a dot.
A rule based separation of these sentences fail in many cases.  I want to split these sentences in individual intents.

One of the approach I am guessing is by using attention mechanism on different parts of sentences. I cant use gensim etc sentence embeddings as I dont have clear sentence boundaries here.
Can someone suggests if attention approach will work? If yes, any similar code if they can point to, that would be helpful as I haven't coded this before.

If any other better approach can solve this problem then please suggest.


texts: It is common to say in ML feature selection that features that are irrelevant in isolation can be important in combination with other features. Is there a simple example (one or two features) to demonstrate this?


texts: It has been customary for the users of different communities to quote funny things about their fields. It may be fun to share your funny things about Machine Learning, Deep Learning, Data Science and the things that you face every day!


texts: Machine Learning - Choice of features for determining hypothesis

texts:
I am a newbie to machine learning and have the following elementary questions.

Given a labeled dataset with multiple features, is it for a ML algorithm determine on its own what features are to be considered for determining the hypothesis function? Or is it for a user who is training the algorithm to explicitly indicate what features are to be considered?
Is there a mapping between problem space versus the hypothesis function type? As an example based on past experience, can we confidently state that for a housing price prediction problem, linear equation is appropriate. While for determining diabetes based on a given dataset, a quadratic hypothesis equation has proven reasonable.

Or is it usual for a designer to specify a custom function and allow the ML algo to determine the coefficients of this custom function (e.g. square root of sum of squares of 9 of 15 features divided by e^n) by reducing the error using gradient descent?


texts: What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

texts:
LDA has two hyperparameters, tuning them changes the induced topics.

What does the alpha and beta hyperparameters contribute to LDA?

How does the topic change if one or the other hyperparameters increase or decrease?

Why are they hyperparamters and not just parameters?


texts: What Models should i try for this problem?

texts:
I need some advice for a problem i'm working on with automobile data. The vehicles provide a series of codes at every second which are bieng stored, though it can vary how many.

For example , at time step one the vehicle might send 3 codes:

(1 , 2) , (3 , 5)  , (7 , 9)

and there can be a varying number of time steps.(Before an Error occurs)

Now some of these codes have associated Error values. For example (1 , 2) might be a critical error denoting a problem with your tire. But the vast majority of them don't have this. There are about 15 of these critical errors while 5500 of other codes.

The codes are actually made up of two separate independent codes.

The task is to predict ahead of time if is its likely that any of these codes will appear.

What i tried :

I tried to do this with a single layered GRU RNN Network with embeddings for all the codes. I am predicting the probability of the next code in the sequence at every step. (considering each code to be a step) .

I tried tweaking the parameters like the number of neurons etc, but i'm not getting any results with this (Accuracy less than 5 percent after several epochs).

How can i improve this model ?
Is there any method by which i can consider all of these codes at every step ?
What other algorithms
outside of deep learning can i apply here ?

Thanks in advance


texts: Running a python script with heavy memory requirements

texts: I need to run a python script where I am loading several(~15) large modules including keras and load several (approx 20) LSTM models and large amount of data for computation. These script has several methods which are needed to build an environment class for a reinforcement learning problem. The process seems to be working very slowly on my 16GB RAM i7 CPU(haven't tried out multiprocessing module yet but might not work as most of the task is not parallelism friendly.) Can you provide me some alternative possibly going for a higher end hardware setup ideal for such heavy computations which can be done on a local machine. Since it's a problem I am facing in ML related work which other researchers over here might have faced, that's why I am adding the question in this forum. Please suggest other forums if possible where I can get a better suggestion


texts:
I am trying out an R exercise where I have to randomly split my data into train/test set 50 times. On each iteration, I use lm() to train polynomial models up to 6th degree, and then predict the models on test set. I then have to do the following:

Plot the test predictions of the polynomial models across all 50 iterations separately. (This I have done; see image below).
Create a plot (either in the same plot or separately) to highlight the best model's test error across all 50 iterations. The error lines of the other models can be shaded in a lighter colour.

Part (2) is the one that confuses me. I am not sure what it is asking for. I thought about highlighting, for each polynomial, the line which returned the lowest test MSE out of the 50 iterations. However, I am quite sure this is wrong as I don't think "test predictions" mean the same as "error lines".


texts: Which model to chose based on learning curve

texts:
I trained my model using different regression techniques, and I'm not sure which model to choose based on the learning curve.

1) Should I choose Lasso, since train and CV converge at the end
2) Should I  choose Gradient Boosting since train and CV converge in the middle, and achieves a lower RMSE score than Lasso


texts: Accuracy improvement for logistic regression model

texts:
I have achieved 68% accuracy with my logistic regression model. I want to increase the accuracy of the model. How can I apply stepwise regression in this code and how beneficial it would be for my model? What changes shall I make in my code to get more accuracy with my data set. I have attached my dataset below. Following is my code:

library(dplyr)
data1 <- read.csv("~/hj.csv", header=T)
train<- data1[1:116,]
VALUE<-as.numeric(rownames(train))
testset<- data1[1:116,]
mylogit <- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family ="binomial")
testset$predicted.value = predict(mylogit, newdata = testset, type="response")
for (i in 1: nrow(testset)){
  if(testset$predicted.value[i] <= 0.50)
    testset$outcome[i] <- 0
  else testset$outcome[i] <- 1
}
print(testset)
tab = table(testset$VALUE, testset$outcome) %>% as.matrix.data.frame()
accuracy = sum(diag(tab))/sum(tab)
print(accuracy)
print(tab)
table(testset$VALUE, testset$outcome)

Following is my dataset:
Link 1: http://www.filedropper.com/hj_2


texts: Classification and clustering of Time series data of temperature

texts:
I have a time series recorded data of temperature. This is what my data looks like:

The change in data represents specific event or a class which I would like to detect when new incoming data. Because I cannot label the data manually I cannot do supervised learning. So, I thought take this sample data and then do some clustering to automatically make 5 classes(clusters). But I am not sure how this will be applied to the new incoming data?
I am naive in time series or data analysis (so not aware of many terms). I can use the sklearn or statsmodels library in python, but cannot use the LSTM or any Neural Network techniques. I searched about the auto correlation and drew the ACF graph of my data which looks like this:

I am not aware what this graph is saying? And not sure what to do with the auto correlation in my scenario. One more thing I had read is the HMM which may be helpful, but again I have zero knowledge on how to apply it or which route to go for this simple problem.
This is what I would like to do:

The data is simply the values taken from sensor and I would like to find out how many types different changes are present in this data.

After that, I would like to figure out which kind of change the new incoming data fall into.


texts:
I need to parse around 1.6k REGEX expressions such as the pair I am writing below.

I have also around 7k documents (1/2 page long each in average) that need to be parsed according to the REGEX expressions.

Right now I am using

library(rebus)
library(stringr)

regex_exp <- rebus::or1("(?i-mx:\\b(?:actroid\\b))", "(?i-mx:\\b(?:robot\\*w\\b)))")

regex_exp <- BOUNDARY %R% regex_exp %R% BOUNDARY

stringr::str_extract_all("This is my text talking about technology, but also about the actroid", regex_exp)

to found matches, but it takes approx. 3.5 minutes per file, which is of course not scalable.

Is there a more efficient library/method to parse regex expression in R?
I am also naive about whether using reticulate to parse in Python and go back to R could be faster.


texts: Optimal projection for data visualization

texts:
I have 90k points in $\mathbb{R}^{32}$ (i.e., a 90k by 32 real matrix) which I want to visualize. I know I can cluster my points (k-means &c), but I want to select a few "interesting" 2-planes in $\mathbb{R}^{32}$, project the points there and scatter-plot them.

How do I select the interesting 2-planes to project to?


texts: How to use an encoder to do feature extraction

texts:
I'm newbie with all of data science.
I have a pre-trained U-Net network from which I get its encoder.
Now I have to use a picture to get its features.
With the whole U-Net I do this with fit method:
X_train, X_valid, y_train, y_valid = train_test_split(dataset[:, 0, :], dataset[:, 1, :], test_size=0.1, random_state=42)

results = u_net_model.fit(X_train, y_train, batch_size=8, epochs=50, validation_data=(X_valid, y_valid))

But... what do I have to do to use the encoder to extract the features from a picture?


texts: method and dataset for credit card fraud detection

texts:
I am trying to create a machine learning model to detect credit card fraud (In our definition, fraud means chargeback). I am kinda stuck now with the dataset that I have. I don't have information on the item, quantity, or shipping address. I have information on the credit card, amount to charge, ip address since we are a payment gateway.

Is it possible to do fraud detection using the this kind of dataset? The features I created out of existing dataset can't detect the chargeback very well.

I have tried:

velocity check analysis. ie. checking numbers of transaction past x hours, past x days. Checking number of unique Card used per IP or unique IP used per card.
anomaly detection. ie. checking out what transaction amount is considered normal for a merchant or how many transactions per day is considered normal.
oversampling/undersampling. We have highly imbalanced dataset and that's why I am doing oversampling.

Is there any other method I can try or is it even possible to do proper fraud detection with my current dataset?


texts: How Retail Analytics Project Can Be Done?

texts:
I have a project and I couldn't understand what I have to do because I am new to retail analytics.
They said "Our goal is to measure the effects of promotion on sales" and "Your goal is to model the effect of promotion on products and stores. In order to answer questions, you should divide products and stores into 3 clusters each. (High, Medium, Low)"
I have two datasets, let's say;
1 - data.csv -> Date, Store, Product, QuantityOfSales
2 - promotion.csv -> PromName, StartDate, EndDate
There are questions like that:

What are your criteria for separating High, Medium, and Low Stores?
Why?
Which items experienced the biggest sale increase during promotions?
Are there stores that have higher promotion reactions?
What is the biggest effect explaining sales change during promotions?
Is there any significant difference between promotion impacts of the
High versus Low items?

Also, they gave me the same dataset but different time intervals to measure how well my model has worked on this new data.
Actually, I am stuck because I don't know what exactly I have to do. Can anybody show me the path I should follow or show samples?


texts: CrossValidation using glmnet and very high values of Lambda?

texts:
I am trying to run crossvalidation (folds=10) using glmnet library on my dataset. My outcome of interest is BMI and predictors include a set of clinical variables. My final goal is to use elastic-net regression to select features and also predict BMI. For crossvalidation, I am using a range of alpha from 0 to 1 using a 0.1 increment. I am using min CVM to decide the values of lambda, with my current code I get extremely high estimates for lambda. Are higher values of lambda acceptable or I am failing to see here? Below is my code snippet.  I appreciate all your help and comments.

size <- floor(nrow(DataFile) * 0.7)
Train_rows <- sample(rownames(DataFile),size=size,replace=FALSE)
Train_Data <- DataFile[Train_rows,]
Train_bmi <- phenoFile[Train_rows,]$BMI

####### Cross Validation Alpha and Lambda #####

myAlpha <-  seq(0,1,by=0.1)

findAlpha_lambda <- function(iAlpha){
   Train_Data <- as.matrix(Train_Data)
   crossModel <- cv.glmnet(Train_Data,Train_bmi,alpha=iAlpha)
   myLambda <- crossModel$lambda.min
   myCVM <- min(crossModel$cvm)
   title <- paste(iAlpha,myLambda,sep="_")
   return(c(iAlpha,myLambda,myCVM))
}

myFrame <- as.data.frame(do.call(rbind,lapply(myAlpha,findAlpha_lambda)))
colnames(myFrame) <- c('Alpha','Lamda','CVM')
myFrame <- myFrame[order(myFrame$CVM),]
print(myFrame)

     Alpha      Lamda      CVM
1    0.0 50.9839208 54.25337
2    0.1  1.7901432 54.37151
3    0.2  3.1427680 54.75240
4    0.3  2.1949422 57.68935
5    0.4  1.8927376 61.68384
9    0.8  1.2510439 63.69622
6    0.5  1.0933677 64.68333
8    0.7  0.2441112 64.73192
7    0.6  2.2050751 65.01727
11   1.0  1.3860429 65.17181
10   0.9  0.5042962 65.70732


texts: LSTMs: what is $W_x$ & $U_z$ in $φ(W_x + U_z + b)$?

texts:
Reading On Multiplicative Integration with Recurrent Neural Networks

  Despite of their varying characteristics, most of them(RNNs) share a common
  computational building block, described by the following equation:
  $φ(W_x + U_z + b)$, where $x ∈ R_n$ and $z ∈ R_m$ are state vectors coming
  from different information sources, $W ∈ R_{d×n}$ and $U ∈ R_{d×m}$ are
  state-to-state transition matrices, and $b$ is a bias vector.

Don't get what meaning of $W_x$ and $U_z$. I know that $W$ is typicaly for weights... what does this equation mean?


texts: 100% accuracy on both train and test after feature engineering

texts:
The original dataset is of ~17K compound structures almost equally divided with labels indicating yes or no, after heavy use of mol2vec and rdkit I have created ~300 datapoints
Using the boosted trees method on the same shuffled train and test dataset gives 98% train accuracy and 89% test accuracy, but a simple neural network gives 100% train and test accuracy
I have checked the code again to ensure target leakage is not occurring, I have also coded it from the scratch twice to ensure I haven't made any mistake, yet I do not believe I should be getting 100% accuracy on both train and test
Does this mean that the model is actually accurate due to so many data points?


texts: What is the difference between a bounding box and ROI (Region of Interest)

texts:
I was reading about the Fast RCNN  for object detection. From what I understand, it uses pre-computed ROI's (using selective search) and uses these to predict the bounding box offsets and uses smooth L1 loss to refine these and get closer to the ground truth boxes.
The paper states the following about the ROI's
While training, R/N ROI's for each image (N=2,R=128) are taken where N are the images per mini batch. Among the ROI's chosen, around 25% of them are taken that have IOU more than 0.5 (foreground) and the remaining (called background) that have IOU between 0.1 and 0.5.

My question here is, are these ROI's similar to bounding boxes and if not, in what way are they different?
Any help would be appreciated. Thanks!


texts:
I am new to applying ML to time series data but I do have experience doing general supervised learning. I have time series that is multidimensional (so several variables over time) with one output variable. I tried following some online tutorials but I am confused on a few points that keep coming up.

Some tutorials stress checking whether the variables are stationary (Usually they do this using a Dickey-Fuller test).

Do I have to check every feature in my time series for this or just the output variable?
Can I keep in individual features that are not stationary?
Do I have to check if my dataset (features or output variable) are autocorrelated?
Do I have to exclude any that are overly autocorrelated?
Any other do's and don'ts for time series predictive analysis?

It seems like if I just lag my features with the output then I can readily apply supervised learning models like multiple linear regression or random forests if I cross-validate in a special fashion. Is that really all I have to do to make my time series a supervised learning task?


texts: ommiting a constant parameter from the learning process

texts:
imagine that your NN output two vectors Q and P. those two vectors you apply to the following expression:
y_hat = Q(a + Pb)

to finally gather the output y_hat
a and b  are constants and as we mentioned Q and P are the learnable parameters.
The question is, whether it makes sense to omit the "a" parameter and let the model learn to generalize without providing this information or if this is wrong and we got to stick to the modeling requirements.
Finally, for those who are curious about the specific context, I am training CNN that is applied to Point Distribution model PDM. which means I take the output of a CNN and apply it to the mentioned equation that translates the PDM model.
PS: by omitting the "a" parameter the training error did not shift considerably comparing to keeping it
thanks


texts:
I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the distribution of sentences.

Now what I want to do is to compare the text generated from the language model of two different corpora on the same topic.

Use case: I want to compare the headline that a right winning vs left winning news outlet would use for a given news content. (my training data would be a large set of headline+news content from both news outlet)


texts:
Before marking my question as duplicate, I would like to say that I have tried all the possible solutions mentioned in similar questions, but that doesn't seem to work.

I am currently working on blood cells classification problem where we basically have to classify blood images (4 classes). The dataset consists of 9957 images, nearly equal number of images of all the 4 classes. The accuracy always hovers around 25-27%  even after trying different optimizers and learning rates. I even tried training upto 100 epochs. Image augmentation doesn't help. Also, it is not that it is predicting same class for all images although for 1 particular batch of images, it predicts the same class. It again predicts some other class for all images in the next batch. So, I would just like to know, what am I possibly doing wrong? Is the dataset not sufficient, or the architecture should have more hidden layers, or am I not implementing optimizer or loss function correctly or is there any silly mistake I am overlooking in my code ?

My CNN architecture: (fs means filter_size, nf means number of filters, s is no. strides)

Input(80,80,1)->Conv(fs = 3, nf = 80, s = [1,1,1,1])
Activation(LeakyReLU)->Conv(fs = 3,nf=64,s=[1,1,1,1])
Activation(LReLU)->Pool(ps = [1,2,2,1],s=[1,2,2,1]
Conv(fs = 3,nf = 64,s=[1,1,1,1])->Activation(LReLU)
Dropout(prob = 0.75)->Flatten
FullyConnected(output_features = 128)->Dropout(prob = 0.5)
FullyConnected(output_features = 4)

loss_value = tf.reduce_mean(loss_fn)
optimizer = tf.train.AdamOptimizer()

loss_min_fn = optimizer.minimize(loss = loss_value)
check_prediction = tf.equal(tf.argmax(y,axis=1),y_pred)
model_accuracy = tf.reduce_mean(tf.cast(check_prediction, tf.float32)

sess.run(loss_min_fn, feed_dict = {x:X_train_batch, y:y_train_batch})
train_accuracy = train_accuracy + sess.run(model_accuracy, feed_dict={x : X_train_batch,y:y_train_batch})
train_loss = train_loss + sess.run(loss_value, feed_dict={x : X_train_batch,y:y_train_batch})

The images kind of look like this


texts:
Consider a binary classification problem with 0 labels denoting normal and 1 abnormal or rare. The number of instances with 0 classes are more in comparison to 1. In general,

1) Does 0 always refer to positive or a negative depending on what we define as a positive and negative? What if the labels are reversed?

2) Is there a particular order that the confusion matrix if displayed? If the confusion matrix is given as:

1   4
0   5

I got this confusion matrix in Matlab. How do I know that the first row is for class 0 or for class 1?


texts: How to use ensemble of models in FM or FFM?

texts:
I am using Factorization Machines ( libfm) and also the Field Aware Factorization Machines (libffm) for a kaggle competition.  I am currently using the single models of each respectively for prediction.
I came to know that we can use ensemble of models for both FM and FFM.
Can someone explain how  this is done ?


texts:
I've been taking the coursera machine learning course and The instructor said that if ( C = 1/lambda )  then the learning algorithm would reach the same optimal value of theta

does this mean that using C instead of lambda is just a convention when using SVM or is there an actual difference ?

P.S
Please dumb down the answer for me I'm kind of new to this :) Thanks


texts: When to consider a target Variable unbalanced?

texts:
i'm performing a binary classification task , and after cheking the target variable , i saw that i had 69% of 0's and 31% of 1's , so , my question is , do i have in this case a unbalanced target variable ?
 and can somebody tell me , if there is a threshold , or when to consider a Target Variable unbalanced ?

Thanks to everyone


texts: Can we take of benefit of using transfer learning while training a word2vec models?

texts:
I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated.
Thanks in advance :)


texts: Source of Arthur Samuel's definition of machine learning

texts:
Many people seem to agree that Arthur Samuel wrote or said in 1959 that machine learning is the "Field of study that gives computers the ability to learn without being explicitly programmed".

For example the quote is contained in this page, that one and in Andrew Ng's ML course. Several articles also contain this quote, and the reference is always the following article, which doesn't actually contain the quote.

Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3), 210-229.

Is there a reliable source? Or is this actually not a quote, but rather an interpretation of Samuel's article?


texts: Label Dependent Covariate in Logistic Regression

texts: Given a sequence of events, I am trying to predict a label for events and I hypothesize that the last time this event occurred with a label $i$ has a bearing on when the next time the label $i$ appears. Is it possible to include a covariate which depends on the label? For example my probability for predicting label $i$ would be $P(x=i|y^i)$ and that for predicting label $j$ would be $P(x=j|y^j)$. This would violate the assumption that $y$ are the independent variables in the model.


texts: NMT, What if we do not pass input for decoder?

texts:
For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output.

What if we do not pass input for decoder and consider it as a 'memory' model for translation.
Is it possible and what will happen?

It seems decoder could be removed and there only exist encoder.

Could I do translation task like text generation?
See:

https://github.com/salesforce/ctrl/blob/master/generation.py

https://einstein.ai/presentations/ctrl.pdf


texts:
This is a complete noob question, I'm new to Python, and I understand the basics of NMF, but when implementing it in sklearn it seems a bit convoluted and I wanted to ask whether anyone knows how to exploit the results.

I've imported data from a file, and created and shaped an array, and now have implemented some initial NMF code I took from the sklearn guide online.

nmf = NMF(init='nndsvd', max_iter=2000, random_state=0, shuffle=False, solver='cd',verbose=0)
W = nmf.fit_transform(myfile);
H = nmf.components_;

#X_new = np.array(?)
#W_new = model.transform(X_new)

But here's the bit I'm getting lost - and why I commented out the last two lines - where does X_new come from, am I meant to be imported another matrix for comparison - I thought the point of NMF was to identify key features of the matrix under analysis?

Any help greatly appreciated.


texts:
I tried to create a learner using the Orange canvas script from the Python script widget, but I have not been able to find an effective example.

What I would like to do is:

1) Create a learner based on scipy.interpolate.interpn (or other interp, this is to get an example)

2) Have the training dataset come from an Orange table (with target defined)

3) Have the output result be the same as existing learners, for example knn

I tried to gain inspiration from existing widgets like the knn one, but when I copied and pasted the code in the script, nothing happened.

Any ideas on how to proceed? Or, scipy examples to share?

Accessing to the scipy functions could be very helpful, for example to compare interpolation and regression output results.

Thanks.


texts: Changing categorical data to binary data is not reflected on the dataset

texts:
I am working through the Titanic competition.  This is my code so far:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv")

train['Sex'].replace(['female', 'male'], [0, 1])
train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])

# Fill missing values in Age feature with each sex’s median value of Age
train['Age'].fillna(train.groupby('Sex')['Age'].transform("median"), inplace=True)

linReg = LinearRegression()

data = train[['Pclass', 'Sex', 'Parch', 'Fare', 'Age']]

# implement train_test_split
x_train, x_test, y_train, y_test = train_test_split(data, train['Survived'], test_size=0.2, random_state=0)

# Training the machine learning algorithm
linReg.fit(x_train, y_train)

# Checking the accuracy score of the model
accuracy = linReg.score(x_test, y_test)
print(accuracy*100, '%')

This line previously looked like this: data = train[['Pclass', 'Parch', 'Fare', 'Age']], which ended up giving me an accuracy score of 19.5%.  I realized that I didn't include sex so I went ahead and did this:

data = train[['Pclass', 'Sex', 'Parch', 'Fare', 'Age']]

Then, I got the following error:

ValueError: could not convert string to float: 'female'

Here I realized that the changes that I've done to my train['Sex'] and train['Age'] did not reflect on the training and the testing of the model, which seems to be the reason why my model performed at 19.5%.  How do I come across this problem?


texts: Pretraining a neural network to teach it general information

texts:
Let's say that I want to train a neural network to recognize symptoms of severe dehydration in distance runners visually. Runners tend to finish races either overhydrated (hyponatremic) or dehydrated, but the two present with fairly similar symptoms, so knowing the difference could be quite useful.

I've got myself a small dataset of a few thousand faces of dehydrated runners, a few thousand of overhydrated runners, and several million faces of the general population. I don't expect to be able to properly train a network to recognize dehydration with just a few thousand faces of the dehydrated, but I'd like to see if I can pretrain the network using the several million faces so it learns to distinguish facial details, then train it on the actual overhydrated/dehydrated faces. But if I take some other problem, like distinguishing men from women, and pretrain it on that, I'd imagine my hydration training phase would output something that distinguishes something between hydration level and gender.

Is there some standard way to pretrain a model to learn general details (of faces for example) before training on the proper (far smaller) dataset?


texts: Does a precision score increasing with a higher number of folds mean the model will improve with more data?

texts:
I have been working on a pretty simple text classifying module (tfidf + Random Forest).
My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about putting the model trained on 70% of the data in production (rather than the model trained on the whole thing). Her rationale was that the model would be more "predictable" and that anything we would gain from adding the remaining 30% would be negligible.

Out of curiosity, I ran a few test using cross-validation. First with 3 folds, then incrementally up to 10. With every increase in fold, the model's precision increases (first quite strongly: there's a 3 point increase in precision from 3 folds to 5 folds, the gains then become more marginal with each fold but still add up to a 5 point boost when reaching 10 folds). The variance between the cross-validation scores is always very low.

Am I right to assume that this can be interpreted as meaning the model is very likely to benefit from being trained on the whole dataset rather than on just 70%?

I understand there may be a risk of overfitting on the training data but a) I don't really see how the risk would be significantly mitigated by training on 70% of the data only, b) these are quite formal, standardized communications and unseen data is unlikely to differ significantly from the training datasets.


texts: How to add words to english model word list in Julius Speech Recognition Engine?

texts:
I want to add some English words to model but how can I achieve this ?
https://github.com/julius-speech/julius


texts:
I just built a Convolutional Autoencoder to try to reconstruct a time series with shape (4000, 10, 30). This is the code, I used a batch size of 32, but I think it is overfitting since it performs well on the training set but starts not reconstruction well in the validation and test set. What are the steps I can do to improve it? How can I define the right number of filters for each layer?
kernel_size = 7
stride = 1
model = Sequential()
model.add(Conv1D(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Conv1D(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1D(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))

This is the optimizer:
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-04)
model.compile(optimizer=optimizer, loss="mse",
              metrics=[tf.metrics.MeanAbsoluteError(), 'accuracy'])

EDIT: Dropout version:
model = Sequential()
model.add(Conv1D(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Dropout(0.2))
model.add(Conv1D(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Dropout(0.2))
model.add(Conv1DTranspose(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))


texts: Semantic segmentation dataset inspection

texts:
Is there any tool that can calculate the statistics of semantic segmentation dataset (and possibly the similar types of datasets ...)?

So that it shows the number of classes, the total number of pixels for each class, and other similar statistics, ... and visualizes those statistics ...?


texts: How to explain calm period in energy consumption

texts:
I have a dataset of energy consumption that looks like this:

I got it from Kaggle.com and all it says is that it's energy consumption data from PJM (this is the PJMW_hourly that corresponds to the western zone, but they are all similar).

I studied and fitted models on the five weeks (April 18 to May 22, 2016) between the two blue lines on the picture.

I fitted a SARIMA (not that it matters for this question) and it obviously didn't predict that the series was going to blow up like it does.

So I was wondering if this kind of behavior can be explained by weather or the price of coal or anything like that?

Thanks


texts: Practical examples/tutorials of using One-Class Support Vector Machines

texts: I am a newbie in machine learning, and hope to solve an anomaly detection task  using One-Class Support Vector Machines (OCSVM). Despite the availability of several general introductions, definitions and academic papers on OCSVM, I do not find tutorials with practical examples except a few provided by scikit-learn. I'd appreciate some pointers to such resources especially with code examples/datasets as this will aid better understanding, than reading academic papers.