texts: How to design two different neural nets for actor and critic RL?

texts:
In order to have an actor critic RL model there are two things to be satisfied .

Value approximation function should converge to a local minimum

$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - f_w(s,a)]\frac{\partial f_w(s,a)}{\partial w} = 0$$

The following condition should be satisfied with the parameterization

$$\frac{\partial f_w(s,a)}{\partial w} = \frac{\partial \pi(s,a)}{\partial \theta} \frac{1}{ \pi(s,a)} $$

So specifically how can we design a model to meet the second condition?

Update

here I want to highlight the value function approximation in actor-critic methods . We need to optimize the critic also as we did for the Q learning but following the on policy which is taking the TD error according to the actor. Here I will put the final equation of actor critic.

Here simply we can take the critic neural net's output as the state value function or the f above. So how to satisfy the condition 2 ?


texts:
Can anyone recommend an alternative to the big 3 cloud computing alternatives? I know they're the best but I found them overly complicated because they cater to massive enterprises. The amount of set up required just to get an instance running is too much.
I am looking for a multi GPU cloud offering which offers RAPIDS pre-installed. I see that Blazing SQL will have an offering soon, does anyone know of anything else that I could use in the mean time?


texts:
library(ggplot2)
library(readr)
library(stringr)
library(neuralnet)

train <- read.delim("../input/train.tsv")

test <- read.delim("../input/test.tsv")

category <- str_split_fixed(train$category_name,"/",3)

category_test <- str_split_fixed(test$category_name,"/",3)

train$category_a <- category[,1]

test$category_a <- category_test[,1]

train$category_b <- category[,2]

test$category_b <- category_test[,2]

train$category_c <- category[,3]

test$category_c <- category_test[,3]

train = train[,!(names(train) %in% c("category_name"))]

train = train[,!(names(train) %in% c("item_description"))]

train = train[,!(names(train) %in% c("name"))]

test = test[,!(names(test) %in% c("category_name"))]
test = test[,!(names(test) %in% c("item_description"))]
test = test[,!(names(test) %in% c("name"))]

train$brand_name <- as.numeric(train$brand_name)
train$category_a <- as.numeric(factor(train$category_a))
train$category_b <- as.numeric(factor(train$category_b))
train$category_c <- as.numeric(factor(train$category_c))

test$brand_name <- as.numeric(test$brand_name)

test$category_a <- as.numeric(factor(test$category_a))

test$category_b <- as.numeric(factor(test$category_b))

test$category_c <- as.numeric(factor(test$category_c))

max = apply(train , 2 , max)

min = apply(train, 2 , min)

max_test = apply(test , 2 , max)

min_test = apply(test, 2 , min)

scaled = as.data.frame(scale(train, center = min, scale = max - min))

scaled_test = as.data.frame(scale(test, center = min_test, scale = max_test - min_test))

set.seed(2)
results = neuralnet(price ~ item_condition_id + shipping + category_a + category_b , scaled, hidden = 3,linear.output = T)
#results <- lm(item_condition_id + price ~ shipping + category_a + category_b , data=train)

pre <- compute(results,scaled_test)

solution <- data.frame(test_id = test$test_id, price = pre)

write.csv(solution, file = 'sample_submission.csv', row.names = F)


texts: Probability for label correctness in semi-supervised learning

texts: I am aware of the existence of semi-supervised learning approaches, such as the Ladder Network, where only a subset of the data is labeled. Are there any methods or papers which consider correctness probabilities for the labels of that training data subset? That is, some labels may be correct with 100% probability, while others may have only 70% or 45% probability of being correct. Any links to papers or work in this direction are highly appreciated.


texts: How the "def match(self, example)" method is automatically calling and how the example arguments is working here?

texts:
trainung_data = [
    ['Green', 3, 'Apple'],
    ['Yellow', 3, 'Apple'],
    ['Red', 1, 'Grape'],
    ['Red', 1, 'Grape'],
    ['Yellow', 3, 'Lemon'],
]
header = ["color", "diameter", "label"]
def is_numeric(value):
    return isinstance(value, int) or isinstance(value, float)
class Question:
    def __init__(self, column, value):
        self.column = column
        self.value = value

    def match(self, example):
        val = example[self.column]
        if is_numeric(val):
            return val>=self.value
        else:
             return val>=self.value
    def __repr__(self):
        condition = '=='
        if is_numeric(self.value):
            condition = '>='
        return "Is %s %s %s?" % (header[self.column], condition, str(self.value))

Now if i call the class like:

Question(0, 'Green')

It give me output like this

Is color == Green?


texts: Optimization function returns the same optimal parameters for two labels

texts:
I've recently enrolled in the Coursera machine learning, and am working my way through making my own classifier for the Iris dataset problem using matlab. I'm training a classifier for each species (the one-vs-all method). The code runs smoothly without any errors, but the results are not quite what I was expecting. Here's my matlab code:

clear, clc

% loading the data
data = csvread('data.csv');

% extracting the features
X = data(:, (1:end-3));
X = [ones(size(X, 1), 1) X];

% computing the number of training examples (m) and features (n)
[m, n] = size(X);

% extracting the labels
y_setosa = data(:, end-2);
y_versicolor = data(:, end-1);
y_virginica = data(:, end);

% initializing the options for optimization algorithm
options = optimset('GradObj', 'on', 'MaxIter', 10);

% creating the initial theta vector
iniTheta = zeros(n, 1);

% tarining a classifier for each label
[theta_setosa, min_setosa] = fminunc(@(theta) costFunction(theta, m, X, y_setosa), iniTheta, options);
[theta_versicolor, min_versicolor] = fminunc(@(theta) costFunction(theta, m, X, y_versicolor), iniTheta, options);
[theta_virginica, min_virginica] = fminunc(@(theta) costFunction(theta, m, X, y_virginica), iniTheta, options);

Here's the code for my costFunction:

function [jValue, gradient] = costFunction(theta, m, X, Y)
    V = X*theta;
    jValue = (-1/m)*sum(Y.*log(sigmoid(V)) + (1-Y).*log(1-sigmoid(V)));
    gradient = (-1/m)*X'*(sigmoid(V)-Y);
end

And finally my sigmoid function:

function sigX = sigmoid(X)
    sigX = arrayfun(@(n) 1/(1+exp(-n)), X);
end

The resulting theta_setosa and theta_versicolor vectors are all zeros, and the costFunction has the same minumum for the both of them. the virginica classifier seems to be working fine however. Although for each fminunc I get the following message displayed

Local minimum possible.
fminunc stopped because it cannot decrease the objective function
along the current search direction.
<stopping criteria details>


texts: Hyperparameter optimization performance comparison

texts:
I have used Bayesian optimization for hyperparameter tuning in a machine learning model. What is the best way to compare the performance of network with and without Bayesian optimization? I found some useful material here, where a comparison was drawn between Random search(grey) and Bayesian Optimization(green) as also shown in below figure. This is exactly what I want to achieve.

Script for hyperparameter tuning [MATLAB]:

% Make some data

Daten = rand(100, 3);
Daten(:,3) = Daten(:,1) + Daten(:,2) + .1*randn(100, 1);  % Minimum asymptotic error is .1
[m,n] = size(Daten) ;
% Split into train and test
P = 0.7 ;
Training = Daten(1:round(P*m),:) ;
Testing = Daten(round(P*m)+1:end,:);
XTrain = Training(:,1:n-1);
YTrain = Training(:,n);
XTest = Testing(:,1:n-1);
YTest = Testing(:,n);
% Define a train/validation split to use inside the objective function
cv = cvpartition(numel(YTrain), 'Holdout', 1/3);
% Define hyperparameters to optimize
vars = [optimizableVariable('hiddenLayerSize', [1,20], 'Type', 'integer');
        optimizableVariable('lr', [1e-3 1], 'Transform', 'log')];
% Optimize
minfn = @(T)kfoldLoss(XTrain', YTrain', cv, T.hiddenLayerSize, T.lr);
results = bayesopt(minfn, vars,'IsObjectiveDeterministic', false,...
    'AcquisitionFunctionName', 'expected-improvement-plus');
T = bestPoint(results)

% Train final model on full training set using the best hyperparameters
net = feedforwardnet(T.hiddenLayerSize, 'traingd');
net.trainParam.lr = T.lr;
net = train(net, XTrain', YTrain');
% Evaluate on test set and compute final rmse
ypred = net(XTest');
finalrmse = sqrt(mean((ypred - YTest').^2))

function rmse = kfoldLoss(x, y, cv, numHid, lr)
% Train net.
net = feedforwardnet(numHid, 'traingd');
net.trainParam.lr = lr;
net = train(net, x(:,cv.training), y(:,cv.training));
% Evaluate on validation set and compute rmse
ypred = net(x(:, cv.test));
rmse = sqrt(mean((ypred - y(cv.test)).^2));
end


texts: In a neural network with tensor input X, it seems there are times when it will never learn... Why?

texts:
import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]])
y_train=np.array([f(x) for x in x_train])

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim, activation='relu'))
model.compile(
  loss=kloss.mean_squared_error
, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=500, batch_size=1, verbose=True)

Output:

Epoch 1/50
3/3 [==============================] - 1s 221ms/step - loss: 225.9046
Epoch 2/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 3/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 4/50
3/3 [==============================] - 0s 2ms/step - loss: 225.9046

...and so on (hundreds more)


texts: encoding datetime features .. label or onehot?

texts: I am working on a cab booking prediction problem where I need to use datetime aspects like hour,day ,week etc for prediction. As I need to do categorical encoding for the purpose. can anyone help me out on which one to use i.e label encoder or one hot encoder. I am not able to gather much information on the web regarding this. I tried doing label encoding but as onehotencoding is preferred for ordinal data like this. Any help would be appreciated.


texts: Are dimensionality reduction techniques useful in deep learning

texts: I have been working on machine learning and noticed that most of the time, dimensionality reduction techniques like PCA and t-SNE are used in machine learning, but I rarely noticed anyone doing it for deep learning projects. Is there a specific reason for not using dimensionality reduction techniques in deep learning?


texts: Should I initiate size before loop and append in Python

texts: I am going to build a 1,000,000 * 2 dataframe in Python and the input I have is in the form of 1*2. So obviously the most straightforward method is to use loop and append a 1*2 pair into the existing dataframe for 1,000,000 times... However, it really takes forever and as a Matlab user, I am almost forbidden in Matlab to do this and always told to initiate the final size of the matrix (or dataframe in Python) first and put elements into the matrix... Is this the same in Python? Thanks in advance!


texts: How to pass 2 features to LSTM , one of them is one-hot-encoded with Keras?

texts:
I have a very simple LSTM model

model = Sequential()
model.add(LSTM(64, input_shape=(seq_length, X_train.shape[2]) , return_sequences=True))
model.add(Dense(y_cat_train.shape[2], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, y_cat_train, epochs=100, batch_size=10, verbose=2)

The input X_train has 2 feature , one is categorical (values 1-4) and the other is numeric (values 1-100). There are 4 classes in y_test that I one-hot-encoded with keras's to_categorical .

Should I encode the categorical input feature as well ? If I do , how can I pass it along with the other feature ? (e.g. now a timestep looks like this for example: [1,44])
Later , I would like to take make a sampling , meaning I need to take the predicted y_hat<t-1> and pass it as x<t> . I will have to pass the second numeric feature (1-100) along with it. How can it be done ?

EDIT : note that I do not want my numeric feature to become categorical since there is importance to the values (meaning 2<10<90 etc)


texts: Normalization(minmax) gives me worse results than before in KNN, follow up actions?

texts:
Hello I'm studying a classification problem with KNN right now.

I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the drop, and with this method my accuracy dropped almost 10%.

I used holdout method and cross validation for the results.

What are some follow up actions I could use to try to up my accuracy?

Should I try another normalization technique?
Should I try another model ?
Anything else?


texts: how to adjusting already built ML predictive model

texts:
How can I continue machine learning model after predicting results?
What I mean by that is that I built a model for my 1 million records dataset, this model took around 1 day to get built.
I extracted the model results using Python and now I have a (function) that I can feed it with my features and it gives me a prediction results
but with time my dataset has become 1.5 million records.
I do not want to redo the whole thing all over again from scratch.
Is there any way I continue of top of thf first model I built ( the one with 1 million records) so the new model take less time to adjust it based on the new 0.5 million records compare to re building everything from scratch for 1.5 million records.
P.S. I am asking for all algorithms, if there is anyway to do this for any algorithm that would be good to know which ones are these


texts: How to download dynamic files created during work on Google Colab?

texts:
I have two different files and on the first, I tried to save data to file as:

np.save(open(Q1_TRAINING_DATA_FILE, 'wb'), q1_data)

On second file, i'm trying to load it the same way using:

q1_data = np.load(open(Q1_TRAINING_DATA_FILE, 'rb'))

I then get the error:

FileNotFoundError: [Errno 2] No such file or directory: 'q1_train.npy'

I searched my google drive but couldn't find this file.

Platform: https://research.google.com

Edit:
I'm trying to run below Kaggle problem on Colab platform. The author has two files (Jupyter and nbs) - one to prepare and 2nd to train. The step on nb1 where it's creating some files - which later to be consumed by file 2 is where I'm struck.

https://github.com/bradleypallen/keras-quora-question-pairs/blob/master/quora-question-pairs-training.ipynb


texts:
I have a galaxy spectrum data set (total 22000). Similar to an electronic wave data, two dimensional (Flux vs Wavelength). A typical set of wavelength plot looks like below

Now I am doing kmeans on this data set to cluster them based on their spectrum shape/pattern only (using sci-kit learn). Some results of the k means clustering is baffling me, I have made a flow chart of how the candidates clustered as I would go on increasing the number of clusters from k=2 to k=5. The flow chart graph of the division of candidates is below (this is on a smaller subset),

Now my most basic question is why is the division of points in clusters happening like it is seen in the plot, as we traverse down the graph. More specifically, why for example in k=4 case in Cluster=1 it makes a group with such a mix bag ? couldnt kmeans instead select it's constituent 1078 or 580 candidates into a single (more cleaner) cluster ?
Also is it a coincidence that there is always an identified 13 members group (in golden arrow) (or a 47).


texts: Reading file from Object Storage in IBM DSX

texts:
I'm just getting started with DSX. I have two files in my Object Storage that I want to read in my Jupyter notebook:

Warehousing-data.json
Warehousing-sales_data- nominal_scenario.json

I have the credentials. Now I need the calling sequence.

Here is what have on my local copy of the notebook.

resource_path= "C:\\...\\warehousing\\"
Warehousing_data_json = "Warehousing-data.json"
Warehousing_sales_data_nominal_scenario_json = "Warehousing-sales_data- nominal_scenario.json"

resp = client.execute(input= [
    {'name': Warehousing_data_json,
     'filename': resource_path + Warehousing_data_json},
    {'name': Warehousing_sales_data_nominal_scenario_json,
     'filename': resource_path + Warehousing_sales_data_nominal_scenario_json}])
    ...

What is the equivalent when the files are located in Object Storage?


texts: Why mini batch size is better than one single "batch" with all training data?

texts:
I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this.

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration; and epoch is a full turn when each of the training instances have been seen by the model. If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?


texts: Naive Bayes vs Full Bayes model classifiers

texts:
I have a hard time to understand when Naive Bayes works better than Full Bayes.
In general, i know that naive bayes does the assumption that features are independent given the class. However, if features indeed independent, does it mean that assuming that they are dependent yield worst result?

e.g. i got this data points for two features, each class is colored in different color.

Now my intuition is that Naive bayes will work well here, given a specific class we have two different distributions of the class and both are "unstructured".
However, i did ran Naive bayes (with normal pdf) and full bayes (with multivariate pdf) classifiers on that data (using multivariate) and got the same accuracy.


texts: Deep Learning Network decreasing in accuracy

texts:
In order to familiarize myself with semantic segmentation and convolutional neural networks I am going through this tutorial by MathWorks:

Semantic Segmentation Using Deep Learning

I did not use the pretrained version of Segnet since I wanted to test on my custom data set. All code is the same, however I have different classes, and fewer labels. Below image shows the label name and amount of pixels associated with each.

To make up for the low pixel data for class 2, median frequency balancing was performed.

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount
classWeights = median(imageFreq) ./ imageFreq

I proceed to train the network using the code provided in the example with the options and lgraph unchanged. The  SegNet network is created with weights initialized from the VGG-16 network.

Unlike the example, I get a much lower global accuracy:

To gain further insight I plotted the Mini-batch accuracy and Mini-batch loss against each iteration.

It is clearly seen that the accuracy fluctuates wildly and ends up worse than it started, so the network learned absolutely nothing! However the loss decreased gradually.

A possible solution I propose would be to use inverse frequency balancing. However, in the example above, median frequency balancing was already performed, so I doubt how much this would help.

Is the terrible performance related to simply not having enough training data? Can anything be be done to improve performance with existing data?

Any suggestions are greatly appreciated.


texts: Does it make sense to concatenate datasets to improve accuracy of model?

texts:
I'm planning on training an NER model, I already do have a large corpus but I did find one more large corpus and I'm quite confident that I can source even more corpora and format its data to my needs.

Assuming all datasets eventually are in the same format and don't have any inconsistencies, does it make sense to concatenate all datasets?

Or does the model converge anyways at some point? Does that make sense at all? What other problems could I encounter?


texts: How not to see the output in the console when adding a new column in a table?

texts:
I have a data table called td and it contains 14k obs and 5k variables.
When I run the command

td[,fv_mc2:=1]

I also get the whole data table as output in my console, which floods my console. Any ideas how to fix it ? (It might also be a setting in R, but I am not aware of it.


texts: How to use CNN to deal with a 2D regression problem?

texts: I have seven measurements (Obs1-7), each measurement has the dimension of [x,y,t] where x and y are coordinates and t is time. Now I want to build a model that uses the first 6 measurements to predict the last one, saying Obs7=f(Obs1~6). I want to use CNN to distil this relationship f. I had some experience using CNN for classification, but have no idea how to deal with such 2D/3D regression. Could someone please give me some ideas? Thanks!


texts: Instance Segmentation using the predefined bounding boxes

texts:
I want to do Instance Segmentation using the images in my dataset which are already annotated and I don't want to train the model but use the pre-trained model. I was following this colab notebook. Here I can upload my image and do the instance segmentation. But the pre-trained model actually outputs the image with predicted bounding boxes and masks which is fine.
However, my questions are,

How do I use the pre-trained model for instance segmentation of an image with pre-defined bounding boxes, i.e. an array of coordinates (top left, bottom right, and class). How can I tell the model to segment objects which are inside that particular bounding box?

How can I save the mask coordinates and class in an output text/JSON file? I want that for further use.


texts: I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?


texts: What is non-decomposable and/or non-differentiable loss function?

texts: I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they different? How do we know if a loss function is non-decomposable/non-differentiable?


texts: Looking for other opinions on approach to classification problem

texts:
I'm looking to implement an "opt-out" filter for my company. The input is short, text-message style messages. A few examples of opt-out messages are:

"remove me from your list"
"remove from list"
"please unsubscribe from list"
etc.

All other messages are "good", and should not be removed.

My thoughts on approaches:

I was thinking of using a Bayesian classifier here, but not really knowing the solution space (or having much of a background in ML), want to be sure I'm not wasting time on a sub-optimal solution.

I'm fine with not having the most cutting-edge solution, but want to be sure I'm not missing an approach that might be equally as straightforward but more effective.


texts: How should the hyper parameters be defined for the other algorithms defined for sklearn_crfsuite.CRF

texts:
In the example given for sklearn_crfsuite, the parameter space that needs to be passed to a cross_validating class like RandomizedSearchCV is defined as below.

params_space = {
    'c1': scipy.stats.expon(scale=0.5),
    'c2': scipy.stats.expon(scale=0.05),
}

In the example the algorithm that is shown is "lbfgs". Other supported algorithms in sklearn_crfsuite are "l2sgd", "ap", "pa", "arow". How should the variable params_space be defined in those cases?


texts: Organizing a csv file of multiple datasets into a list of Pandas dataframes

texts:
I have a csv file, containing results from a Computational Fluid Dynamics (CFD) simulation (a sample of my csv file is attached as a google drive link; file size: 1,392KB). In particular, the csv file has information about multiple streamlines (number of streamlines may reach 1000 depending on the case). All the data for all the streamlines are saved back to back in the csv file (so there is no empty row or something to tell us the end of one streamline and start of the next one). The only way I can distinguish streamlines from each other is that when the value in column "IntegrationTime" is zero, it indicates the start of a new streamline, until we hit another zero in the"IntegrationTime" column which is the start of the next streamline.
I need to read this csv file, and organize its data into a list of Pandas datafreames, like:
streamlineList = [df_for_streamline_1, df_for_streamline_2, ...., df_for_streamline_N]

Note (extras question here): This is not crucial but would be nice to have: if you look at the end of my csv file, you see multiple rows where IntegrationTime is zero (100 rows to be exact). Preferably, I don't need these lines to be included in my final list of data frames.
Can somebody help me a way to do this?
https://drive.google.com/file/d/1lJhOJadGrno1C-KZOUqxV-KNkaOHIJNk/view?usp=sharing


texts: Why does a confusion matrix in Orange not show results?

texts: I ran a neural network on a data set and when I checked the confusion matrix I got nothing but a message that says: Confusion matrix cannot show regression results.


texts:
I am using the InceptionV3 model for training. Here is the link for the code (https://github.com/maxmelnick/tensorflow/blob/no_random/tensorflow/examples/image_retraining/retrain.py) Initially I have a small size dataset. So, I used the augmentation technique to increase the size of the dataset.
While training phase dataset was divided into training, validation, and testing. During the training phase, it shows 96% accuracy for 11 classes. But When I predict any new input image(Unseen data) it gave 56% accuracy. What's the problem lies with the model?
I have already used Dropout, Cross-validation, OverSampling techniques but not achieved good results over the new input image.
Parameters used while training.
Training Samples - 800 images in each class

Training Samples - 70%
Validation Samples - 20%
Testing Samples - 10%

Testing Samples (Unseen data other than Training Samples) - 51 images in each class
Epochs - 10,000
Thank You.


texts: I am currious if there is a shortcut for runing the  selected cell in IPython notebook such as alt+F5 or ctrl+f5 or something simmiliar  that is present in Visual Studio (I don't recall exactly now ) , cause I found it annoing to go with  the mouse on  the run  button every  time I need to run a cell?


texts: Using iGraph to build a Distribution Model

texts:
I would like to analyze the distribution of the Customers from a Shop, if the Shop is closed or terminated. Consider the following sample data-set;
| ShopID | MonthlyCVisitCount |   Lat     |     Lng    |
--------------------------------------------------------
|  A1    |      15000         |  39.84349 |  116.33986 |
|  A2    |      24560         |  39.84441 |  116.33995 |
|  A3    |      14789         |  39.84615 |  116.34012 |
|  A4    |      35479         |  39.84891 |  116.34039 |

I would like to build a distribution model using NetworkX or iGraph based on the distance (i.e., lat, lng) and determine how the distribution (MonthlyCVisitCount) would spread if I supposedly close the shop A1.
Is it possible to obtain the shortest path from A1 to A2, A3, A4 and based on some business rules, I can analyze the distribution.
Edit 1:
Consider the modified dataset;
| PrimaryShopID | ToShopID |  Dist |
------------------------------------
|     A1        |    A2    | 0.125 |
|     A1        |    A3    | 0.354 |
|     A1        |    A4    | 0.950 |

The dataset above has distance between ShopID A1 to multiple shops, now that I have distance calculated and based on this distance I provide weights, say;

0 to 200 - 50%
201 to 500 - 20%
501 & above - 0%
Meaning that 30% of Customers I have lost.

How do I proceed with this scenario, I am mostly comfortable in working with R, any example or a pointer to the same problem will be helpful


texts:
I know this must exist, but I'm having enormous trouble finding the right search terms.

Say I have a bunch of labelled 3D points, and I capture multiple 2D images of it. If I want to reconstruct the 3D points, are there well-established algorithms/libraries for doing this?

This is presumably the basis for 3D facial recognition, which is a well-established field of research, but the general case (i.e. non-faces) doesn't seem to have an obvious literature that I can find.

One way I can see for approaching this is an optimisation problem where each 2D image establishes a minimum distance constraint between each point, and one can reconstruct the 3D points by minimising the distance required to satisfy these constraints. This does feel like one of those problems that has a sneaky linear algebra solution, though.

Does this class of problem have a literature I can search? Does it have existing libraries? (I'd be really surprised if there wasn't something in OpenCV, but I don't really know what I'm looking for)


texts:
I hope this question isn't out of place here.

I have some time series data from a Zooz power plug that I'd like to visualize.  Excel works fine to a point, with small sets of data, but it's not easy to move the data window, to zoom in on particular moments, for example.

Are there any free data visualization apps that accept time series data and implement controls for viewing it dynamically?  I'm happy to pay for something, too, if it works well, but this is for a personal hobby project, so I'm trying to spend my money carefully :)  Looking for off-the-shelf solutions before I dive into R.  Thanks.


texts:
I want to apply some segmentation on a dataset for preprocessing purposes. I have tried the "otsu thresholding" approach in order to segment the image. It's a good method, however, I think a clustering algorithm such as K-Means can be even more succesful for the basic segmentation. Here is some problem with Otsu Thresholding:

As you can see, in some samples, Otsu thresholding may fail like in the picture 1. I think that, using a pixel-wise positioning approach as well as color thresholding may increase the success even further.

However, I couldn't manage to work with scikit-learn's KMeans function to work with an RGB image, since it is a 3 channel 2-d matrix(basically 3d matrix).

How can I combine both pixel-wise distance approach as well as color clustering with K-Means clustering or alike method?


texts: How to handle associated features in machine learning

texts:
I am working on a classification project in which some features are linked and I'm not sure how to handle them.

I will simplify my project like that :

There are different jobs, and multiple persons working on those jobs.
Those persons are fit for their job.
Replacing someone can only have negative effect or no effect.

I have a person1 that can be replaced by a person2 for a job, and the goal is to predict if it has a negative impact or not.

Each person has is own properties like : weight, age, height, IQ,..
A job also has properties like : manualJob, localization, temperature,..

When I list my features I have something like that :
(Person1 = P1, Person2 = P2, and the data has been normalized)

     P1_weight   P2_weight   P1_IQ    P2_IQ    manualJob   tempJob   neg_impact
0       0.25       0.50      0.25     0.25        1        0.25       1
1       0.75       0.25      0.50     0.25        0        0.50       0
2       0.50       0.75      0.75     0.50        1        0.25       1
...

There should be a high interaction between P1_weight && P2 weight features, (as well between IQs features) that we want to capture in order to predict the neg_impact feature.

1. Now the change between P1_weight and P2_weight is important, but does a classic model as RandomForest can capture the link between those 2 features ? Same for other subject properties (P1_IQ && P2_IQ, P1_height && P2_height,..)

2. I'm afraid that if I reduce the difference between P1_weight and P2_weight into a single feature as the diff (P1_weight - P2_weight), I will lose some information. For example the P1_weight would probably be correlated to the 'manualJob' feature, and if I delete P1_weight this information would be lost?

3. I was thinking that maybe I could preprocess those linked features with out-of-fold prediction, and use those predictions as input with the rest of the features original features (manualJob, tempJob, ..). Is it a good idea ? Which kind of model would be better for the preprocessing to capture the correlation of those linked features ?

NB : my set only contains ~1000 elements


texts:
I have a method that calculates a certain variable. This variable is biased by an error which should be related to the observables that were used to calculated the variable. I don't have an exact formula of the calculation and also no errors of the observables, therefore I can't just do error propagation.

I do have training data (the true error). Now I want to use machine learning to find the correlations between the input observables and the error of the calculated output. For now I tried to gather as many observables and used their statistics (mean, variance etc.) as "feature vectors" for machine learning. So far I tried approaches like SVR, linear Regression, SGD Regression etc. but all showed the same bad result.

I'm quite new in this field. So some things are still a little bit unclear to me. Do you have any ideas how to find the relations? Or do you think there is no relation at all? What if there are observables that are totally random; do they destroy the learning?

Thanks in advance!


texts: Which neural network to choose for classification from text/speech?

texts:
I am considering two tasks:

Dialog Act Classification from Text (e.g. classify to: question; opinion; ...)
Emotion Recognition from Speech (e.g. happy; calm; sad; ...)

Which DL model should perform better for such tasks? I am planning to use CNN which should work for both of them, however not sure how well. Can I apply LSTM or some other methods? I used Keras before.

Is it good to apply attention mechanism or some other approaches for these 2 tasks?


texts: Replace Missing Values with Most Frequent number under Condition

texts:
I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster
df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]

I tried to do that using SimpleImputer:
from sklearn.impute import SimpleImputer
Imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')

Imputer.fit_transform( pd.DataFrame(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]) )

but it doesn't work and tried to save values to the column:
df.loc[(df.Age.isnull()) & (df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]), 'Age'] = Imputer.fit_transform( pd.DataFrame(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]) )

but doesn't work also.
I tried to do it manually using fillna()
df.loc[(df['Sex'] == 0) & (df['Pclass'] == 1), 'Age'].fillna(int(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)].mode()), inplace=True)

I tried to use indexes to access rows and update their values:
mod = int(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)].mode())
indices = df.loc[(df.Age.isnull()) & (df.Sex == 0) & (df.Pclass == 1), 'Age'].isnull().index
df.loc[ind, 'Age'] = mod
df[(df['Sex'] == 0) & (df['Pclass'] == 1)]['Age'].isnull().sum()

it worked and the output was: 0, but when I'm trying to apply it in for loop it gives me an error
for i in range(1,3):
    for j in range(1,4):
        indices = df.loc[(df.Sex == i) & (df.Pclass == j), 'Age'].isnull().index
        mod = int(df.Age[(df['Sex'] == i) & (df['Pclass'] == j)].mode())
        df.loc[ind, 'Age'] = mod

I want to know what is the wrong of first 2 ways an why the 3rd doesn't work in loop?


texts: Looking for a model architecture that can work with an unordered list of arbitrary length

texts:
I'm looking for a machine learning model architecture that takes in an arbitrary number of inputs and generates one output. Pretty much like GRU or LSTM, it's just that the order of the items in the input is irrelevant. So f([x1, x2, ..., xn])=y, where each x is of shape [i] while y is of shape [j] (not considering the batch dimension). And f([x1, x2, ..., xn])=f([xn, xn-1, ..., x1]) or any other order of the input. In other words, f treats its input as a set, unlike RNNs that treat their inputs as a list.
Is there such an architecture?


texts: How can I detect blocks of text from scanned document images

texts:
ORIGINAL IMAGE:

GOAL:

I want to separate texts into individual paragraphs by placing bounding boxes over them (as shown above).

I tried it do this via traditional computer vision approach using opencv.

I plotted character level bounding box
Next, I gray-scaled the image, binarized it.
Applied dilation
And finally placed bbox over the dilated image.

This is what I get:

> #Morphological Transformation

kernel = np.ones((3,4),np.int8)

dilation = cv2.dilate(im_bw, kernel)

cv2.imwrite('dilated.png', dilation)

Plotting rectangular box

ret,thresh = cv2.threshold(im_bw, 127,255,0)
image, contours,hierarchy = cv2.findContours(thresh,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE )

for c in contours:
    rect = cv2.boundingRect(c)
    if rect[2] < 50 or rect[3] < 50 : continue

    print (cv2.contourArea(c))
    x,y,w,h = rect
    cv2.rectangle(im_new,(x,y),(x+w,y+h),(0,255,0),2)

cv2.imwrite('sample_res_inner.jpg',im_new)

Since the image is scanned image plus the line-spaces between them are small I couldn't able to segment them based on paragraphs.

How can I get my desired result?


texts: How does Google's Universal Sentence Encoder deal with out-of-vocabulary terms?

texts:
It seems to output embeddings even for random jibberish, and the similarity is even high for this particular pair of jibberish.
np.inner(embed('sdasdasSda'), embed('sadasvdsaf'))

array([[0.70911765]], dtype=float32)

I'm wondering how sentences are tokenized and what preprocessing steps are done internally. Also, how is the embedding model trained? As I understand it, they use a Deep Averaging Network, which is another neural network applied on the average of the individual word embeddings?


texts:
I am building a 3 neural network models on dataset that is already separated  to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train set. And this gives a certain limitation or maximum capacity to my neural network model(s). By this I mean, I can not seem to improve the accuracy even if I change the hyper parameters or the parameters of my models.

I have created 3 neural networks models and varied almost everything:

Number nodes/hidden layers,
Input features (performed feature selection and space reduction),
Activation functions and loss functions,
regularization, optimizer and more,

When I try to average the predictions of the 3 models, I don't see any improvements. Although I've read a lot that if I change such parameters I might have some uncorrelated models. But this wasn't the case for me because I always find correlation between my model predictions when I compute Pearson Correlation

After building all these models, I am pretty sure that the training set and test set are not drawn from the same distribution (i.e. they are not a random split of some full original dataset), which means that other features probably also have a different distribution.

Some proposed I could merge the training+test, but I don't want to do that as this dataset was developed in this way. But I would like to draw my validation set from the test set, is this possible? Can I use a validation set randomly drawn from the test set to tune models?


texts: How is the target variable passed to the final estimator in this pipeline?

texts:
There is a pipeline like below. X is features and y is the target variable.
I would like to know how y is passed to the estimator, LinearSVC. As far as I know, StandardScalerreturns only transformed X. So, I thought that y was not passed to LinearSVC. However, this code worked and I could make prediction. Thus, I would like to know how y was reached to the final estimator.
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.svm import Linear SVC
svm_clf = Pipeline([
            ("scaler",StandardScaler()),
            ("linear_svc",LinearSVC(C=1,loss="hinge")),
          ])
svm_clf.fit(X,y)


texts:
How do I perform weighted loss in multiple outputs on a same model in Tensorflow? This means I am using a model that is intended to have 3 outputs. I did this because I would like the network to learn the relationships of the input variables. More specifically, this is multi-output regression.

The output model shape is like the following:

tf.Tensor: id=606, shape=(3, 3), dtype=float32, numpy=
array([[ 0.01483282,  2.9515972 , -0.07268244],
       [-0.8814691 ,  2.543654  ,  0.08576971],
       [-0.6933001 ,  1.3419302 , -0.25192362]], dtype=float32)

The model.summary() is below:

Layer (type)                 Output Shape              Param #
=================================================================
simple_rnn (SimpleRNN)       (3, 32)                   1152
_________________________________________________________________
dense (Dense)                (3, 3)                    99
=================================================================
Total params: 1,251
Trainable params: 1,251
Non-trainable params: 0
_________________________________________________________________

The shape is (3,3) with 1st index being the batch size (the network is an RNN). I need to set weighted loss for each of the outputs. I tried the argument weighted_losses on tf.keras.models.Model but the error returned:

ValueError: When passing a list as loss_weights, it should have one entry per model output. The model has 1 outputs

What is the correct method to have weighted loss using a single model?


texts: Transform a skewed distribution into a Gaussian distribution

texts:
I have a skewed distribution that looks like this:

How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as the order of values remains the same. I'm doing this to experiment if different distributions change the behavior of my ML models.
I'm working with Python/NumPy/Pandas/scikit-learn.
Edit: I should clarify that I have a lot of features and I'm looking to automatically transform all feature distributions. I was able to find a reasonable transformation for a single feature with a lot of experimentation, but it doesn't generalize to other features:
normalize(np.log(0.30 + original)).
** here would be image i.stack.imgur.com/uzorK.jpg but I don't have enough rep to post more than 2 images **
normalize(np.log(0.17 + another_feature_distribution)).

In this image the purple bars represent the original distribution of another feature, green bars represent the transformed distribution. No matter how much I tweak the constant, I don't get the high green bar on the left extreme to disappear. Also, I don't have time to manually find a formula for each feature. Not sure if these are bell-shaped enough anyway?


texts: How to force weights to add to $1$ in Linear regression

texts:
I am using a linear regression using scikit-learn in python. However, I would like to force the weights to add to $1$. Is there a way to do this? I know that I need to add a constraint but am not able to figure out how. My regression looks something like this $Y= a_0 + bX_1 + (1-b)X_2$

Thanks in advance


texts: Machine learning PhD Interview technical questions

texts:
I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical questions do professors generally ask in Ph.D interviews ?

Thanks in advance.


texts: Why does my random forest classifier predicts one class more often?

texts:
I have a random forest classifier that predicts 0 class about twice as often as class 1. It also predicts class 0 with higher probabilities than class 1.
It is not a imbalanced dataset.
I tried setting class 1 weight to 100 and it seems to solve the problem, though I suppose it's not a correct solution :D
K-NN  gives the same problem.
Since I changed y from 0 and 1 to B and A it started to predict second class more frequently. So can the problem be somehow connected to data type?
Code:
dataset = pd.read_csv('regtraining.csv')
X = dataset.iloc[:, :-5].values
y = dataset.iloc[:, 50].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
classifier = RandomForestClassifier()
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

print(confusion_matrix(y_test, y_pred))

When I try multiple random states of train test split, one of predicted classes is always predicted much more frequently.
Edit: After some research I think the random forest splits so that the classes are predicted in an alphabetic order (A, B values for y give more of A but B, A values for y give more of B).
Thanks for your answers, I am new to machine learning :D


texts:
I have daily data for 2.5 years , but with more data points as 0, so when i excluded them in the cases which seems to be invalid. Can i use any other model than models used in time series or should i consider time series with missing values and proceed ?
since they are quite a lot of missing values and am new to time series i dont know how to proceed with this. Any input is highly appreciated.


texts: Logistic Regression or regression SVM for probability of outcome

texts:
I am working on a prediction question: what's the percentage of Y = 1 using a number of features?

The output Y values I have for training are in binary.  In this case, should the prediction be treated as classification or regression?
Would logistic regression that returns the probability be suited or the probability of SVM, e.g., this?


texts: How can I train a model to modify a vector by rewarding the model based on the modified vectors nearest neighbors?

texts:
I am experimenting with a document retrieval system in which I have documents represented as vectors. When queries come in, they are turned to vectors by the same method as used for the documents. The query vector's k nearest neighbors are retrieved as the results. Each query has a known answer string.

In order to improve performance, I am now looking to create a model that modifies the query vector. What I was looking to do was use a model that rewarded the model for each of the top k nearest neighbor's corresponding documents that contain the answer string and punish when the string is not present.

In looking for a solution I have been mainly finding results related to multiple rounds of discrete decision making, so I am also not sure if this would count as reinforcement learning or something else entirely. Thank you for any help.


texts: How does Wav2Vec 2.0 feed output from Convolutional Feature Encoder as input to the Transformer Context Network

texts:
I was reading the Wav2Vec 2.0 paper and trying to understand the model architecture, but I have trouble understanding how audio raw inputs of variable lengths can be fed through the model, especially from the Convolutional Feature Encoder to the Transformer Context Network.
During fine-tuning (from what I have read), even though audio raw inputs within a batch will be padded to the length of the longest input in the batch, the length of inputs can differ across batches. Therefore this implies that the output from the Convolutional Feature Encoder will have varying lengths across batches.
However, the Transformer Context Network has a fixed input dimension; the BASE Wav2Vec 2.0 model uses a transformer of model dimension 768. This means that the output from the Convolutional Feature Encoder must somehow be "manipulated" to become dimension 768 in order to feed into the Transformer.
How is this "manipulation" done? The HuggingFace's Wav2Vec model (see below) showed that there is a Wav2Vec2FeatureProjection layer between the Convolutional Feature Encoder (a.k.a., Wav2Vec2FeatureExtractor) and Transformer Context Network (a.k.a., Wav2Vec2Encoder). The Wav2Vec2FeatureProjection contains a linear layer that takes input with dimension 512 and output dimension 768. How is the input dimension 512 determined when the raw inputs can have varying lengths across batches?
Wav2Vec2ForCTC(
  (wav2vec2): Wav2Vec2Model(
    (feature_extractor): Wav2Vec2FeatureExtractor(
      (conv_layers): ModuleList(
        ...
        (6): Wav2Vec2NoLayerNormConvLayer(
          (conv): Conv1d(512, 512, kernel_size=(2,), stride=(2,), bias=False)
        )
      )
    )
    (feature_projection): Wav2Vec2FeatureProjection(
      (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      (projection): Linear(in_features=512, out_features=768, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): Wav2Vec2Encoder(
      (pos_conv_embed): Wav2Vec2PositionalConvEmbedding(
        (conv): Conv1d(768, 768, kernel_size=(128,), stride=(1,), padding=(64,), groups=16)
        (padding): Wav2Vec2SamePadLayer()
      )
    ...


texts: How to handle non-stationary data in online neural network based one-class classifier for anomaly detection?

texts:
I have developed online neural network based one-class classifier and also enabled it for forgetting mechanism. So, now online learning with forgetting mechanism is possible. But how to handle if data is non-stationary. As it is one-class classification so, suppose we have two class data i.e. normal and outlier class then trained by only normal class data in online fashion. But training data might have different distribution compared to distribution of normal and outlier data in testing as their distribution are changing. So, how algorithm will handle this drift/shift.

One more doubt regarding non-stationary data handling: whether we will pass trained labelled data once and further labelled data for training are collected from unlabelled testing data?


texts: Cross Validation (sklearn train test split) - ValueError: not enough values to unpack

texts:
I have written this code and getting error:

from sklearn.model_selection import train_test_split
X0 = pd.read_csv('C:/Users/Desktop/Dataset for Experiment - 2019/ANFIS modeling data/china_data.csv')
train_test_split. X0 = X0.drop(columns = ['EFFORT'], axis = 1)
X_train, X_test, y_train, y_test = train_test_split(X0, test_size=0.8)

The error which is showing for this code

X_train, X_test, y_train, y_test = train_test_split(X0, test_size=0.8)

  ValueError: not enough values to unpack (expected 4, got 2)

I tried many times to remove this error but it is still persistent. Can anybody explain to me why I am getting this error & how can I remove this error?


texts: Difference between two images but in certain region only

texts: I have two images, where i need to find the difference. i can easily did with the help of SSIM function. Now i want to find the difference between two images but only selected region (i want to ignore some region , which i marked as a rectangular).


texts: Need to find out algorithim for entity recognition

texts:
I am new to named entity recognition. I have a requirement to extract some specific data from corpus.
Two corpus example:

Hi, Rajesh Das want to buy this car. Rakesh Das will use this car.
Hi, Rakesh Das is brother of Rajesh Das. Rajesh Das will buy this car.

From these above example I need to extract the buyer name. So obviously the name which is near to buy word will be buyer name. Here position of Rajesh Das is different for two corpus. I can not figure out which algorithm need to implement for extracting buyer name.
I followed the https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da post which implement SpaCy and NLTK. It only provides all name list. Can someone provides me the idea how to get buyer name.


texts: Predict how many days late or early someone will finish their work

texts:
So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself were articles, so I also have the word count for each. How do you, based on the previous data, calculate the amount of days earlier or later somebody will most probably finish their work?

As a concrete example of the problem I'm trying to solve:

John finished his last 5 projects 5,4,3,6,2 days late. What is the most probable amount of days earlier or late he will finish his work?

Basically I'm looking for an appropriate machine learning algortihm to implement to calculate this probable end date.


texts: Which algorithms should I use for identifying similar characteristics between data points (the intersections)?

texts:
I am working with a dataset that has been coded and categorized, so that each datapoint has a set of coded characteristics. An example data point would be something like the following:

Example Data Point:

Quality
Service & Support
Price

Each data point can have multiple codes associated with it.

What I'm looking to do is identify the "intersections" between the data points so that I can answer questions like the following:

When a data point has "Quality" as a characteristic, 25% of the time it also has "Price" as a characteristic

I've been struggling with the right way to ask this question in my Google searching and realized I should just come to the experts on topics like this and get your help and guidance.

To do this type of work, what algorithms should I be investigating?

Thank you for your help!


texts: Optimizing execution speed of a series of deep neural networks

texts:
I have a series of Neural Networks that I run on some video data. First network detects bounding boxes, then it extracts features for object tracking, matching each box to an id frame by frame and finally some other CNN is applied to all "boxes" extracted.

Naturally this is very slow and I understand that this is just so much you could do, but I wonder if there's a way to optimise this for speed?

Some suggestions I've heard is to concatenate my networks. I've used Thread class with Queue with some effect but want to try something a bit more radical


texts: BPE vs WordPiece Tokenization - when to use / which?

texts: What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a general overall answer, backed up with specific examples.


texts: Alternative Hunspell dictionary for stemming

texts:
I am using Hunspell to spellcheck and stem the words in my documents to reduct dimensionality. For spellchecking Hunspell works great with the default en_US dictionary by SCOWL (and friends), but not so for stemming.

The reason is that the dictionary is pretty inconsistent. (e.g. some words have their plurals in the dictionary, but some others doesn't, so it's inconsistent if the stem of a word in plural will be in plural or not).

Hunspell is definitely able to stem english words consistently, but it needs an other dictionary that is more "correct" morphologically.

What alternative dictionaries can I use to improve the quality of stemming English text with Hunspell?
If there are no such dictionaries, what other tools can I use for stemming? EDIT: note that I'm using python, which can be hooked up with most libraries and command-line tools easily, but R or Matlab specific tools might not fit in my current ecosystem that well.


texts: When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

texts:
I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.

This completely changed my perspective of how I work with data. Not only should I clean the data, but the data should also be formatted correctly. Seems pretty obvious when you think about it, but that is besides the point.

I decided to start applying those principles into my data cleaning workflow, however, I was wondering whether there are times where having tidy data would not be ideal?

When would we not want to "tidy our data"?

Ideally when should we use Tidy Data and when should we avoid it?

Your input to this would be highly appreciated.


texts: In which situation should we consider a dataset as imbalanced?

texts:
I'm facing a problem about making a classification on a dataset. The target variable is binary (with 2 classes, 0 and 1). I have 8,161 samples in the training dataset. And for each class, I have:

class 0: 6,008 samples, 73.6% of total numbers.
class 1: 2,153 samples, 26.4%

My questions are:

In this case, should I consider the dataset I used as an imbalanced dataset?

If it was, should I process the data before using RandomForest to make a prediction?

If it was not an imbalanced dataset, could somebody tell me in which situation (like what ratio for each class) I could consider a dataset as imbalanced?


texts: How can I train a model for localizing objects(classification not required) in Python

texts: I need to make a model that creates bounding box around objects(but does not classify them) for a competition. Which libraries or pre-trained models should I use. I need values of x1,x2(x1+w),y1,y2(y1+h), where all dimensions are measured from top left corner. The accuracy is measured by IOU(Intersection over Union).


texts: Curation of a dataset of audio files in different formats

texts:
I have a large dataset of audio files (around 80k files of around 45 minutes long each) that are related to my work. Some of them are in format ".m4a" and others in format ".mp3". I am not a computer scientist, but I understand that these are two different formats.

I will use the audios later on for a variety of analyses. My main concern for now is to compile a homogenous dataset. It is bugging me that the audio files are encoded in two different formats. I could convert one into another, but I am afraid I will lose quality on the converted files.

What would be a good strategy for me? Should I just keep the files as they are, and only convert local versions for analysis?


texts: Python: how to handle categorial values in dataset to build models

texts:
I have a training dataframe dfTrain and the output of dfTrain.head() is shown below:

    C0 C1                              C2   C3 C4  C5               C6
0   1  73                 Not in universe   0   0   0  Not in universe
1   2  58  Self-employed-not incorporated   4  34   0  Not in universe
2   3  18                 Not in universe   0   0   0      High school
3   4   9                 Not in universe   0   0   0  Not in universe
4   5  10                 Not in universe   0   0   0  Not in universe

There are total 38 features and they are both categorical and numerical. Ignoring C1 and scaling numerical features, I am trying to build a Logistic Regression model. Since, the dataframe has categorical features, I am creating another dataframe which has dummy variables.

X = pd.get_dummies(dfTrain)

The shape of X now has 160 features which is much more than that of dfTrain.

Then I pass X and y (where y is target variable) to Logistic Regression Classifier

modelLogistic = LogisticRegression(C=10**-2, class_weight = 'balanced')

modelLogistic.fit(X, y)

The reason to use class_weight = 'balanced' is that there are 17 classes in y and highly imbalanced.

My question is: is my approach correct? Am I missing anything?


texts: Does image resizing lower the prediction accuracy of MLP?

texts: I am implementing a vanilla neural network (MLP) to do image classification in python using tensorflow on images of honey bees to detect their health status. The images in my dataset are of different shapes and sizes, so I decided to do image resize using cv2. All my images are now of the same size (64 by 64) but some of them have been stretched/shrieked due to resizing. Does this have an effect on the low prediction accuracy I am getting from my MLP?


texts:
I wanted to start off by saying this is not an exact duplicate of the other question. I checked it and it didn't have what I urgently need.
So here is the problem. I have a dataset with 30 features and 100000 rows of train data. I want to make a Binary classification model which determines whether the person is eligible or not for membership at our club. I am a rookie data scientist, and binary classification is a first for me. So please help me and tell me which model would be the most accurate for this purpose. Also, the time taken by the model to train doesn't matter... Thank you so much

update:
Ok, I have used logistic regression and instead of giving some members as accepted(1) it is showing me al members do not get the membership...So I thought there might be a mistake in choosing the Model, so I am looking for other models...


texts: Why we shift target(output) by one offset in language modelling

texts:
I have been working in sequence prediction tasks (very similar to language modelling)  where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach like the below for data preparation
[0,1,2,3] -> [4]
[4,5,6,7] -> [8]

or
[0,1,2,3] -> [4,5,6,7]
[4,5,6,7] -> [8,9,10,11]

I have recently seen people taking output/target as one offset shifted from X. Since the goal of language modelling is to predict next token. Why and in which situation would one use this data format?
For example a famous book D2l.ai provide example format of the data as
X:  [[27. 28. 29. 30. 31.]
 [12. 13. 14. 15. 16.]]
Y: [[28. 29. 30. 31. 32.]
 [13. 14. 15. 16. 17.]]

Isn't it predicting token 28 after tokens [27. 28. 29. 30. 31.]wheras actaly Y(output) should be [32] or [32,33,34,...] ?
How would next prediction (which actually is 32) be retrieved after token 31?


texts: What does it mean to be “stable to deformation”?

texts: In the context of image classification, what does it mean to be stable to deformation? Say I were trying to classify digits, what would the difference be between an operation that is stable vs unstable to deformation?


texts: sklearn - SimpleImputer in an empty Pipeline

texts:
When building a Pipeline I'm ending up at a scenario that can be simplified like this:

FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))

Since this is one intermediary step in a larger Pipeline, I'm feeding the preceding inputs into both of these and select the corresponding dtypes within the Numerical and Categorical Pipelines.

For some datasets, however, no Categorical Columns are left leading the Pipeline to fail. I've tried returning an empty list and 'None' but both of these did not result in the Pipeline skipping the "empty" CategoricalPipeline.

After further investigation it turns out that the SimpleImputer() in the CategoricalPipeline causes the error. Depending on the order of steps the following messages are shown:

ValueError: Found array with 0 feature(s) (shape=(150, 0)) while a minimum of 1 is required.

ValueError: at least one array or dtype is required

Any ideas on how to pass the Imputer when no Column is present?


texts: LSTM: many to one and many to many in time-series prediction

texts:
I am trying to predict the trajectory of an object over time using LSTM. I have three different configurations of training and predicting values in my mind and I would like to know what the best solution to this problem might be (I would also appreciate insights regarding these approaches).

1) Many to one (loss is the MSE of a single value)

The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
The loss function is the MSE of the predicted value and its real value (so, corresponding to the value in position $n+1$).
During the online test, a sequence of $n$ values predict one value ($n+1$), and this value is concatenated to the previous sequence in order to predict the next value ($n+2$) etc.. This way, a whole trajectory of $n + t$ values is calculated.

2) Many to one (loss is MSE of multiple values)

The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
To compute the loss function, the same strategy used before for online test is applied. LSTM predicts one value, this value is concatenated and used to predict the successive value $t$ times. The loss is the MSE of all the predicted values in the trajectory and their real values. Backpropagation is only done when the whole trajectory has been predicted.
Online testing is equal to the previous situation.

3) Many to many

The $input$ is a sequence of $n$ values, the output is the prediction of $m$ consecutive values.
The loss function is the MSE of the $m$ predictions and their corresponding ground truth.


texts: Wrong result when solving: "chance that two random cards differ in color and value?"

texts:
I'm trying to build a simulation for this question:

"There are 50 cards of 5 different colors. Each color has cards
numbered between 1 to 10. You pick 2 cards at random. What is the
probability that they are not of same color and also not of same
number?"

(From Glassdoor)
I should have a result like "73%" but with my code I get (consistently) "72%" or "71.8%".
Here is my code:
import numpy as np

# Building a deck of 10 cards for each of the 5 colors
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(1, 11)])

def random_cards_differ():
"""Returns True if two random cards differ"""
    a, b = np.random.choice(cards, 2, replace=False)
    if a[0] != b[0] and a[1] != b[1]:
        return True
    else:
        return False

nb_success = 0
nb_tries = 100000

for i in range(nb_tries):
    if random_cards_differ():
        nb_success += 1

print(nb_success / nb_tries)
>>> 0.71892

Is this normal? Is there a mistake in my code or is it a "random gotcha" caused by some seed or something else?


texts: Region Proposal Network - How to subsampling 256 fg/bg anchors

texts:
I am trying to understand training process of RPN. I have problem with creating mini batches of 256 anchors. If features map has shape 18x25=450 and every position has 9 anchors it is 4050 potential anchors. Output class shape will be 18x25x18 and output regression shape will be 18x25x72. How to select only 256 anchors? I read that we have to select 128 fg and 128 bg randomly. If we label fg anchors with [1,0] and bg anchors with [0,1], how to label anchors which should be ignore? With [0,0]? I don't think [0,0] will prevent backpropagation of loss over anchors which should be ignored.
For example, if we have 9 anchors for same position on features map and only 1 of 9 anchors is fg, while others are bg (based on iou calculations). My question is how to discard 7 other anchors, ie. how to make 50:50 fg:bg ratio (in this example 1fg, 1bg and 7 discarded anchors)? If we can't change the output shape, we have to somehow discard redundant bg anchors. We don't want RPN to backpropagate loss over all of 9 anchors. It will be biased to bg anchors.
Please correct me if I am wrong and tell me what is wrong.


texts: I've been trying the kaggle dataset of Credit card fraud detection Dataset . I've used ANN using keras and tensorflow. You can find the code in the screenshot. The only problem is im getting accuracy to be around 99.9 % , so i think it's surely a case of some false hopes or over fitting. Can you please tell whats wrong with it? And even my test set gave a result of 99.93% accuracy.


texts: Why are mini-batches degrading my conv net MNIST classifier?

texts:
I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 maxpool layer and a 10 node dense layer with softmax as the activation function. I am using cross entropy loss and SGD.
When I train the network on the whole training set for a single epoch with a batch size of 1, I get 95% accuracy. However, when I try with a larger batch size (16, 32, 128), the learning becomes very noisy and the end accuracy is anywhere between 47%-86%. Why is it that my network performs so much worse and noisier on mini-batches?


texts:
We consider the below function :
$f(x,y,z,a) = x*y*z*a$, where $x,y \in \mathbb{Q}\cap[0,1000],$ $ a\in\{2,3\} $
and $z=z(x)$, taking values from the below table based on the level-range that $x$ belongs to.

For instance, if $x=150$, then $z=5.$
I am looking for the proper visualization tool that can produce all possible 2-dim and 3-dim graphs of the above function.\
For instance,

if I fix $x=150$ and $a=2$, I want the 2-dim graph $(y,f(150,y,5,2))$
if I fix $a=2$, I want the 3-dim graph $(x,y, f(x,y,z(x),2) )$
and if I fix $y=10.5$, I want the two 2-dim graphs $(x,f(x,10.5,z(x),2))$ and $(x,f(x,10.5,z(x),3))$

I would like your suggestions on the above. Would Tableau do the job here ? I am not familiar with any tool and I am trying to chose the most situable to start with. Should I consider some other tool maybe? (Power BI or ??)
In addition, please advise on the functionality and the advanced options provided by the above tools.


texts: Users tend to click on results ranked highly by search engines much more often than those ranked lower. How do you train a search engine using click data / search logs without this bias? I.e. you don't want to teach the search engine that the results that are currently ranked highly should necessarily continue to be ranked highly just because they were frequently clicked.


texts: Building a data frame and matrix for citation network

texts:
I have a question about how to create a matrix that will take the lists defined below and make them the entries for the columns and rows. My data involves two lists: one lists originals sources (or papers) and the other lists those sources' individual citations.

The code is done in RStudio.

A sample of my original dataset looks like this:

 # List all the sources
 sources.vertices.levels <- cit.total$Sources
 sources.vertices <- (as.character(sources.vertices.levels))

 citation.edges.levels <- cit.total$Citations
 citation.edges <- (as.character(citation.edges.levels))

 # Break up the factors into character lists
 list.sources <- levels(sources.vertices.levels)
 list.citations <- levels(citation.edges.levels)

Below I have created an empty matrix (50X1497). I would like to make the row names be each individual source, and the columns be each individual citation.

Also I am lost on how to create a binary matrix that matches each individual source (from the rows) to the individual citations (to the columns).

 # Building an aggregate matrix
 matrix.network <- matrix(cit.citations,ncol=1497, nrow=50)

An example of what kind of matrix I want to do is below:


texts: how should I train a model with transformed parameters

texts:
Lets say that we have the following model:

$y = sigmoid(w1)*x1 + w2*x3 + w3*x3$

and we would like to fit it in a dataset $(xn, yn)$. How can we do this? (e.g using gradient descent? Also, what could be the error function?

Same question for model $y = sigmoid(w11, w12)*x1 + w2*x3 + w3*x3$

Thanks.


texts:
I have a table with many financial information of thousands of companies around the world (revenue, income, number of employees, total assets etc) and an extra information indicating whether that company owns a specific high value machine ("1" if the company has and "0" if the company hasn't).

My goal is to identify which companies don't have this machine, but are potential customers. My first approach was to use a classification algorithm, such as SVM, random Forest etc and use those financial characteristics as predictors of the "has machine"/"hasn't machine" column. The "potential customers" would be, actually, the individuals the model classified as "1" (has machine), but they are actually "0" (hasn't machine). In other words, the false positives of this model.

I am not comfortable with this approach (after all, a perfect model wouldn't indicate a single potential customer!), but I don't know any other way I could approach this problem. I was wondering if someone could give me some direction!

Thanks


texts: NLP - Found input variables with inconsistent numbers samples

texts:
I'm trying to train a model to read the greetings from the sample dataset collected from Tripadvisor and I've been getting the following error when I am trying to train the model sets.
Here's the link to the dataset.
Here's my code:
import streamlit as st
import numpy as np
import pandas as pd

# NLP Pkgs
import matplotlib.pyplot as plt
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import confusion_matrix
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

import os

# Main Stuff

st.title("Greetings NLP - Presence")
st.subheader("Created using Streamlit - Harshil Parikh ")

# Loading the data into streamlit
@st.cache
def load_data(nrows):
    #data = pd.read_csv('/Users/harshilparikh/Desktop/INT/data/selections.csv', nrows=nrows)
    dataset = st.cache(pd.read_csv)('/Users/harshilparikh/Desktop/INT/data/selections.csv')
    return dataset

data_load_state = st.text('Loading data...')
dataset = load_data(1000)
data_load_state.text('Data loaded.')

#Displaying all data first
if st.checkbox('Show Raw data'):
    st.subheader('Raw Data')
st.write(dataset)

# GREETING TAB

st.subheader('Greetings')
greet = st.sidebar.multiselect("Select Greeting", dataset['Greeting'].unique())

select = dataset[(dataset['Greeting'].isin(greet))]
# SEPARATING ONLY TWO COLUMNS FROM THE DATA
greet_select = select[['Greeting','Selected']]
select_check= st.checkbox("Display records with greeting")
if select_check:
    st.write(greet_select)

#Text- Preprocessing  - Range from 0 to 6758 total feedback
st.subheader('Naive Bayes')
nltk.download('stopwords')
corpus = []
for i in range(0, 6758):
    review = re.sub('[^a-zA-Z]', '', str(dataset['Selected'][i]))
review = review.lower()
review = review.split()
ps = PorterStemmer()
review = [ps.stem(word) for word in review if not word in set(stopwords.words('english'))]
review = ''.join(review)
corpus.append(review)

#BAG OF WORDS
cv = CountVectorizer(max_features = 6758)
X = cv.fit_transform(corpus).toarray()
y = dataset.iloc[:, 1].values

#Training sets (800 values)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
#X_train[0, 0:10] #First 10 rows of the first column of X_train.

# NLP - Naive Bayes algorithm

classifier = GaussianNB()
classifier.fit(X_train, y_train)

I tried shaping and I'm getting (1,1) for X and (6758,) for y. I'm trying to learn basic NLP.


texts: parsing a simple sentence using syntaxnet (on Ubuntu)

texts:
could someone help me in with the following. I would appreciate it.

I would like to use syntaxnet to analyze a sentence. I installed Ubuntu on windows following this link.

What do I have to do next to make syntaxnet to produce result. The sentence is for example, "David put a book on shelf".

Thank you in advance.


texts: Linear regression: variables with high P-value

texts:
Often, after training a linear regression model on data, some variables/features would have high P-value, which means they are not statistically significant. Although there are automated methods like variable selection, such as Step-wise, LASSO, etc. I would like to understand the best practice for manually selecting variables.

One thing I can think of is to check for collinearity and perhaps discard or combine variables that shows high correlation. One can also compare models using the F test, or perhaps use domain knowledge. Aside from that, what other best practices are there?

I feel simply discarding variables that shows high P-value is too simplistic. And there is no guarantee that the remaining variables will all have high P-value after you have discarded some variables.


texts: Cross validation while preserving a column (not the target ) distribution

texts:
So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set was generated, a function that sampled it and tried to preserve the ratio for the target classes, and a particular column ) . My local CV is a bit lower than my score on the test set, and i think the problem is stemming from the fact that i'm using stratification only for 'y'.

Can lack of stratification of that feature be the reason of Cv & test scores aren't really close?

And if so how can i perform stratification for the target and a feature!
Thanks

Edit : i'm already doing stratification on the target since my data is imbalanced.


texts: How does batch normalization work for convolutional neural networks

texts:
I am trying to understand how batch normalization (BN) works in CNNs. Suppose I have a feature map tensor $T$ of shape $(N, C, H, W)$
where $N$ is the mini-batch size,
$C$ is the number of channels, and
$H,W$ is the spatial dimension of the tensor.
Then it seems there could a few ways of going about this (where $\gamma, \beta$ are learned parameters for each BN layer)
Method 1: $T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{x,y}} {\sqrt{\sigma^2_{x,y} + \epsilon}} + \beta$ where $\mu_{x,y} = \frac{1}{NC}\sum_{n, c} T_{n,c,x,y}$ is the mean for all channels $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
$\sigma^2_{x,y} = \frac{1}{NC} \sum_{n, c} (T_{n, c,x,y}-\mu_{c})^2$ is the variance of the minibatch for all channels $c$ at spatial location $x,y$.
Method 2: $T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{c,x,y}} {\sqrt{\sigma^2_{c,x,y} + \epsilon}} + \beta$ where $\mu_{c,x,y} = \frac{1}{N}\sum_{n} T_{n,c,x,y}$ is the mean for a specific channel $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
$\sigma^2_{c,x,y} = \frac{1}{N} \sum_{n} (T_{n, c,x,y}-\mu_{c})^2$ is the variance of the minibatch for a channel $c$ at spatial location $x,y$.
Method 3: For each channel $c$ we compute the mean/variance over the entire spatial values for $x,y$ and apply the formula as
$T_{n, c,x,y} := \gamma*\frac {T_{n, c,x,y} - \mu_{c}} {\sqrt{\sigma^2_{c} + \epsilon}} + \beta$, where now $\mu_c = \frac{1}{NHW} \sum_{n,x,y} T_{n,c,x,y}$ and $\sigma^2{_c} = \frac{1}{NHW} \sum_{n,x,y} (T_{n,c,x,y}-\mu_c)^2 $
In practice which of these methods is used (if any) are correct for?
The original paper on batch normalization , https://arxiv.org/pdf/1502.03167.pdf , states on page 5 section 3.2, last paragraph, left side of the page:

For convolutional layers, we additionally want the normalization to
obey the convolutional property – so that different elements of the
same feature map, at different locations, are normalized in the same
way. To achieve this, we jointly normalize all the activations in a
minibatch, over all locations. In Alg. 1, we let $\mathcal{B}$ be the set of all
values in a feature map across both the elements of a mini-batch and
spatial locations – so for a mini-batch of size $m$ and feature maps of
size $p \times q$, we use the effective mini-batch of size $m^\prime = \vert \mathcal{B} \vert = m \cdot pq$. We learn a pair of parameters $\gamma^{(k)}$ and $\beta^{(k)}$ per feature map,
rather than per activation. Alg. 2 is modified similarly, so that
during inference the BN transform applies the same linear
transformation to each activation in a given feature map.

I'm not sure what the authors mean by "per feature map", does this mean per channel?


texts: Does it ever make sense for upper layers to have more nodes than lower layers?

texts:
Usually people construct neural nets with fewer nodes in the higher layers than the lower layers (excluding the input layer, which could sometimes have fewer columns than the lowest hidden layer).

Are there situations where it makes sense to have a diamond-shaped network, instead of a pyramid shaped network?  I suppose that such a network could be viewed as a basis expansion in the hidden units, much as a network with a few inputs and many hidden layers could be used to model nonlinearities in those hidden units.

Computationally, a diamond-shaped network has many fewer weights as well.  This is because thousands of variables become many many weights when each one feeds into several nodes.

Of course, all information from those thousands of variables needs to pass through a few nodes.  This might not be rich enough for some problems.  But to what degree can this be mitigated by just adding lots and lots of layers?

Alternatively consider the question this way:  given that neural nets with one layer and enough nodes are universal approximators, is it true that a neural net with a small, number of nodes per layers, but many layers, will also be a universal approximator?


texts: Feature selection - QR code localization

texts:
I just started learning about machine learning recently and have a project where I have to develop a program for QR code localization so that a QR code can be detected and read at any angle of rotation. Development will be done in Python.

The plan is to gather various images of the QR codes at different angles with different backgrounds. From this I would like to create a dataset for training with neural networks and then testing.

The issue that I'm having is that I can't seem to figure out a correct feature design for the dataset and how to identify the QR code from the images for feature processing.
Would I use ground-truth images or edge magnitude maps?

Any help with this would be amazing?
Thanks for your time.


texts: What is parts of speech technique in sentiment analysis?

texts: In an article, I saw Sentiment Analysis using Parts Of Speech(POS) technique. When I searched I got some paper on POS but I couldn't understand what POS basically is. Though I am new to sentiment analysis please help me to understand POS.


texts: What techniques are used to understand call patterns?

texts:
I have customer data since 2013 and there is a file which has the customer unique id, a timestamp, and the reason for the call (a drop down from the person who handled the call).

I did some cumulative counts based on customer ID and the timestamp and I saw that one customer called in over 1000 times alone. What's the best way to make sense of the call driver data when I'm looking at millions of rows and around 200 categories of call types?

Is there a broader topic which looks into 'downstream'  issues or predicting the probability of future calls or events?  The end goal would be to visualize these calling patterns and focus on reducing the call backs. This is a specific problem but it seems like it should be common and I can learn about addressing it in a bigger picture manner.


texts: Getting array out of bound error while running h2o.predict function

texts:
Getting array out of bound exception while running  h2o.predict function for prediction . All the columns names are exactly same as in the model object.
Let's say df is a data frame I have taken a sample of 80% from it to build a glm model and now trying to predict on the full data that is df


texts: optimal combination of hyper parameters and model selection

texts:
This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc.

When we tune the hyper parameters of a deep learning model every possible combination of hyper parameters results in a different model. And we select an optimal combination based on the loss curves. What exactly is an optimal combination of hyper parameters?

My question is exactly this: There is an infinite number of combinations of hyperparams possible. We know that there are many possible configurations of hyperparams possible that give similar generalization error. What should the model selection decision be based upon?
And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?


texts: How do standardization and normalization impact the coefficients of linear models?

texts:
One benefit of creating a linear model is that you can look at the coefficients the model learns and interpret them. For example, you can see which features have the most predictive power and which do not.
How, if at all, does feature interpretability change if we normalize (scale all features to 0-1) all our features vs. standardizing (subtract mean and divide by the standard deviation) them all before fitting the model.
I have read elsewhere that you 'lose feature interpretability if you normalize your features' but could not find an explanation as to why. If that is true, could you please explain?
Here are two screenshots of the coefficients for two multiple linear regression models I built. It uses Gapminder 2008 data and statistics about each country to predict its fertility rate.
In the first, I scaled features using StandardScaler. In the second, I used MinMaxScaler. The Region_ features are categorical and were one-hot encoded and not scaled.
Not only did the coefficients change based on different scaling, but their ordering (of importance?) did too! Why is this the case? What does it mean?


texts: Identify appropriate variance stabilization transform

texts: I have a heavy tailed data which is heteroscedastic in nature. I need to apply some ML algorithms on this data that are good for normal distribution. So I want to transform the data. I tried Box cox transformation, square root and log transformation but it does not help much and the data still retains its heavy tailed and heteroscedastic nature. What are some of the other variance stabilizing that can be of help here.


texts: Analytical solution for optimization with inequality constraints

texts:
Let the following be known matrices with dimensions as:
$M = nXk$, all elements >0
$w_b = 1Xn$, all elements >0 , sums to 1
$S = nXn$, a positive semidefinite matrix
$C_c = 1Xk$, all elements >0
$C_b = 1Xn$, all elements >0
$1_k = 1Xk$ vector of 1's
I need to solve for $w_c = 1Xk$ in the following:
$Min_{w_c}( (w_cM' - w_b) S(w_cM' - w_b)')$  such that $1_kw_c'  = 1 $ , and $C_cw_c'  = C_b w_b'$ , and $w_c >= 0 $ (all elements of $w_c $ >=0)
I can solve with only the equality constraints with lagrange multipliers
Obj:
$ (w_cM' - w_b) S(w_cM' - w_b)' - \lambda_1 (1_kw_c'  - 1) - \lambda_2(C_cw_c'  - C_b w_b')$
However, I am not sure how to solve with the third constraint, i.e. $w_c >= 0 $. Can this problem be solved analytically? Or do I have to use some numerical routine?


texts:
I do not know if it is possible to somehow connect different datasets, that are related in some way, in Orange 3.
I am working with EEG multiple files (of channel values results) and one main data file "Ratings", consisting of subject number, score in idle mode and overall test score.
The point is that I want to make Test & Score to compare two algorithms, but I do not know how and what to do with all those files, to make it work.
F.E. obe subject from "Score" has two additional different dataset files ("low" and "high", both have same attributes of EEG channels and numeric values). So, how I imagine, I'd need to somehow use those high and low files, but they should be known (or somehow be connected) as "Ratings" dataset attribute "subject" number 1 data.
Sorry for my english but this problem is kinda hard to explain. Maybe I'll post an example of those files:
Main results are stored in "Ratings" dataset and final test score would be the target variable:

Then all of the subjects have two files, f.e. subject 1 has "sub01_hi" and "sub01_lo" with same channels but different values. There are 14 channels just containing values in both of those files. And that goes on for all of the subjects.
sub01_hi:

sub01_lo:

I want to use somehow those two files for evaluation by using different algorithms (like kNN or Bayes) and include the scores from "Ratings" dataset too, so that they'd be like categories or main target variables.
I'm just kinda lost here.. :/


texts: Theano vs Tensorflow for building Neural Networks for NLP tasks

texts: I am trying to learn Theano and TensorFlow for building neural networks for NLP based tasks. Any suggestions as to when one should choose one over the other or what works better and when or is it just a personal choice of what one is comfortable with?


texts: One Hot Encoding for any kind of dataset

texts: How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how to keep track of the new one hot encoded data with the original dataset?


texts:
How can I perform conceptual clustering in sklearn? My use case is that I have English Wikipedia articles that I'm doing unsupervised learning on (tfidf -> truncated svd -> l2 normalize), and I'd like to create a hierarchy for them such that the nodes at the top are the most general articles (e.g. Programming Languages -> Functional Languages -> Haskell).

I tried using hierarchy.linkage, but it seems that the algorithm uses n^2 space, and I ran out of memory. I also tried using a KDTree on the l2 normalized vectors, and then setting each node to be the normalized sum of its children recursively, but this did not produce desirable results.

What is the right way to perform conceptual clustering with cosine similarity in scikit-learn without using quadratic space?


texts: When to split data into multiple regression models instead of one model?

texts:
I'm playing with regression models in scikit-learn. The goal is to predict how much inventory we should purchase for the next 90 days. My data set has hundred of product categories. Each category has many unique features that do not apply to every category.

For Example: Shirt category could have "size" and "color" features where as the category Razors could have a "number of blades" feature.

Should I split my data up by category and make a different model for each? Or is it suffient to have one model in which I keep the products category as one of the features?


texts: How to deal with memory exceptions while processing large amount of data?

texts:
Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps:

Take backup of the file
Extract new records of previous day transactions,append to original/base CSV file and store the data in dataframe.
Perform mathematical operations on the big dataframe
Convert big dataframe to CSV and store in same location for nex day processing.

Similar process runs next day as well and so on...

I'm getting memory exception error while processing the step 3 above (probably step 1 & 2 would have consumed most of the memory max-3GB limited space). Even if I extend the space in the server, I foresee a problem as my input file size is going to be increase daily.

I need to dip through historical data daily for mathematical operations along with daily transactions so can't avoid storing/accessing of base CSV file which contains historical data and can't afford to buy space in cloud as well.

I have used pd.read_csv for reading records from CSV file in chunks, for mathematical operations pandas and numpy has been used.

The script runs without any issue on my local machine, however I have the memory issue while processing on the server. So, as far as the code goes, it seems to be good.

I believe that if I can change the processing in steps 1 to 4 to be more efficient, the memory exception can be solved and with limited space my script can be executed.

Can someone suggest the best way to handle the above steps?


texts: How do I know if my training data set is sufficient?

texts:
If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions.

Is my data set is sufficient?


texts: How to fit the LSTM model correctly with series data

texts:
I have a small data set with the transformed series vector data with the music genre. The goal for me is to use the LSTM model to learn the non-linear pattern each row for genre classification. The LSTM model is one of the most suitable models for doing this task.
So, my original data set look like this:
vec_1,vec_2,...,vec30,genre
  50    30        0    pop
  .
  .
  .

The shape of the training data is (7678,30), so, at first, I just randomly reshape the training data such as (7678,2,15) to fit the input requirement (samples, time steps, features) for the LSTM model.
However, I am confused about the input data, since the time series data I've seen so far, from many tutorials is like:
time,vec_1,vec_2
 0    50    30
 1     .     .
 .     .     .
 .     .     .
 .     .     .

Should I transpose the matrix before I fit in the LSTM model like this?
        0  1  2  3  .  .  .  7678
vec_1  50
vec_2  30
  .
  .
  .
vec_30  0
genre  pop

The shape of this data will be (30,7678), and it looks right to fit in the LSTM model.
If so, when I reshape it to (7678,2,15), does it mean the same before I transpose the matrix? meaning that with 7678 samples, with 2 timesteps, with 15 features.


texts: Ensemble models - neural network input both original data and predictions of other models?

texts: From my understanding in order to improve accuracy with ensemble models you need a wide range of independent ensemble methods. I was wondering whether using the ouput of a random forest model as one of the inputs for a neural network where the other input is the original data and the targets remain the same could improve the model? Why add extra complexity? basically the problem is a multi-dimensional regression problem and although the random forest gets a smaller MSE the neural network is bette at preservering some of the properties of the target labels. Therefore, I was wondering if by putting these two models together I would get a lower MSE while preserving some of the properties. Is it worth a shot? or will it just drastically overfit?


texts: How can I download data from online table quickly?

texts:
I am trying to do download all the marathon results from 1996 to 2016 from the following link. There are more than 400000 entries, but I can only show 1000 results per page.

Is there any way that can let me download all of them quickly?

Thanks in advance!

http://chicago-history.r.mikatiming.de/2015/


texts: Calculation of VC dimension of simple neural network

texts:
Suppose I have a perceptron with one-hidden layer, with the input - one real number $x \in \mathbb{R}$, and the activation function of the output layers - threshold functions:
$$
\theta(x) =
\begin{cases}
0, x \leq 0 \\
1, x > 0
\end{cases}
$$
The hidden layer may contain $k$ units and I would like to calculate the VC dimension of this feed-forward neural network. The VC dimension is defined as a cardinality of the maximal set, that can be shattered by properly adjusting the weights of the neural network.

The threshold functions have a VC dimension of $n+1$, where $n$ is a number of input neurons, because by a plane $n-1$ plane one may split $n$ points in any way. So when considering the results in the first layer, we have a VC dimension of $2$ for each gate, and the total number of points, that can be separated by the activation is $2 k$. Then we have a vector $\in \mathbb{R}^k$ to be processed to output, and the output unit has a dimension $k + 1$.
Do I understand correctly, that the resulting VC dimension of this simple neural network is :
$$
2 k + k + 1 = 3k + 1
$$


texts: Doing feature extraction from name data?

texts: I'm working with a genre prediction application right now, and I was curious about handling name data. I was planning to try to use that in the prediction(as a normal human can usually estimate attributes like country of origin from a song, and based on what words are in the name potentially what genre it is(I.E Anime Songs tend to have common words in them based on the series they're inspired from). I was wondering how I should go about handling the name data.


texts: Interpretable xgboost - Calculate cover feature importance

texts:
When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance.

There are same parameters in the xgb api such as: weight, gain, cover, total_gain and total_cover. I am not quite getting cover.

  ”cover” is the average coverage of splits which use the feature where coverage is defined as the number of samples affected by the split

I am looking for a better definition of cover and perhaps some pseudocode to understand it better.


texts: Generalization error for simple linear regression

texts: Lets say we have a training data and we have estimated a fit for a model of square ft of living area vs price of houses. Suppose we know the probability distribution of sq ft and for a fixed sq ft we know the probability distribution of its price. Now, we know the joint probability for each sq ft and price pair. How do I calculate the generalization error(both total and for a particular sq ft)??


texts: When it is a good idea to train a model also on the test data?

texts: I have a small restricted dataset, it is not very small but accuracy will be much better if I will have more data. I have split it to train and test datasets:  85%/15%. I have chosen NN model and trained it, measured loss, it is ok. Now I should use it in production. Should I train it additionally on the test data, because in such way I will get better accuracy?