texts: classification imbalance data - bias and class weight

texts:
This page shows a classification problem. They have used bias as well as bias along with class weights.

What is difference between bias and weights? In some other techniques such as Random Forest we could adjust cut off from 50% to say 90% so that if the edge node has 90% examples of high density class then only it will predict that class. Out of bias and weights, which one is similar to cut off probablity?
When should I use just one of them or both? I can always run 3 models 1 with bias, 1 with class weight and 1 with both and compare the results. But is there any better refrence?

keras.layers.Dense(1, activation='sigmoid',
                         bias_initializer=output_bias)

and

weighted_history = weighted_model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_features, val_labels),
    # The class weights go here
    class_weight=class_weight)


texts: Machine Learning with fixed and variable inputs and multiple outputs

texts:
I've been doing machine learning and neural networks for a couple months, and finished a 5 months online course of it last year, but I'm struggling in an specific scenario:

Let's say we've got this data: X1, X2, F3, F4 and Y1

X1 and X2 are possible to change.
F3 and F4 change only with time and arbitrary.
Y1 only changes as a result of X1, X2, F3 and F4 changing.

I've collected thousands of samples ready to train, but I don't know what type of algorithm should I use.

This is how I want to use it:

I want to "reach" a desired target value of Y1, setting the combination of values of X1 and X2, respecting the existing and unchangeable values of F3 and F4.

E.g.

[X1, X2] = run_algorithm(Y1, F3, F4);

And what if I also decide to include these X1 and X2 in the algorithm too, so it finds the most efficient way of doing it knowing the already existing values of X1 and X2.

E.g.

[X1, X2] = run_algorithm(Y1, F3, F4, X1, X2);


texts: Thoughts on Feature Engineering of a duration_in_program Variable

texts:
So I am trying to predict which customers would leave a loyalty program sponsored by X firm, using an ML classification model.

I further believe that the duration for which a customer has been in the program affects their likelihood of staying/leaving the program, for reasons such as long-term customers get more loyalty discounts etc.. which may raise the indirect cost/price of them leaving a program.

However, one issue that I am currently facing, is that I am calculating duration_in_program, based on the start_date and end_date for each customer.

However, I think one issue with this approach of coding the variable is that there are values for duration_in_program that don't map to any "stayed" outcomes. Which kind of makes sense. Like, if everyone in the program with a duration_in_program of 3 yrs left the program, then the model will just learn to always predict that as "left".

It is crucial to point that: the program allows customers to stay in the program for a maximum period of five years, after that point, they receive regular prices paid by other cable customers.

Therefore, one way I am thinking of dealing with this is that duration_in_program, is determined by an arbitrary cutoff point. For instance, if they still have not left the program by the date at which the 1st cohort completes the program (i.e. 31/08/2017), then we consider their duration_in_program equal to that. So someone that joined in (i.e. 01/09/2013), and still has not left by 31/08/2017, then we set their duration_in_program=4.

Any thoughts on my approach above?


texts:
when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained it will backpropagate to adjust the weights.
My question is how deep the backpropagation is?

does it happen only till linear layer weights(fully connected neural net) ?
OR does it extend to all the decoder layer weight matrices(Q,K,V) and Feed forward layers weights?
OR does it extend to the even the encoder + decoder weights ?

Please help me with the answer.


texts:
I was trying to print  the index  of each of the maximum probability of the probability
 Array, While Executing the following code I get the error

" The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"

for prob_ in prob:
        max_prob=max(prob_)
        index=clf.classes_[prob.index(max(prob_))]

where the prob is an array


texts:
I need to train 5 models each for 10 times using tensorflow and keras for a homework. 2 of the models are multilayer perceptron models and remaining 3 of them are CNN models. I am training using my GPU, GTX1660Ti. I have no issue training MLP models, but when I try to train the CNN models I get the error below:

I have added these two lines of code to solve this issue:
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

However, when I try to train all of the models in a single script, my process gets killed after succesfully training 3 models:

Training dataset contains 27000 images, test dataset contains 5000 images, and validation dataset contains 3000 images each 28*28. I suspect gpu memory is the issue but I do not know the details. Any help is appreciated.


texts:
I've recently started playing SciKit ML. I just got my hands on classification algorithms (SGDClassifier, LinearSVC) and I'm not sure how to properly represent feature labels.

Suppose I'm trying to predict a football match outcome. Given N features the result is either 1=WIN or X=DRAW or 2=LOSS. Can/should I label my features using string values?

Eg.:

X_train = [[n0, n1, …, nn], …]
y_train = ['1', 'X', '2', …]

Or maybe I should use boolean matrices instead, like [win,draw,loss], eg:

y_train = [[1,0,0], [0,1,0], [0,0,1], …]

Or maybe I should use a single digit to map each label 1=win and 0=draw, -1=loss, eg:

y_train = [1, 0, -1]

Which one would be best to pick and why? I'm a newbie, thank you for your patience.


texts:
When I try to install clogitL1 on my work server I get

Warning message:
package ‘clogitL1’ is not available (for R version 3.5.1)

I get the same warning message when I try to install it using previous versions of R.

At cran I see

Package ‘clogitL1’ was removed from the CRAN repository.
Formerly available versions can be obtained from the archive.
Archived on 2018-06-17 as check problems were not corrected despite reminders.

Questions:
What's going on here?
It seems that this package can no longer be installed?
Is there still a way to install and use it?

I use this package extensively professionally.  We just had our servers upgraded and I would like to keep using this package...


texts: Should I connect my two GPUs with SLI or not? (for Keras + TensorFlow)

texts: I have two GPUs, NVIDIA GTX 1070 Ti. For using Keras with TensorFlow back-end, should I connect them with SLI or not? If not, then they will be treated separately, and one model will be trained on one card. These are the two options from what I understood so far. Thank you.


texts: CNN vs RNN: Classification on 20_newsgroup data

texts:
I am very much a beginner in ML space. I am learning keras to get hands-on experience.
I picked the classification of 20_newsgroups data for my task, I used glove.6B.50d.txt for embeddings.

I chose to train a RNN for the task cos it has the capability to learn time series data better. Trained with different combinations of LSTM layers and output dimensions before converging to one combination. Then I tried CNN model for the same task.

What I found was CNN model could learn faster and gave about 65% accuracy at the end of 15 epochs, where as the RNN model took 50 epochs to get the same validation accuracy even after trying several learning rates. But at the end both models were giving about 65-70% validation accuracy after training for some more duration. I stopped my exploration there.

My question is I assumed/expected RNN to perform better. According to my understanding of RNN, it builds up the memory of time series data. What is it I am missing here ?


texts: How to add more features in addition to a 100D word vector

texts:
I am working on sentiment analysis using tweets text. I am able to build a word vector using Keras text_to_sequence() method with pretrained GloVe embeddings in Python.

The question is how to add extra features beside the created GloVe embeddings and shape it in right way to feed to the embedding layer and the algorithm in general? For example, assuming the embedding's length is 100 and the extra features are 3, will appending the 3 features to 100 vector resulting in 103 vector length is the right approach? If so how would you implement it.

Where:

corpusAsList is the corpus as text in a list.

maxlen is the maximum number of columns to pad the word sequence with, so that all vectors have the same length.

maxVocabulary is the maximum amount of words to be used in the sequence and ignore the rest of the words.

And tokenizer is used to represent a word as a digit in the vocabulary.

For instance, if we called texts_to_sequences() like this:

   # Create tokenized corpuss
    listTokenized = tokenizer.texts_to_sequences(corpusAsList)

    # Sequence dataset
    sequenceMatrixPrediction = sequence.pad_sequences(listTokenized, maxlen=maxlen)

And then trained a Bidirectional LSTM model like this:

def getLSTMModel(maxlen, maxVocabulary, embedSizeLSTM=100, weights=None):

    inp = Input(shape=(maxlen,))
    x = Embedding(maxVocabulary, embedSizeLSTM,
                 weights=[weights], trainable=False)(inp)
    x = Bidirectional(LSTM(80, return_sequences=True))(x)
    x = GlobalMaxPool1D()(x)
    x = Dropout(0.25)(x)
    x = Dense(70, activation="relu")(x)
    x = Dropout(0.25)(x)
    x = Dense(1, activation="sigmoid")(x)
    model = Model(inputs=inp, outputs=x)
    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

return model

How could we add three extra features to this model?


texts: Using sensor data and a know reference point infer the position of a moving robot

texts:
Say, the robot is starting at a known position and I've data coming off of the robot as it traverses the grid layout. Exploiting the nuances captured in the data - like the implication of unequal rpm in the wheels indicating the robot is turning off to the side of slower wheels- I would like to dynamically infer the location of the robot relative to its starting position.

The ideal solution would be as the data is streaming in my model would dynamically infer its location from the data. I'd like to know if there are any existing solutions for this problem or any ideas that can help me get started are appreciated.


texts: how can I solve label shape problem in tensorflow when using one-hot encoding?

texts:
I used tensorflow to recognize text from natural images by using convolutional neural network; there is no specific number of characters in the text. To make a successful training I should convert the categorical labels into binary using one-hot encoding. So, for each label, I used integer encoding for each character and stored them in one numpy array in order to create TFRecords. For example:

alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ '
TrainLabel  = ["CNN in Tensorflow"]

# define a mapping of chars to integers
char_to_int = dict((c, i) for i, c in enumerate(alphabet))
integer_encoded = [char_to_int[char] for char in TrainLabel[0]]

if (len(TrainLabel[0])) < 51:
    for j in xrange(51- (len(TrainLabel[0]))):
        integer_encoded.append(52)

# one hot encode
onehot_encoded = []
for value in integer_encoded:
    letter = [0 for _ in range(len(alphabet))]
    letter[value] = 1
    onehot_encoded.append(letter)

label = np.array(onehot_encoded, np.float32)

51 is the maximum number of character in the text, so if the text has less than 51 characters, pad it to 51 characters with spaces.

If we print the label, it will be like this::

array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ...,
       [ 0.,  0.,  0., ...,  0.,  0.,  1.],
       [ 0.,  0.,  0., ...,  0.,  0.,  1.],
       [ 0.,  0.,  0., ...,  0.,  0.,  1.]], dtype=float32)

after creating the batch queue, the label has shape [batch_size, 2703]. 2703 is come from 51*53 which 53 is the number of classes

My problem is in loss function:: the label shape in tf.nn.sparse_softmax_cross_entropy_with_logits() must be [batch_size], but the label that I used here has this shape [batch_size, 53] because I used one-hot encoding?

How can I deal with that??

This is the problem::

  (labels_static_shape.ndims, logits.get_shape().ndims)) ValueError:
  Rank mismatch: Rank of labels (received 2) should equal rank of logits
  minus 1 (received 2).


texts: Is max_depth in scikit the equivalent of pruning in decision trees?

texts:
I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I prune a decision tree using scikit?

dt_ap = tree.DecisionTreeClassifier(random_state=1, max_depth=13)
boosted_dt = AdaBoostClassifier(dt_ap, random_state=1)
boosted_dt.fit(X_train, Y_train)


texts: First Differencing to remove seasonality and trends

texts: I am trying to remove seasonality and trends from my time series data. I found this post that said to use df_diff = df.diff().diff(12).dropna() (https://www.tobiolabode.com/blog/2020/12/30/how-to-convert-non-stationary-data-into-stationary-for-arima-model-with-python). I don't understand why we need to use diff(12) from pandas. Could someone explain this to me? Thanks!


texts: How can I get testing accuracy using tensorboard for Detectron2?

texts:
I'm learning to use Detecron2. I've followed this link to create a custom object detector.
My training code -

# training Detectron2
from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
import os

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.DATASETS.TRAIN = ("pedestrian",)
cfg.DATASETS.TEST = ()   # no metrics implemented for this dataset
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"  # initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.02
cfg.SOLVER.MAX_ITER = 300    # 300 iterations seems good enough, but you can certainly train longer
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

It saves a log file in output dir thus I can use tensorboard to show the training accuracy -

%load_ext tensorboard
%tensorboard --logdir output

It works fine and I can see my model's training accuracy. But When testing/validating the model -

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set the testing threshold for this model
cfg.DATASETS.TEST = ("pedestrian_day", )
predictor = DefaultPredictor(cfg)

Although from Detectron2 tutorial I've got -

from detectron2.evaluation import COCOEvaluator, inference_on_dataset
from detectron2.data import build_detection_test_loader
evaluator = COCOEvaluator("pedestrian_day", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "pedestrian_day", mapper=None)
inference_on_dataset(trainer.model, val_loader, evaluator)

but this gives the AP, AP50, AP75, APm, APl and APs for both training and testing.
My question is how can I able to see the testing accuracy in tensorboard like the training one?

NB: I've already asked this question on statckoverflow but since I get no response I asked the same question here.


texts: Can a neural network be of variable depth?

texts:
It is very common for neural networks to be asymmetric about the x axis, that is, to have many more nuerons in the first few layers than in the last few layers. Common example:

But can neural networks be structured in such a way that they're asymmetric about the y axis, that is, such that some parts of the network are 'deeper' than others?
An example of where this could be useful is where a problem involves some unsophisticated features (that require only a small number of layers), and some much more sophisticated features (which may benefit greatly from having more layers - i.e. greater 'depth').


texts:
I am new to time series and therefore trying to feel my way around. I am required to do some analysis and possibly come up with a model. However, I have at most 2 years worth of quarterly data (at most 8 datapoints). Can I build any meaningful model based on the data? What possible model apart from linear regression with time as regressor can be built?

Thanks


texts: Does scikit-learn use regularization by default?

texts:
I just fitted a logistic curve to some fake data. I made the data essentially a step function.

data = -------------++++++++++++++

But when I look at the fitted curve, the slope is very small. The function that best minimizes the cost function, assuming cross entropy, is the step function. Why does it not look like a step function? Is there some regularization, L1 or L2, done by default?


texts: Why is eulers number used as a constant in sigmoid

texts:
I was asking myself why eulers number was used in the sigmoid function 1/(1+e^-x) instead of any other constant like for example 2 or 3?

I am pretty new to data science stuff, but I read somerwhere that eulers number is the natural growth of a curve, so would this mean, that eulers number is used in the sigmoid function because it makes it possible to output values that are evenly distributed between the values 0 and 1?


texts: How to create a neural network with many different values in Matlab?

texts:
I want to create a neural network in Matlab.

I have 16 import matrices in each iteration,  my import data changes in every iteration and there are more than 100000 iterations. The target is a matrix like [x y z] and is sum of imported data.

Do you have any suggestion for creating this kind of neural network?
any idea where should I start?

Thanks


texts: LSTM Time-series classification - derived feature

texts:
I have a time-series dataset and I want to derive a new feature based on a date column which I believe might improve my predictive model. The feature is if it's weekday or weekend.

I am not sure how to create this for modeling purposes. Do I represent it as a boolean function or a numerical one (1/0) or just add 2 new features (weekday, weekend) and represent them as 1 and nan?


texts: Tensorflow Object Detection API, update performance measurement

texts:
I am using Tensorflow Object Detection API to classify objects in the images. I am using own dataset. I am facing the problem of performance. I need a suggestion about the training strategy.

I have updated cocoapi\PythonAPI\pycocotools\cocoeval.py file to check mAP per class. After line 466 I have added these lines:

             if ap == 1:
                avg_ap = 0.0
                class_count = 0
                my_class_ids = p.catIds
                num_classes = len(my_class_ids)

                for i in range(0, num_classes):
                    my_mean = np.mean(s[:, :, i, :])
                    if my_mean > -1:
                        print('{}. category# {}: {}'.format(i, my_class_ids [i], my_mean))
                        avg_ap += my_mean
                        class_count += 1
                if class_count > 0:
                    print("Class average#{} mAP : {}".format(class_count, avg_ap / class_count))

The output looks like:

 11. category# 19: 0.183361433778217
 13. category# 21: 0.02725855003082726
 16. category# 24: 0.4594024961926443

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.202

Here training data have examples: category# 19: 581, category# 21: 144 and category# 24: 453. So, data is a little bit skewed. I have used a pre-trained network trained with COCO data.

The network is learning, the loss is going down and mAP is 0.202. However, some object classes have very low mAP like 0.02 shown above.

How can I update my training strategy to improve performance measurement for those tough objects?


texts:
I am referring to the following implementation of node2vec: https://github.com/eliorc/node2vec
Is there a way to have a "smart initialization" with node2vec, i.e., to start the algorithm with an embedding that is not random but precomputed in a certain way? For instance, when I use the spring_layout from NetworkX, I pass the initialization via pos = my_initialization, and the computations are a lot faster. Obviously, there is no such argument with node2vec, but I was wondering if there is some workaround via underlying word2vec.


texts: What is the correct formula for Jaccard coefficient with integer vectors?

texts:
I understand the Jaccard index is the number of elements in common divided by the total number of distinct elements. But it seems to be some discrepancy or terminology confusion about Jaccard being applied to "binary vectors", meaning a vector with binary attributes (0, 1), or, "integer vectors", meaning any vector with integer values (2, 5, 6, 8).
There are two formulas depending on the type of elements in the vector?
This answer comments about "binary vectors" which "they can be interpreted as sets of indices with value 1". However there are examples where Jaccard Coefficient is calculated with an integer vectors, so it seems to be valid. Besides, scikit-learn seems to define 3 cases:
Binary vectors
y_true = np.array([[0, 1, 1],
                   [1, 1, 0]])
y_pred = np.array([[1, 1, 1],
                   [1, 0, 0]])

Multilabel cases
(what is a Multilabel case is not defined in the scikit-learn documentation)
"Multiclass problems are binarized and treated like the corresponding multilabel problem"
y_pred = [0, 2, 1, 2]
y_true = [0, 1, 2, 2]

An additional test with a R library which uses the equation form

TP / (TP + FP + FN)

results in an undefined behavior:
library("ClusterR")
pc <- c(0, 1, 2, 5, 6, 8, 9)
tc <- c(0, 2, 3, 4, 5, 7, 9)
external_validation(pc, tc, method = "jaccard_index")
[1] NaN

Is using the set based formula only suitable for binary vectors?
$$
J(A,B) = {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}}
$$


texts: How to implement a Multivariate multi-site application in LSTM?

texts:
I am trying to make a multivariate multi-site classification LSTM model using Keras. I have followed this tutorial from Jason Brownlee: https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/.
This method removes the first few observations to create lags, which is not desired in my case.
I would like to change it to a one to one sequence model where it uses all previous observations to predict the classes instead of a fixed time frame. This is shown in Jason Brownlee's book
as shown here.
I have multiple ID's that each have their own sequence, with variable timesteps for each ID. I would like to predict the class of the next observation using all previous observations, and then do this for the whole sequence for each ID. I have made an example of the data in the table below. I would like to predict Class of row 2 using row 1. Then predict class of row 3 using row 1+2. Then predict class of row 4 using row 1+2+3. Then there is ID 2, so a new sequence, where I would like to do the same thing.
I couldn't find any examples on the topic. Does someone have example code how to:

Structure the data in a 3 dimensional way to use it for multivariate multi-class multi-site LSTM.
How to deal with uneven sequences (Probably padding)
How to implement this in the LSTM network

ID
Date
feature1
feature2
Class

1
1
0.6
0.2
1

1
2
0.2
0.7
2

1
3
0.4
0.1
0

1
4
0.2
0.9
1

2
1
0.3
1
0

2
2
0.2
0.1
0

2
3
0.3
0
1

3
1
0.1
0.2
2

3
2
0.3
0.1
1


texts: Shouldn't L2 regularization be normalized for the number of nodes in a layer?

texts:
I'm naively thinking it would be better to normalize L2 regularization to the number of elements in a tensor, but I don't see anyone doing that. What am I missing?

I'd say in a multi layer fully connected NN:

x  = input            # vector length n_in
w1 = weights          # shape n_in x n_1
b_1 = bias            # shape n_1
h_1 = Relu(w1 * x + b_1)

w2 = weights          # shape n_1 x n_2
b_2 = bias            # shape n_2
h_2 = Relu(w2 * h_1 + b_2)

Then the L2 loss is normally defined as:

L2 = lambda * ( sum((w1) ^2) + sum((w2) ^2) )

If the first layer is much bigger than the second layer, than the first term in above L2 definition is much bigger, effectively making the second term useless. I'm thinking this would be better:

L2 = lambda * ( sum((w1) ^2)/(n_in*n_1) + sum((w2) ^2)/(n_1*n_2) )

This would also make the hyper parameter lambda independent of the number of nodes you choose. However, I don't see anybody doing this so I suspect I'm missing something. What is it?


texts: Unblanced classes: classifier only predict one class

texts:
I am trying to use a multiclass classification using python. For that I used few algorthims like Random Forest, Xgboost, Logitic regression.

My problem is simple, I have users, Images, and people ratings on those images. I devided the ratings into 3 classes:

class 1: good marks

class 2: bad marks

class 3: medium evaluations

At first I got these results

  I have 70% of bad marks (class2) 18% of good marks (class1) 10% of bad
  marks (class 3)

so  all was good I did the classification and I got a good accuracy (75%).

I have collected more data and more (good marks) and the accuracy kept decreasing.

So I understood that the accuracy was good only because the algorithms where predicting that almost all the marks are bad, so basically only one class- class 2, and when I got more data in class 1 and class 3 the accuracy decreased.

This is example of the confusion matrix I was having

Predicted Marks     1    2  3
Actual Marks
1                  48   85  3
2                  17  250  4
3                  10   89  1

I understood that I was getting this problem because I have unbalanced data, so I was predicting bascially the highest frequency class.

I did some researches and I found that there are option called `

  class_weight='balanced'
  So I used it in the classification algorithms.

And I got this result

Predicted Marks   1    2   3
Actual Marks
1                  53   61  29
2                  66  161  53
3                  24   40  17

So my question is:

First I know that the option balanced try to  requilibrate the data
but I don't understand how. I found in the explanations that it"considers each class as important as the other"` but still don't understand how.
Does it duplicate rows of the minor categories ?
When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones.
( example in the second confusion matrix the class 1 was predicted as class 2 66 times , but in the confusion matrix 1 only 17 times)

Am I analysing the problem right? and what are your suggestions for such a problem?

Sorry for making this long, any help will be appreciated !!


texts: What's the best approach of finding target groups of a product based on it's options?

texts: I have a dataset with booked options of a product and want to find out the specific target groups. For example 'A young family with high income buys the options x,y,z more likely'. In my opinion this should be an unsupervised learning problem. The features are both numeric (e.g. age, income) and categorical (e.g. gender). I don't have a clear idea how to start / proceed.


texts: Tensorflow - uninitialized variable under tf.cond(...)

texts:
I have an problem with tensorflow. I created a class for fully-connected layer with parameter in its constructor is_hidden_layer by which I decide to use or not to use relu and batch normalization on layer output. Code goes like this:

class FullLayer(object):
  def __init__(self, neurons, is_hidden_layer):
    self.is_hidden_layer = is_hidden_layer
    self.neurons = neurons

  def build(self, input):
    # ... uninteresting initialization code ...
    W = tf.Variable(tf.truncated_normal([input_shape, self.neurons], stddev=0.1))
    b = tf.Variable(tf.truncated_normal([self.neurons], stddev=0.1))
    hidden = tf.Variable(tf.constant(self.is_hidden_layer), name="is_hidden")
    output = tf.matmul(input, W) + b
    output = self.batch_norm(output)
    return tf.cond(tf.equal(hidden, True), lambda: tf.nn.relu(output), lambda: output)

  def batch_norm(self, input):
    with tf.variable_scope('batch_normalize'):
      mean = tf.Variable(tf.constant(0.0, shape=[input.shape[-1]]),
                        trainable=True)
      # ... not interesting batch normalization code ...

This code works fine. But if change last two lines of build to this one:

return tf.cond(tf.equal(hidden, True), lambda: tf.nn.relu(self.batch_norm(output)), lambda: output)

(I put batch_norm under cond) I get an error:

  FailedPreconditionError (see above for traceback): Attempting to use
  uninitialized value fc1/is_hidden      [[Node: fc1/is_hidden/read =
  IdentityT=DT_BOOL, _class=["loc:@fc1/is_hidden"],
  _device="/job:localhost/replica:0/task:0/cpu:0"]]

Exception is thrown here:

with tf.Session() as session:
  session.run(tf.global_variables_initializer())

When I replace hidden by hidden.initialized_value() I get the same error for variables in batch_norm method. I would make the same change there as well, but other side effects take place, e.g. global_step variable (variable I pass to AdamOptimizer to count steps) is not incremented and remains equal to 0 all the time. This is the thing I do not understand completely.

What is a reason behind this?


texts: Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?

texts:
So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras.

But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch Normalized Recurrent Neural Networks (Laurent, 2015). What's the difference between those three?

There is this Related Work section I don't understand:

  Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the recurrent batch normalization layer to 0.1 makes significant difference in the final performance of the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In weight normalization, instead of the variance, the L2 norm of the incoming weights is used to normalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is not a re-parameterization of the original neural network. The layer normalized model, thus, has different invariance properties than the other methods, that we will study in the following section


texts: Which optimization technique to use with noisy and slow black-box system?

texts:
I have a complex physical system that depends on many continuous input parameters (let's say 10 important ones) and produces an output that I can boil down to one continuous figure of merit. I want to optimize this system and find the set of parameters that produces the best output.

The challenges are the following:

The system is sufficiently complicated that I cannot predict the behavior well enough to build a useful model. I have to treat it as a black box and do all tests with the actual system.
The input parameters are generally correlated. So, optimizing the parameters individually is not enough.
The system is slow to react. When changing the parameters it takes some time until an equilibrium state is reached again, depending also on the magnitude of the change.
The system is noisy. Even when not touching the system the output figure of merit will fluctuate strongly. I want to optimize the long-term average of the figure of merit, but in order to get a good estimate of that I have to measure for some time (many minutes if I wanted to bring the noise down to a negligible level).

On the other hand, the system is relatively well behaved when it comes to the shape of the function to optimize:

I know that I'm already close to the global optimum and there are probably no local optima to get stuck in.
I expect no sudden changes of the figure of merit. The gradients vary by not more than maybe two orders of magnitude over the permissible parameter space.

Question: What is a good technique to apply to this problem? Naïve approaches like a simple grid search are prohibitively time-intensive.

It seems like there will be a trade-off between long averaging and dealing with noisy data. So I would either need an algorithm that makes as few queries as possible, or one that's very robust to noise.


texts: Avoid saturation in activation function (sigmoid)

texts:
Background
It is said Sigmoid/Tanh are not to use because of the saturation issue.

Question
The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -1 and 1, for instance using Batch Normalization (BN), avoid the issue?
Please help understand if saturation of sigmoid/tanh is still the issue when such normalization is used.

If the issue is avoided, then is tanh still a valid activation which can avoid the dead ReLU issue?


texts: How data are prepared during training, testing and in production?

texts:
Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we also standardize/normalize feature columns before feeding it to train an model.
Before modelling  we also split our dataset to training and testing sets.
My first query is how do we do feature engineering in this splitted dataset?
Do we use a global mean of the unsplitted features  to replace the missing value of those features in both training and testing set or should we use local means of those sets?
Like the above question how do we do normalization to a train, test dataset?
The last but an important question, in productions we mostly get feature values one at a time (think a row of features), how do we feature engineer such data rows?


texts:
I'm using spikeslab for the first time, but don't understand what the output means. It was suggested to me that I use it to tell which variables my dependent variable is most correlated to, in a ranked order.
Particuarly, what is "bma" bma.scale" "gnet" and "gnet.scale"? I also don't understand how to read the corresponding plot to the model.Thanks for any help!

For example, this is one of the models I created using spikeslab, with its output:

model2_ss <-spikeslab(Risk_Pct ~ Race
+                    +hispanic
+                    +Born_In_US
+                    +Highest_Education
+                    +Marital_Status
+                    , na.rm = TRUE, data = LabeledData)
> model2_ss
-------------------------------------------------------------------
Variable selection method     : AIC
Big p small n                 : FALSE
Screen variables              : FALSE
Fast processing               : TRUE
Sample size                   : 26
No. predictors                : 5
No. burn-in values            : 500
No. sampled values            : 500
Estimated mse                 : 0.4299
Model size                    : 3

---> Top variables:
                bma      gnet     bma.scale     gnet.scale
Marital_Status  0.516     0.516     0.319         0.319
Born_In_US      -0.469    -0.447    -0.440       -0.419
Race            0.458     0.421     0.926         0.851


texts: Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?


texts: Train neural network for regression with negative samples

texts:
I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?


texts: The SMOTE could only be performed on the training data, so how can we do it using Weka? It means we have to put the training and test data in two separate files and run the SMOTE on the training file, so how can we load two datasets to Weka and perform these steps?


texts: Turning R scripts into a "data science tool"

texts: I have R scripts that process a client's datasets (make figures, summary tables, etc). I'd like to package these scripts into a standalone program, aka data science tool, that I can give to the client and that requires no interaction with code on their part. Ideally they will open the tool, load data, click run, and be given reports. How can I accomplish this?


texts:
def preprocess_features(X):
    output = pd.DataFrame(index = X.index)
    for col , col_data in X.iteritems():
        if col_data.dtype == object :
            col_data = pd.get_dummies(col_data , prefix = col)
        output = output.join(col_data)
    return output
X_a = preprocess_features(X_all)

X_train , X_test , Y_train , Y_test = train_test_split(X_a , Y_all , test_size = 0.15, random_state = 2)
#print(X_train.shape,X_test.shape,Y_train.shape,Y_test.shape)
clf = LogisticRegression()
clf.fit(X_train , Y_train)

When I run this code, I get the following error:
How do I resolve it?

  ValueError: Input contains NaN, infinity or a value too large for dtype('float64').


texts: Benefits of stochastic gradient descent besides speed/overhead and their optimization

texts:
Say I am training a neural network and can fit all my data into memory. Are there any benefits to using mini batches with SGD in this case? Or is batch training with the full gradient always superior when possible?

Also, it seems like many of the more modern optimization algorithms (RMSProp, Adam, etc.) were designed with SGD in mind. Are these methods still superior to standard gradient descent (with momentum) with the full gradient available?


texts: Test data also being processed in batches

texts:
# After loading input(features) and output(label) datasets, ...
EPOCHS = 1200

BATCH_SIZE = 5
batch_size = tf.placeholder(tf.int64)
n_batches = int(len(X_transformed)/BATCH_SIZE)

a, b = tf.placeholder(tf.float32, shape=[None, len(X_transformed[0])]), tf.placeholder(tf.float32, shape=[None, 8])

dataset = tf.data.Dataset.from_tensor_slices((a, b)).batch(BATCH_SIZE).repeat()  # creating dataset for iterating
iter = dataset.make_initializable_iterator()
features, labels = iter.get_next()
print(features)
print(labels)
# Creating an initializable iterator
# saver = tf.train.Saver({"weights": weights})
# --------------------------------------------------------------------------

prob = tf.placeholder_with_default(1.0, shape=())  # default prob = 1 for
dropout so that we can test
# layer = tf.nn.dropout(layer, prob)
regularizer = tf.contrib.layers.l2_regularizer(0.001)
hidden1 = tf.layers.dense(inputs=features, units=len(X_transformed[0]),
name='first', activation=tf.nn.relu,
                      kernel_regularizer=regularizer)
dropout = tf.layers.dropout(inputs=hidden1, rate=prob)

net = tf.layers.dense(hidden4, 40, name='net1', activation=tf.nn.relu,
kernel_regularizer=regularizer)

prediction = tf.layers.dense(net, 8, activation=None,
kernel_regularizer=regularizer)
# regularizer = tf.nn.l2_loss(weights)
# loss = tf.reduce_mean(loss + beta * regularizer)
loss = tf.losses.mean_squared_error(prediction, labels) +
tf.losses.get_regularization_loss()

test_loss = tf.losses.mean_squared_error(prediction, labels)
train_op = tf.train.AdamOptimizer().minimize(loss)

# --------------------------------------------------------------------------

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    sess.run(iter.initializer, feed_dict={a: train_data[0], b: train_data[1], batch_size: BATCH_SIZE, prob: 0.4})
    print('Training...')
    for i in range(EPOCHS):
        tot_loss = 0
        for j in range(n_batches):
            _, loss_value = sess.run([train_op, loss])
            tot_loss += loss_value
        print("Iteration: {}, Loss: {:.2f}".format(i, tot_loss/n_batches))

# ----------------------------------------------------------------------
    # initialise iterator with test data
    dataset_test = tf.data.Dataset.from_tensor_slices((a, b)).batch(1).repeat()
    # creating dataset for iterating
    iter_test = dataset_test.make_initializable_iterator()
    sess.run(iter_test.initializer, feed_dict={a: test_data[0], b: test_data[1], batch_size: BATCH_SIZE})
    for i in range(len(test_data)):                        # put batch_size: test_data[0].shape[0] again in above line
        # creating dataset for iterating test data
        plt.scatter(test_data[1][i][:4], test_data[1][i][4:8])  # plot actual test image coordinates
        # plt.show()
        print('Test Loss: {:4f}'.format(sess.run(test_loss)))
        _, predicted = sess.run([train_op, prediction])
        print(predicted, len(predicted[:4]), len(predicted[4:8]))
        plt.scatter(predicted[0][:4], predicted[0][4: 8])  # plot predicted coordinates
        plt.show()

I think that I have created a new iterator after the 3rd (#-------) in the above code for test data that takes batches of size 1 but I still get batches of size 5. I want to plot test data values and predicted test data values one by one.


texts: I would like to manually define connections in neural network between layers using TensorFlow and keras with Python. By default connections are beween all pairs of neurons. I need to make connections as in picture below.


texts: What is the effect of oversampling on logistic regression?

texts:
I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign conversion rate is around 10%. Thus, to handle this imbalanced data set and to capture more info I took a stratified sample from this data(whole 5 campaigns) such that there are 50% of sample with negatives and 50% positives. Thus, I am oversampling my positives.
My doubt is if I use logistic regression where it estimates coefficients using maximum log likelihood. Will this oversampling will generate bias results?
Also, I think this oversampling won't create any problem with random forest?


texts: I would like to know if there is a metric used to compute the similarity between two scatter plots?


texts: How to form and minimise custom features for classification in supervised learning

texts:
I am having an issue in understanding how to form the features based on particular math formula, and how to adjust the weights with.
The aim is to draw ellipses for each unique category of points. Ellipse keeps inside as much of possible selected type of dots, and as little as possible other types (in the picture attached selected yellow ones).
Here is the a formula, with 5 variables, based on which we can draw any ellipse.  https://www.desmos.com/calculator/65hlgysuy7

I can make a model and calculate based on the data set whether the particular point is within the ellipse (score less than 0) or outside, and create accuracy based loss function, or just to use cross entropy for the loss function. Then i could find derivative for cross entropy, but then what's next? How basically i could adjust the weights in such case?

The the big question how i could manipulate those variables (in the link a,b,c,l and h) to form different models, and optimize them with gradient descent (if thats the appropriate technique for such task)?

Can someone please explain me the logic of how custom features in such case can be formed and weights found with gradient descent?


texts: Python & Pandas : (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' (timed out)")

texts:
I'm writing a Python Script to store JSON data into MySQL Database. I used pandas to store into MySQL Database.

I used two different modules (MySQLdb and sqlalchemy) to connect to MySQL dtaabase.

Python Code:

user@sys:~$ cat pandas_to_mysql.py
import pandas
from pandas.io import sql
from sqlalchemy import create_engine
import MySQLdb

json_string_final='[{"id": 772, "name": "abcd"}]'
df=pandas.DataFrame(eval(json_string_final))
print(df)

#----Using MySQLdb---
con=MySQLdb.connect(host='localhost',user='user1',passwd='pwd',db='myDB')
try:
    df.to_sql(con=con, name='myTable1', if_exists='replace')
except Exception as error1:
    print("Error with connection 1 ------> ")
    print(error1)

#----Using Engine----
engine = create_engine("mysql+pymysql://{user}:{pw}@localhost/{db}".format(user="user1",pw="pwd",db="myDB"))
try:
    df.to_sql(con=engine, name='myTable2', if_exists='replace')
except Exception as error1:
    print("Error with connection 2 ----->")
    print(error1)

I used two diffrent connections to store the data into MySQL dtabase. But connection was not established. Below is the error message.

Output:

user@sys:~$ python3 pandas_to_mysql.py
    id  name
0  772  abcd
Error with connection 1 ------>
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': not all arguments converted during bytes formatting
Error with connection 2 ----->
(pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' (timed out)")
(Background on this error at: http://sqlalche.me/e/e3q8)

But i'm able to connect to MySQL database manually from the same terminal using the following command.

MySQL Connection:

user@sys:~$ mysql -u user1 -h localhost -D myDB -p

It means, there is no issues with host, user, password, database name. Then what is the issue?


texts: Multiple time-series predictions with Random Forests (in Python)

texts:
I am interested in time-series forecasting with RandomForest. The basic approach is to use a rolling window and use the data points within the window as features for the RandomForest regression, where we regress the next values after the window on the values within the window. Just plain autoregressive model (with lags), but with Random Forest instead of linear regression.
Problem:
If I have more than one time-series (multiple time-series), how to pass them in RF regression?
For example: given two time series $y_1(t)$ and $y_2(t)$, the outcome time series is $z(t)$ and I am interested in predicting the values of $z(t)$ based on the combination of $y_1$ and $y_2$.
What I need, is to use rolling window for each $y_1$ and $y_2$, and then feed these values within the window from both time-series into RF regression, to predict the value of $z(t)$.
Question:
How do I incorporate the data from both rolling windows into the input for RF regression?


texts:
I am working on a litigation support application using the Enron corpus, which contains about 600,000 unique text documents.

In litigation, one is often concerned with whether a document is responsive or non-responsive. One produces responsive documents to the opposing side, unless they are privileged (e.g., attorney client communication).

Here, I have sample sets of over 200 responsive and non-responsive documents.

The challenge here is that the topic of the responsive documents is about one thing, whereas the non-responsive documents could be about any number of topics,  ranging from spam, to soccer practice, to business documents, etc.  The non-responsive is what I would call diluted.

I don't know what those non-responsive classes are up front, and there is no purpose or value to breaking them out up front. If a document is non-responsive, then it needs to be quickly (and cheaply) dismissed.

If the samples are random when the classification is applied to the corpus my customers expect the split between corpus responsive and non-responsive to be close to the split of the sample.

What is the best approach to classify responsive versus non-responsive in this situation?

Below I briefly describe what I have tried. If there is a better approach, please share.

Using tf-idf, create an average vector for each document class (responsive and non-responsive).
Take the first 1000 terms (sorted by weight), such that each vector is the same length.
Normalized the vectors.
Note: these two vectors are only a 0.03 cosine similarity to each other.
For each document in the responsive sample set, calculate the cosine similarity to the single average vector.
Note: the average is 0.06.  It is a small number, as documents in the sample set have around 40 terms, as compared to an average vector of 1000.
In the non-responsive perform the same analysis (compare each non-responsive sample document to the non-responsive average vector).
In the non-responsive that same comparison is 0.03. Basically, the non-responsive average vector is diluted.

Based on this approach, I cannot really conclude if document has a higher cosine similarity to responsive compared to non-responsive then it is responsive.  By that measure 1/3 of the documents in the non-responsive sample set would be classified as responsive.

The non-responsive needs to be handicapped.  In this approach, how do I handicap it?  Is there another approach that would accomplish the same thing?


texts: Why some papers write false positive rate per case instead of percentage rate?

texts:
In some published works, especially in medical image analysis, instead of writing FP rate as percentage, they write it per case, for example, FP: 128.52 [/case].
What is the meaning of this? Is it different from percentage rate? How to calculate it per case?


texts: Creating a composite score from dataset with no target variable

texts:
I have a dataset that includes 6 variables about prospective sales opportunities (probability of closing, days until expected close, age of opportunity, etc.). 2 of the columns are categorical and 4 are continuous. I am looking to create a composite score of each opportunity based on these variables.

I have tried manually assigning scores to each column based on arbitrary ranges, but the composite scores ended up being too similar. I want to optimize the scoring for each variable. Is there a way to do this through machine learning? I want to find out which variables are the most important, or "stand out" the most from the rest of the pack. I also would like to see if any variables are redundant.


texts:
I created a multi-scale CNN in python keras. The network architecture is similar to the diagram. Here, same image is fed to 3 CNN's with different architectures. The weights are NOT shared.

I coded the following multiscale CNN in keras which loosely resembles the architecture in the diagram. But I keep getting "Out of memory ERROR" even when the train_dir has 2 images. Would appreciate help...

#main CNN model - CNN1
main_model = Sequential()
main_model.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(32, 3, 3))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(64, 3, 3))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2))) # the main_model so far outputs 3D feature maps (height, width, features)

main_model.add(Flatten())

#lower features model - CNN2
lower_model1 = Sequential()
lower_model1.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model1.add(Activation('relu'))
lower_model1.add(MaxPooling2D(pool_size=(2, 2)))
lower_model1.add(Flatten())

#lower features model - CNN3
lower_model2 = Sequential()
lower_model2.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model2.add(Activation('relu'))
lower_model2.add(MaxPooling2D(pool_size=(2, 2)))
lower_model2.add(Flatten())

#merged model
merged_model = Merge([main_model, lower_model1, lower_model2], mode='concat')

final_model = Sequential()
final_model.add(merged_model)
final_model.add(Dense(64))
final_model.add(Activation('relu'))
final_model.add(Dropout(0.5))
final_model.add(Dense(1))
final_model.add(Activation('sigmoid'))
final_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

print 'About to start training merged CNN'
train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(224, 224), batch_size=32, class_mode='binary')

test_datagen = ImageDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(args.test_images, target_size=(224, 224), batch_size=32, class_mode='binary')

final_train_generator = zip(train_generator, train_generator, train_generator)
final_test_generator  = zip(test_generator, test_generator, test_generator)
final_model.fit_generator(final_train_generator, samples_per_epoch=nb_train_samples, nb_epoch=nb_epoch, validation_data=final_test_generator, nb_val_samples=nb_validation_samples)

UPDATE 1: Providing more system info

I am using Theano-0.9.0.dev5 | Keras-1.2.1 | Python 2.7.12 | OSX Sierra 10.12.3 (16D32) | Macbook Pro 16GB RAM | CPU mode

~/.keras/keras.json contents

{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float64",
    "backend": "theano"
}

Dont have .theanorc file

Please note individual CNN models are training fine. Only the merged code above causes issues.

UPDATE 2: on 27th January, 2017. Tried the following -

Reduced the no. of parameters of the CNN from 53 million to 100K. But still no use. The network eventually fails to train due to memory issues.
Reduced the batch size of images training to 8. The network training fails with the same reason.

No working solution at the time of writing this update...


texts: Why using a frozen embedding layer in an LSTM model

texts:
I'm studying this LSTM mode:
https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.

As you can see here:

embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)

The embedding layer is frozen which means that the weights are not changed during training.

Why is this done?


texts: Does resizing images during training affect the bounding box annotations?

texts: I am using tensorflow object detection API to train my own custom dataset and I am preparing annotations for the same. I see from the config file of my pre trained ssd inception net, the size of the image is reduced to 300 x 300 during training. My doubt is whether the resize would now change the position of my object according to annotation? I mean now the xmin, ymin width and height of the bouding box would be different since it resized. should i annotate on the resized images(resize them myslef before training?) or original one that i give to training?


texts: Why does NAG cause unstable validation loss?

texts:
I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD makes a huge difference in the optimization process.

When I use vanilla SGD, optimization is really smooth. Training and validation decrease at a similar rate, and seem to converge properly past a sufficiently large number of epochs:

However, when I switch to NAG, without changing any other parameters, suddenly the validation loss fluctuates a lot, and starts to increase after a small number of epochs:

To my understanding, this type of behaviour is usually caused by overfitting. It seems like NAG converges to a region of optimization space which is significantly less generalizable, but I'm not sure why. I didn't find any papers which explain why NAG generalizes poorly in some cases.


texts: Document clustering to merge common labels

texts:
I am building a recommendation system and I have to clean up some of the labels that I have. For example of the data
df['resolution_modified'].value_counts()

Gives
105829
It is recommended to replace scanner                                                                                1732
It is recommended to reboot station                                                                                1483
It is recommended to replace printer                                                                                881
It is recommended to replace keyboard                                                                                700
                                                                                ...
It is recommended to update both computers in erc to ensure y be compliant with acme                                                    1
It is recommended to configure and i have verify alignement printer be work now corrado                                                 1
It is recommended to create rma for break devices please see tt for more information resolve this in favor of rma ticket create         1
It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update                        1
It is recommended to switch out dpi head from break printers                                                                            1

Notice that It is recommended to replace keyboard and It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update are very similar. Ideally, I would like to just converge to the string that occurs more frequently thus the second string should convert to the first.
I am thinking of using document clustering to handle this approach. I have tried using fuzzywuzzy but since I have a lot of strings the process below is too slow
from fuzzywuzzy import fuzz

def replace_similars(input_list):
    # Replaces %90 and more similar strings
    for i in range(len(input_list)):
        for j in range(len(input_list)):
            if i < j and fuzz.ratio(input_list[i], input_list[j]) >= 90:
                input_list[j] = input_list[i]

def generate_mapping(input_list):
    new_list = input_list[:]  # copy list
    replace_similars(new_list)

    mapping = {}
    for i in range(len(input_list)):
        mapping[input_list[i]] = new_list[i]

    return mapping
res = h['resolution_modified'].unique()
res.sort()
mapping = generate_mapping(res)
for k, v in mapping.items():
    if k != v:
        h.loc[h['resolution_modified'] == k, 'resolution_modified'] = v

I wanted to know if there is some document clustering I could apply that weighs in the strings that occur more than once and thus I would just take the common strings related to them that occur less and converge them to the more frequent occuring string. Does anyone have any recommendation on which method to use?
What I have Tried Thus Far:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
v = TfidfVectorizer()
x = v.fit_transform(df['resolution_modified'])
kmeans = KMeans(n_clusters=2).fit(x)
test_strings = ['It is recommended to replace keyboard', 'It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update']
kmeans.predict(v.transform(test_strings))

Which gives
array([1, 0], dtype=int32)

Obviously not working so far, will try to increase the number of clusters.


texts: Identifying sequences of behavioral interactions between multiple individuals

texts:
I'm wondering if anyone might have some novel insights as to the best way to analyze the following data.  It's a problem I've been thinking about in the back of my mind for a while, so I thought that I'd ask here. I have data that look like this:

     day event actor recipient
995    8   128     G         J
996    8   129     G         K
997    8   130     G         B
998    8   131     B         G
999    8   132     H         G
1000   8   133     G         H
1001   8   134     E         G
1002   8   135     G         J
1003   8   136     B         H
1004   8   137     G         H
1005   8   138     G         H
1006   8   139     B         J
1007   9     1     D         J
1008   9     2     A         J
1009   9     3     A         J
1010   9     4     H         J
1011   9     5     A         J
1012   9     6     D         H
1013   9     7     A         F
1014   9     8     D         J
1015   9     9     A         H
1016   9    10     D         J
1017   9    11     A         J
1018   9    12     F         J
1019   9    13     F         J
1020   9    14     F         H
1021   9    15     F         G
1022   9    16     F         H
1023   9    17     C         F
1024   9    18     C         G
1025   9    19     D         H

What you see here is an extract of a R dataframe.  The first column being the rownumber of the df, then the four variables.    These data start at day1 and end at day 22.  There are between 13 and 215 'events' on each day - each event is a separate behavioral event. Higher number events occur later in time than earlier numbered events.  Individuals are in the 'actor' and 'recipient' variables.   The data are available in csv format here:

There are 11 individuals (A - K). One thing you'll notice is that recipients tend to be lower down the alphabet, and actors tend to be higher up the alphabet.

A key question I'm interested in working out a methodology to address is to see if the likelihood of becoming an actor increases if an individual has recently been a recipient. You can see this on line 997 that G-B and then B-G occur followed by H-G and G-H.    An individual recipient doesn't have to appear on the next line to count as having an increased likelihood of appearing as an actor - I'm interested in the decay in the probability of this occurring over events (but not continuing to the next day).

Further, I don't think this will be true for all individuals, so I am keen to test which individuals it is true for.

Lastly, I am interested to know if an individual who has recently been a recipient but is now an actor is paired with more often an individual of a higher or lower letter than themselves.

I hope that these questions are clear and make sense.   I obviously don't expect a full analysis.  But I would be keen to hear of ideas for this type of analysis.  I believe that examining some Markov processes may be useful but I am interested in hearing about other ideas.


texts:
This solution describes how to simulate statistics to find a confidence interval.
A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B.
The observed value came out to be 511 votes for A and 489 votes for B.
this makes us think that candidate A will win.
But we need to know if this sample is truly representative of the underlying population distribution. To find this, we simulate this poll 1000 times through below python function.

def sample(A,n=1000):
    return pd.DataFrame({'vote': np.where(np.random.rand(n) < A,'A','B')})

s = sample(0.51,n=1000)

dist = pd.DataFrame([sample(0.51).vote.value_counts(normalize=True) for i in range(1000)])

what I cannot understand is, what is the significance of parameter A in the function definition.

Is it trying to simulate a sample where A occurs 51% times?
If I am just trying to random samples from a population, why am I relying on 0.51 to do so?


texts: Versatile data structure for combined statistics

texts:
Not sure if this is Math, Stats or Data Science, but I figured I would post it here to get the site used.

As a programmer, when you have a system/component implemented, you might want to allow some performance monitoring. For example to query how often a function call was used, how long it took and so on. So typically you care about count, means/percentile, max/min and similiar statistics. This could be measurements since startup, but also a rolling average or window.

I wonder if there is a good data structure which can be updated efficiently concurrently which can be used as the source for most of those queries. For example having a ringbuffer of rollup-metrics (count, sum, min, max) over increasing periods of time and a background aggregate process triggered regularly.

The focus here (for me) is on in-memory data structures with limited memory consumption. (For other things I would use a RRD type of library).


texts: Variable Importance in unsupervised anomaly detection algorithms

texts: I am working on an anomaly detection problem to detect fraud in insurance claims. I have used the PyOD package and used algorithms like ABOD, CBLOF, Isolation Forest, and AutoEncoder. I couldn't find any way to identify the important features which make the data points anomalies ( like variable Importance in Random Forest). Is there a way to identify the important features in unsupervised anomaly detection?


texts: In Natural language processing, what are the major applications of Hidden Markov Chain (HMM), and what are the alternatives that usually can outperform HMM, is RNN and LSTM always the choice right now?


texts: Is there a method that is opposite of dimensionality reduction?

texts:
I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.

I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.

I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.

When plotting the data, it is clear to the trained eye that there are three different linear trends, the three color lines show the directions.

When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.

EDIT, Additional info: I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.


texts: Keras/TensorFlow in R - Additional Vector to Custom Loss Function

texts:
I would like to pass a vector that is outside of the training data, but the same length as the training data, to a custom loss function.

The vector represents a post-prediction funnel (one or zero) that an observation has to pass through before they can yield (one or zero).  Obviously, I can't use this funnel as a feature, but I would like to use it in a loss function. Something along the lines of cross entropy plus an additional penalty...

funnel <- k_constant(fnl)
loss <- function(y_true,y_pred){
   k_sum(-((y_true*k_log(y_pred)) +
   (1-y_true)*k_log(1-y_pred) +
   k_transpose(funnel)*(1-y_true)*k_log(1-y_pred)))
}

This code works syntactically, but I'm concerned that when I go to fit the model, batch_size and shuffle arguments rearrange the data in a way that funnel no longer lines up with the training data.

Question...

How can I be sure that the funnel and the (y_true, y_pred) pairs are referencing the same observation when calculating the loss?

More info on the 'why' of this problem, as well as additional code that shows my progress thus far, can be found here.

Thanks in advance for any help!  I'm relatively new to deep learning, so please bear with me.


texts: Determine most important features in diagnostic data

texts:
I have a dataset of device diagnostics. I have two tables: one relating each device to failures code. Two devices can share a failure code for example a common chip malfunction. The second table links the device to signs of failure for example overheating. Each sign has a severity score and devices can have overlapping signs. For instance:
Device_code,Failure_code
X001,8989
X001,8650
X001,8490
X002,8989
X002,8650
X002,3433
....

Device_code | signs | severity
8989, s1234, 1
8989, s2345, 4
8989, s4567, 2
8650, s1234, 4
8650, s9999, 1
...

How can I find the most important signs for each Failure_code?
I thought about merging the tables, converting each sign to a binary column, where 1 and 0 mean the failure has the feature or not, resp. Then running a logistic regression to predict failure_code and sort by coefficients. However, in this method I cannot use the severity feature. What is a good method to approach this problem?
Thanks!


texts: How can I find correlation between features?

texts:
The problem

I want to figure out how routers correlate between each other. Like, if a specific error occurred in router A, and almost at the same time the error occurs in router B, they probably have some connection with each other (are at one line).

The Data

Suppose I have a dataframe that looks like this:

|Router|Error|Duration|Timestamp          |
|DB-XX |GSM  |26.5374 |2019-05-01 00:20:14|
|DT-XY |AUC  |15.5400 |2019-05-01 01:15:01|
|DR-YY |AUC  |02.0333 |2019-05-01 01:17:13|
|DP-YX |LOC  |45.2609 |2019-05-01 00:01:10|

The question

What is the best way to deal with it? Regression (one vs the rest) for each router? The problem is, that there are hundreds of models and I also want to reduce computational costs...


texts: How to avoid overfitting in Reinforcement Learning

texts:
I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its performance on different levels from the ones used for training. This way, I can assess if the knowledge learnt by the model generalizes well to previously unseen levels.

After doing some testing and plotting the results, I have noticed an issue. Although the model generalizes well, it ends up overfitting the training set (levels used for training) if the training process continues for too long. This makes sense since early stopping is a common technique used to prevent overfitting. The problem is that the longer the training lasts, the more samples the agent is trained on, which should improve its test performance. This doesn't happen, however, due to overfitting, as I have explained.

I would like to run the model and plot the performance (reward obtained) obtained on the test set as the number of samples in the experience replay increases. The plot I want to obtain should show how the performance increases as the number of samples grows, but overfitting causes that, after a point, the performance decreases, even if more samples are used for training. I want to avoid this. These are the solutions I can think of:

Use techniques to prevent overfitting. Right now I'm using Dropout but it still occurs.
Use an exponential decaying learning rate. Right now I'm using a constant one, since I don't want the newer samples to impact the training less than the older ones.
Give up and change the number of training steps depending on how many samples I want to use for training. Then, to obtain the plots, I will have to repeat the training once per each point (number of samples) of the plot, in order to obtain the best performance for each different size of the experience replay. Of course, this is my last resort.

What do you think I should do?

Edit: I have just tried L2 regularization. It works more or less the same as Dropout.


texts: What scale does LightGBM use for output?

texts:
Let's assume I'm modeling a process like:

$$y=log(x1+x2+x3+N)$$

Where $x_i$ are features and $N$ is some error/noise value.

With the way that decision trees work and the way that LightGBM works, will this be modeled correctly? Or will it be necessary for me to train using $e^y$?


texts:
I'm trying to build a classifier for my dataset and I'm having an issue with using my gridsearchCV and pipeline together. Here is my code:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

imp = Imputer()
scaler = StandardScaler()
clf = LogisticRegression(multi_class='multinomial')

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.3, random_state=42)

pipeline = make_pipeline(imp, scaler, clf)

param_grid = {'penalty':["l1","l2"], 'C':np.arange(0.001, 1, 0.01),
                  'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']}

search = GridSearchCV(estimator=pipeline, param_grid=param_grid, cv=5)

Once I call the method on search, I get the following error.

search.fit(Xtrain, ytrain)

ValueError: Invalid parameter C for estimator Pipeline(memory=None,
 steps=[('imputer', Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)), ('standardscaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('logisticregression', LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
      intercept_scaling=1, max_iter=100, multi_class='multinomial',
      n_jobs=1, penalty='l2', random_state=None, solver='liblinear',
      tol=0.0001, verbose=0, warm_start=False))]). Check the list of available parameters with `estimator.get_params().keys()`.

I'm not sure how my C parameters are invalid. Does anyone know what I'm doing wrong?


texts: Passing TFIDF Feature Vector to a SGDClassifier from sklearn

texts:
import numpy as np
from sklearn import linear_model

X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])
Y = np.array(['C++', 'C#', 'java','python'])
clf = linear_model.SGDClassifier()
clf.fit(X, Y)
print (clf.predict([[1.7, 0.7]]))
#python

I am trying to predict the values from arrays Y by giving a test case and training it on a training data which is X, Now my problem is that, I want to change the training set X to TF-IDF Feature Vectors, so how can that be possible?
Vaguely, I want to do something like this:

import numpy as np
from sklearn import linear_model

X = np.array_str([['abcd', 'efgh'], ['qwert', 'yuiop'], ['xyz','abc'],  ['opi', 'iop']])
Y = np.array(['C++', 'C#', 'java','python'])
clf = linear_model.SGDClassifier()
clf.fit(X, Y)


texts: What should be the main concerns when generating extra data?

texts:
I've been participating in a project where I had to cut most of the dataset due to mismeasurements/invalid values and we ended up with a small dataset.

The dataset is regarding  the following:

Tool Deflection : Metrology - Sensor Measured
Tool Wear : Metrology - Sensor Measured
Piece - Form Accuracy : Metrology - Sensor Measured
Piece - Roughness : Metrology - Sensor Measured
Machine Input - Parameters : System Inputs

I thought of generating more data for each metrology dataset, using the Machine Inputs as reference, however, I feel like it could potentially bias my model.

As you can see, I'm quite confused with respect to this subject.

What should I expect if I move towards this solution?
Is there any restriction I should be aware of?


texts: Testing for gender composition between groups

texts:
I did kmeans with k=4 and I would like to find out if there are any differences in the composition of gender between the 4 clusters.

Do I use Fisher's Exact Test of Independence? Is there any recommended library in r that can handle this as well as post-hoc tests between groups?


texts: How much data is needed for a GBM to be more reliable than logistic regression for binary classification?

texts:
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.

Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.

I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.

I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.


texts:
I want to build a Recommendation System to recommend products to users.
This is for research purposes.
The context-system the engine will be integrated in is also not build yet.

So right now I am starting the project, building kind of a E-Commerce & Social-Network-Platform for research purposes.

To realize the recommendation system, I want to integrate Neo4j and Elasticsearch with each other. For the content based filtering part Elasticsearch should do its job nicely. For the collaborative filtering part I want to use the Graph in Neo4j.

I would like to ask you for some experience or suggestions about the following questions regarding this topic:

Should I use another database as the main system storage and just use Neo4j to store recommendation data? Or is it a suggestive way to just store everything in the same graph?
How would you determine which part of the recommendation computation should happen online in Realtime and which offline as a precomputation?
Does anyone here have some experience with this kind of realization? What did your architecture look like?

Sorry, if this is a little vague described here and there. I am also new to this and want to expand my horizon.

Thanks alot for your help. I would be really happy to get some input here?

Cheers TJ


texts:
I'm trying to predict movie genres using a neural network. I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?

Also, how would I have to format my data to make it work with Keras?


texts: Label construction for predictive maintenance

texts:
I am trying to do a binary classification related to predictive maintenance. The question I address is "What is the probability that the asset will fail in the next X units of time?" There is a guide from Microsoft, which describes how to do it quite well. It describes the labeling process as well.
However, it only shows it for a single failure event:

Label 1 means "about to fail", label 0 means "normal operation".

My question is: how to proceed when the failure point spans multiple records, i.e. the failure point spans multiple black dots?

I thought of discarding data points and to leave just a single black point per failure point. But I have doubts for cases when I calculate (for example) aggregated features for 2 months and there are multiple failures within 2 months. This means that I will remove some data, which might be relevant for the second failure.

UPD: To clarify the situation with multiple failures. The scenario is like this: asset operates normally, then fails at some point. It continues to report data although it is in a failure state. Then, the asset is repaired and continues to operate normally until the next failure. Each failure has a timestamp indicating when the failure has happened and a duration (i.e. time until asset is in normal operation). For simplicity, I can say that multiple failures are of the same type, i.e. assigned to the same class.


texts: Logistic Regression performing better than SVM with a Gaussian kernel performing better than a linear SVM

texts:
I am very new to machine learning.

I am working with a data set, and my algorithm for logistic regression (with lasso regularization) is performing fairly well (~0.8 AUC), my SVM with a Gaussian kernel is doing well (~0.7-0.8 AUC), and my SVM without a kernel is doing fine but not as well (~0.7 AUC). It seems odd to me that a SVM with a kernel would have a performance in between two algorithms which do well with linearly separable data. Is this normal, and what could be going on here? Thanks for the help!

EDIT: Could it be because of Lasso's hard thresholding?

In response to Jason's query, my dataset has roughly 800 features and 800 observations, and is all numerical.


texts: Eigenvectors and eigenvalues for natural language processing

texts: How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?


texts: Should I create metafeatures for my XGBoost training set?

texts: Say I've got two (not necessarily independent) features A and B for my dataset. Should I create metafeatures from them? say for example the ratio: $$\frac{A}{B}$$


texts:
Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

Particularly, I still have over 88k variables (features) left and one cannot do much with them without massive amounts of memory. But does doing models on "blocks" lose the interactions occuring between blocks or are there some technique for "aggregating" these?


texts: What are some options to add or remove nodes from a multiclass classification model?

texts:
I'm building a classification model that will need to classify into one of many possible outputs. I know in advance that I will need to add and subtract nodes from the output layer as circumstances change. Please refer me to any resource you are aware of that can help me understand potential approaches. I can retrain the model from scratch every time, of course. I'm looking for alternatives. I am only aware of two approaches:

Transfer learning: Only retraining the final layer with the new desired outputs.
Stacking one binary classifier for every potential output and streaming my data through all of them.

I need a level of specificity that is pretty granular, for example, I'd like to be able to identify a document that is primarily focused on a topic like "Changes in goat-herding practices in Uzbekistan." This is a made-up requirement, but the point is that it may have very specific features. Please provide input on the pros and cons of the approaches that I mentioned and add any approaches that I haven't thought of.


texts:
The following is the error code generated while running lasso.py.
Can anybody help in fixing the same.

Here is the code:

from cvxpy import *
import numpy as np
import cvxopt
from multiprocessing import Pool

# Problem data.
n = 10
m = 5
A = cvxopt.normal(n,m)
b = cvxopt.normal(n)
gamma = Parameter(nonneg=True)

# Construct the problem.
x = Variable(m)
objective = Minimize(sum_squares(A*x - b) + gamma*norm(x, 1))
p = Problem(objective)

# Assign a value to gamma and find the optimal x.
def get_x(gamma_value):
    gamma.value = gamma_value
    result = p.solve()
    return x.value

gammas = np.logspace(-1, 2, num=100)
# Serial computation.
x_values = [get_x(value) for value in gammas]

# Parallel computation.
pool = Pool(processes = 4)
par_x = pool.map(get_x, gammas)

for v1,v2 in zip(x_values, par_x):
    if np.linalg.norm(v1 - v2) > 1e-5:
        print("error")

Error: while running

Traceback (most recent call last):
  File "lasso.py", line 31, in <module>
    objective = Minimize(sum_squares(A*x - b) + gamma*norm(x, 1))
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 45, in cast_op
    return binary_op(self, other)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 435, in __sub__
    return self + -other
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 45, in cast_op
    return binary_op(self, other)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 423, in __add__
    return cvxtypes.add_expr()([self, other])
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/affine/add_expr.py", line 33, in __init__
    super(AddExpression, self).__init__(*arg_groups)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/atom.py", line 41, in __init__
    self._shape = self.shape_from_args()
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/affine/add_expr.py", line 41, in shape_from_args
    return u.shape.sum_shapes([arg.shape for arg in self.args])
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/utilities/shape.py", line 49, in sum_shapes
    len(shapes)*" %s" % tuple(shapes))
ValueError: Cannot broadcast dimensions  (10,) (10, 1)

As suggested by multiple links, I tried with all possible python and library versions. But I am unable to fix this. Kindly throw some light.


texts: How to improve naive Bayes multiclass classification accuracy?

texts: I have around 9 string features which I have indexed using string indexer and used vector assembler to get the feature vector and used a normalizer to normalize across features . These are the transformation I have made and used naive Bayes classifier as the estimator , all the above operations after enclosed in a pipeline model. I trained on around 3 lakh samples and tested on 20k samples  the model predicted same label for all of them. I handpicked the features is that the problem? Or any tuning parameters available ? I found out a smoothing parameter which I set it to 1.0 should I increased or decrease to improve the model accuracy? Please help


texts: How to perform link prediction in text based relationship data

texts:
I need to establish if there is a link between 2 columns from two different datasets with one matching column, where;

Dataset1: bipartite:  (M, DS)
M    G
m23  ds3
m23  ds67
m54  ds325
...  ...

Dataset2: tripartite: (M, G, DG)
M    G    DG
m23  g6   dg32
m23  g8   dg1
m54  g32  dg65
...  ...   ...

These 2 datasets have one column in common(i.e., M), and the relationship among the elements is shown below:

M  ----affects----> G
M  ----causes-----> DS
DG ----affects----> M

Primary Goal: To calculate the probability of a possible link/edge that might exist between indirectly related columns(eg. DG and DS) via the common column(M).

So, for a given list of DS entries, how to find the probability of the existence of a link/edge between
selected DS, and all the other DGs

DS <---- ----> DG

If DS; (ds3, ds67) were selected, the output should be like this:

element1 - element2 - probability/statistical value to signify the existence of direct relationship OR link.

ds3 - dg32  - 100% (common M value)
ds3 - dg1  - 100%  (common M value)
ds3 - dg65 - 43.66%
---
ds67 - dg32 - 100% (common M value)
ds67 - dg1  - 100% (common M value)
ds67 - dg65 - 55.12%

I am trying to code this in Java, but Python based solutions can work too.

I am sorry I am not too familiar with graph theory, a little descriptive solutions would be really appreciated.
Thanks.


texts: How To Merge Features in the Dataset Forest Cover Type Classification Problem?

texts:
The forest coverType Dataset Contains the following attributes distributed in many boolean features.

Wilderness_Area (4 binary columns) / qualitative / 0 (absence) or 1 (presence) / Wilderness area designation
Soil_Type (40 binary columns) / qualitative / 0 (absence) or 1 (presence) / Soil Type designation

Training the SVC classifier with RBF kernel without attributes re-engineering takes too much time. Can we merge all 40 binary columns for form a Soil_Type Attribute representing those columns. What are the pros and cons of this approach ?

Also, do we have any direct approach to achieve this in weka or sklearn ?


texts: Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

texts:
I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent.

At least I understand that one iteration of mini-batch gradient descent should be faster than one iteration of batch gradient descent.

However, if I understand it correctly, since the mini-batch gradient descent must update the weights by the number of the batch size in one epoch, the training would be slower than the batch gradient descent, which computes and updates the weights only once in one epoch.

Is this correct? In that case, is it worth worrying about the loss of the overall training time?


texts:
After plotting a sklearn decision tree I check what it says in each box and there is one feature "value"
that I am not sure what it refers.

The first line will be the column and the value where it splits, the gini the "disorder" of the data and sample the number of samples in the node.

But value?


texts: Mask RCNN: Random predictions during inference for the same image

texts:
I recently trained the Mask RCNN (matterport's implementation) on some satellite images, but during inference mode, I'm getting random predictions for the same set of weights for the same image.

That is to say, for a given set of weights and the same image I'm getting different bounding box and mask predictions.

Does anyone have an idea of why this is the case?

The following are the different predictions that the model produced for the same image.

On some cases, the model produces reasonable predictions like that of the 2nd image but other times it produces completely random results. I can't figure out where this randomness is coming from.

I would highly appreciate any insight into this problem.


texts: XGBoost - feature importance just depends on the location of the feature in the data

texts:
I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data is by far most important and then second is second, etc.
It seems like a sign that the model is not working properly as its not really learning anything... any advice on what could be going wrong?
UPDATE:
Correlation Matrix
https://ibb.co/3shDJjD
Model Code:
params = {
  'subsample':0.5,
  'learning_rate': 0.3,
  'max_depth':8,
  'num_parallel_trees' : 20,
  'objective': 'reg:squarederror',
  'verbosity':0,
  }
watchlist = [(train, 'train'), (test, 'val')]
reg = xgb.train(params, train, num_boost_round=5, early_stopping_rounds=5, evals=watchlist)

Results:
[0] train-rmse:0.274535 val-rmse:0.27431
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 5 rounds.
[1] train-rmse:0.273472 val-rmse:0.273653
[2] train-rmse:0.272796 val-rmse:0.27341
[3] train-rmse:0.272318 val-rmse:0.27334
[4] train-rmse:0.271943 val-rmse:0.273346
[5] train-rmse:0.271604 val-rmse:0.273374
[6] train-rmse:0.271218 val-rmse:0.273442
[7] train-rmse:0.270927 val-rmse:0.273529
[8] train-rmse:0.270641 val-rmse:0.273561
Stopping. Best iteration:
[3] train-rmse:0.272318 val-rmse:0.27334

Feature importance (note that 0 and 1 are first). If I change the order of the columns in the xtrain, the feature importance will also change and first two columns will always be two most important features.
https://ibb.co/QcHwbNg


texts: Is it possible to forecast the evolution of cars?

texts:
Let's say for example that I have a dataset about the cars that a company (e.g. Toyota) produced, over the course of the years 1990 - 2016.
Considering that I have already completed the feature engineering step, is there an algorithm that can output predictions on what attributes future Toyota cars will have?
What I mean is, can we produce output that will have the same features/attributes as the input data-points?
simple example:
For input X0-Xn = [EngineType, #OfWheels, #OfDoors, hasRadio, hasGPS, ProductionYear] --> Produce output
Y = [EngineType, #OfWheels, #OfDoors, hasRadio, hasGPS, ProductionYear]


texts: Noramlization Time Series to Predict Stocks exact Price

texts:
I am trying to made a model for neural network that tries to predict prices of the stocks point by point using a LSTM (yeah I know that probably did not get anything and I should predict up/down or something realted boolean decision but I like to try this first)

I wil use as features HIGH, LOW, OPEN of last 60 days and try to predict ADJ_CLOSE of next day.

The probem come when I need to normalize the data. I am using data from AAPL from 1980 to 2019 the main problem is that in 80's stocks prices are in pennys and in 10's they are in houndreds of dollars, so I decided to do window normalization using MinMaxScaler from Sklearn.
There is no problem with the train data, I have 7887 data points so I scaled them by windows of 500 and last remaing separetly. The code is like this

x_train = train.values

window_size = 500
for i in range(0,7500,window_size):
    x_train[i:i+window_size] = min_max_scaler.fit_transform(x_train[i:i+window_size])

x_train[i+window_size:] = min_max_scaler.fit_transform(x_train[i+window_size:])

The problems comes when testing I do not know that doing something like

x_test = test.values
x_test = min_max_scaler.fit_transform(x_test)

is kind of chating because they are fiting by themselves. But I can not do

x_test = test.values
x_test = min_max_scaler.transform(x_test)

after window normalization of train data(using last fitted scaler) because the range between min and max on last window of train data difers a lot from the range in test data and in that case the normalization would not be well done and the test data would not fit between [0,1].

Any help?


texts:
I am trying to learn different pruning methods for decision trees. I have put together a list of methods below.

Reduced Error Pruning
Cost Complexity pruning
Minimum error pruning
Pessimistic Error Pruning
Critical Value Pruning
Error Based pruning

I got enough content to learn the first two algorithms, but I am unable to get enough content and understand the other algorithms. Can any one help me out to understand these algorithms thoroughly? Also, which algorithms are mostly used for pruning and why?


texts: On the choice of LSTM input/output dimension for a spatio-temporal problem

texts:
I am using LSTM neural networks from (R)Keras for a matter of spatio-temporal interpolation. I manage to get the network to output predictions but the results are not outstanding (very little improvement on validation loss). I am wondering about the shapes of training data and labels.

Say I have 50 dates of measurements of the variable of interest $y$, accompanied by about 100 covariates $x$ (spatial coordinates, temperatures...). Each date has 24 measurements of $y$, so nsamples=50*24=1200. If I set the timestep hyperparameter of LSTM to e.g. 3, and use a moving window of step 1, I have therefore an input table $X$ of shape (1200, 3, 100).

On the other hand, should the labels table $Y$ be of dimension (1200, 3) or (1200, 1) ? More precisely, which of the following describes the problem the best:
$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t-2} ; Y_{n,t-1} ; Y_{n,t})
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t} ; Y_{n,t} ; Y_{n,t})
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t}
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t+1}
$$

Or are they all plausible ways of addressing slightly different problems? As I said, I'm trying to spatially interpolate $Y$ for the 50 dates of measurements, as well as predicting $Y$ for the year(s) to come.  So I expect one is more relevant than the others but I have no clue on which one.

I hope this is understandable as I clearly miss some technical vocabulary here.


texts:
I have a big dataset containing almost 0.5 billions of tweets. I'm doing some research about how firms are engaged in activism and so far, I have labelled tweets which can be clustered in an activism category according to the presence of certain hashtags within the tweets.

Now, let's suppose firms are tweeting about an activism topic without inserting any hashtag in the tweet. My code won't categorized it and my idea was to run a SVM classifier with only one class.

This lead to the following question:

Is this solution data-scientifically feasible?
Does exists any other one-class classifier?
(Most important of all) Are there any other ways to find if a tweet is similar to the ensable of tweets containing activism hashtags?