texts:
1) What are the appropriate activation and loss functions for multi-class classification problem?

Is it so that:

Up to 2 classes $\rightarrow$ Binary classification $\rightarrow$ Activation: Sigmoid $\rightarrow$ Loss: binary_crossentropy
From 3 classes $\rightarrow$ Multi-class classification $\rightarrow$ Activation: Softmax $\rightarrow$ Loss: categorical_crossentropy

If so then...

'Dogs and cats' are binary classification but 'dogs, cats and horses' multi-class classification?
If we have 2 classes then we can't output probability ranges for prediction results?

2) Is it so that for multi-class classification, the last dense layer must have a number of nodes equal to the number of classes?

In the following example there are two classes (dogs and cats) and last dense layer has one node (why not 2 nodes?). If we have 3 or 10 classes then last node should have 3 or 10 nodes?


texts: Why not continue next epoch with best weights

texts:
I've seen the following happen in our training: (numbers are paraphrased)

epoch 10 - val loss 0.0500 - loss 0.0450
epoch 11 - val loss 0.0400 - loss 0.0400
epoch 12 - val loss 0.0420 - loss 0.0410

Now clearly epoch 11 is better than epoch 12. I assumed epoch 13 would start with the weights of epoch 11, but have been told they do not.

In my opinion worst of all, our training pipeline is set up with ReduceLROnPlateau lowering the learning weight after 4 epochs of no improvement. Now say epoch 15 has a validation loss of 0.0415 and a loss of 0.0405, we could've better trained on with the weights of epoch 11.

Why are new epochs not started with the weights of the previous best epoch?


texts: What is the ideal size to a binary CNN? Is my dataset long enough?

texts:
I would like to know what is the ideal size to a CNN, or there's a mathematical function to determine it, or it change through the differents scopes?

And also, I'm doing a binary classification CNN with 700 images each class(total - 1400). Is it too small? How could I argue that it is small?

Note: I get 83% accuracy with my cnn.


texts: Number of epochs in Gensim Word2Vec implementation

texts:
There's an iter parameter in the gensim Word2Vec implementation
class gensim.models.word2vec.Word2Vec(sentences=None, size=100,
alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=0,
seed=1, workers=1, min_alpha=0.0001, sg=1, hs=1, negative=0,
cbow_mean=0, hashfxn=<built-in function hash>, **iter=1**, null_word=0,
trim_rule=None, sorted_vocab=1)

that specifies the number of epochs, i.e.:
iter = number of iterations (epochs) over the corpus.

Does anyone know whether that helps in improving the model over the corpus?
Is there any reason why the iter is set to 1 by default? Is there not much effect in increasing the no. of epochs?
Is there any scientific/empirical evaluation of how to set the no. of epochs?
Unlike classification/regression task, the grid search method wouldn't really work since the vectors are generated in an unsupervised manner and the objective function is simply by either hierarchical softmax or negative sampling.
Is there an early stopping mechanism to cut short the no. of epochs once vectors converges? And can the hierarchical softmax or negative sampling objective converge?


texts: What are some of the important CNN architectures one should know about? What CNN architectures did well on the ImageNet ILSVRC challenge? What CNN architectures are good candidates for transfer learning?


texts: How should I retrain my neural network with new images?

texts:
I trained a convolutional neural network with a large (>90k) dataset of images. Since there is some confusion between two classes, I collected all the images that were misclassified and I augmented them with several methods. Furthermore, I collected new images that were not present in the original dataset. Now I would like to re-train my network in order to reduce the confusion. What is the best approach? Should I perform fine-tuning using only the new images (case A) or should I add the new images to the whole training set and re-train the network from zero (case B)?

My concerns are the following: in case A, most of the images belong to the same class, therefore the dataset would be very unbalanced. In case B, I am afraid that the impact of the new images on the whole dataset would be very small.


texts: How is the salary for Data Scientist different among countries?

texts:
I would like to know the salary for Data Scientists by countries, but failed to find any resources on it.

I know that US gives data scientists an insanely high amoung of money (well, it might not be "insanely high" on the standard of US), but I rarely hear about any other countries, possibly except UK, Canada, and Australia in some cases.

Is there any research or survey or something that shows the diversity of salary for Data Scientists by countries, something like for software engineers?


texts: Problem when exporting scored labels to excel

texts:
I have done a regression using Azure ML Studio and it ok. The r squared is 95% on the test sample.
When predicting, the results look ok when visualizing the data.
The results appear as expected, with the predicted label having statistics very similar to the trained ones.
But when I try to export it to Excel either:
 1. consuming it from the web or
 2. reading directly the saved dataset
I obtain "weird" number like  instead of 41,36.
How can I obtain right values?


texts: R tutorial for multitask regression tree for missing value prediction?

texts:
I have a dataframe with 20 columns. Data is integrated from five different data sources. I have shown 4 columns from each dataset. 4/5 sources have discrete data (either 1 or 0). The last section has continuous values which is my target (Contains NA). I intend to implement Random Forest algorithm to predict the missing values. I have seen tutorials which predict only one Y variable. Is this a regression tree for missing value prediction task for continuous multioutput Y? kindly share some tutorial links which could provide some reference code in R.

  Input File Sample

  D000022 D000038 D000152 D000163 A01AB06 A01AB07 A01AB08 A01AB09 76 77 80 84 fp131 fp132 fp133 fp134 C0000727 C0000731 C0000737 C0000768
1       1       0       0       1       0       1       0       0  1  0  1  0     0     1     0     1     0.03       0    0.0406    0.26
2       1       1       0       0       0       0       0       0  0  1  0  1     1     1     0     0       NA       NA      NA     NA
3       0       0       0       0       0       0       0       0  1  0  1  0     0     1     1     0        1        0   0.1900    0.3


texts: SMOTE train test split with validation data

texts:
Would like to ask, in which way to use SMOTE?
My dataset is imbalanced and a multiclass problem. As I read in many posts, use SMOTE method only for the training dataset (X_train and y_train). Not for the test dataset (X_test and y_test). There I include validation data. How do you handle SMOTE with validation data?
df = pd.read_excel...

X=df.drop('column1',axis=1)
y=df.column1

#Training part
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)

smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

#validation part
X_train_smote, X_val, y_train_smote, y_val = train_test_split(X_train_smote, y_train_smote, test_size=0.5, random_state=42)

Is this correct?
and is it right, that the validation datasets (X_val and y_val) have both SMOTE inside?  or should I make it out of the normal train test split:
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.5, random_state=42)?
Im confused.


texts:
I am working with a binary classification problem, and regardless of what changes I make, the model seems to just be guessing between 0 (Negative) and 1 (Positive). The dataset is imbalanced at a ratio of 5:1. To counter this problem I created cross validation sets where the training set is perfectly balanced (8000 samples of each), and putting the rest into the testing set. This seemed to fix to the problem of the model either picking all 0's or all 1's. But at this point it just seems to be guessing. I've messed around with the hyperparameters a bit, but this is what I currently have. learning rate = 0.00001, batch size = 100, loss function = cross entropy, optimizer = Adam. I'll attach my train function below
def train(model, train_loader, optimizer, epoch, device, loss_function):
model.train()
with torch.enable_grad():
    for batch_idx, sample in enumerate(train_loader): #get batch
        data = sample["image"]
        truth = sample['label']

        data = data.to(device)
        truth = truth.to(device)

        output = model(data) #pass batch

        truth = truth.long()
        optimizer.zero_grad() #zero gradients

        loss = loss_function(output, truth) #calculate loss

        loss.backward() #calculate gradient
        optimizer.step() #update weights


texts: How to quantify the numerical influence of categorical variables?

texts:
I have trained a simple neural net, to make predictions based on three inputs. For examples sake lets say I m trying to work out "how many days does it take to complete an accademic study", where the data could be:

f1 = institution
f2 = student age
f3 = study topic

and the label is the total number of days to complete.

If you imagine the following data structure:

f1, f2, f3, label

A,  D,  G,   v
B,  E,  H,   w
C,  F,  I,   x
A,  E,  H,   y
A,  D,  G,   z

Is there anyway that I can work out how long each feature adds to the total number of days to complete, so for example I want to be able to say that if you choose f1 = A that 4 days will be added to the total days to complete the study.

I can work out how much difference there is between each choice, eg choosing f1 = A might be 7 days quicker than if you chose f1 = B, but I do not know how to roughly ( maybe average ) identify how long each choice represents.


texts: How to predict specific user from session logs?

texts:
Let's say I have a dataset with 800 rows (40 entries for each of 20 users). The entries are user session logs (columns are - browser, os, time, date etc for a specific session).

Now each user has unique id (1-20). Let's say user_id=1 is special one and I need to detect it whenever new data comes.

So for new data I need to predict whether that session is of user_id=1 or not.

My question is: how can I do that?

One way I thought of is to make a feature representation for each of the 20 users and whenever new data comes in, take the distance of the data from each of feature and see the minimum distance.

However, the problem is that when I make a unique feature representation for each user, how do I deal with the columns like browser, os - because a user can have used multiple browsers in all those 40 sessions?


texts: Selecting models for ensemble from large group of models with high uncertaintly

texts:
I'm in a situation where many models have been created, and I have their cross-validation performances as well as performance on test data. I need to select models for inclusion in a simple bagging ensemble that are most likely to generalize to new data.
Conventional wisdom would dictate choosing models with high CV performances and low correlation with each other, since each individual model should have a good chance of generalizing and there will be an error-correction effect with the model diversity.
However, it seems like given a large number of models, the odds of having a high test AUC by chance are not insignificant, and therefore choosing uncorrelated models could actually be more dangerous since that lack of correlation could indicate that they've found vastly different mechanisms to reach a high CV performance, one of which could be incorrect/overfit. Perhaps the safer way is actually to pick models within a certain range of correlation (like Pearson or Spearman in the range of 0.7-0.9, for example), to maintain some error-correction effect while ensuring that the mechanism is fairly consistent (and therefore, perhaps reliable).
I've been searching for literature on this, and haven't been able to find anything. I'd really appreciate any guidance on how to approach this, or what papers to read - thanks!


texts: Comparing youtube view counts between historical release dates

texts:
I've written a script to use youtube-dl to collect all view-counts and ratings from all listed videos on a youtube channel: https://gist.github.com/philshem/ca0b8f964458c63ee4f0e68c0f700f39#file-get_stats-sh
To compare ratings is relatively simple, because they are already averaged. But I would like to compare views, which is not clear to me because the number of views is related to how recently the video was released.
I am looking for a metric that normalizes somehow the age of the video with the number of views. I could do something simple, such as view_count * days_since_release, but I don't expect that to be statistically meaningful.
Is there an example metric for comparing historical, cumulative data?

https://twitter.com/philshem/status/1295725069268910081


texts: Predictive Maintenance - Top books involving any domain

texts:
After working with predictive maintenance using ML algorithms for a while, I felt like I needed more theoretical background. To do so, I started some courses, but they don't dig into the hole and just explain demo cases.

I started searching for books (in reddit mostly), but I feel like the books that I find are not that good.

Can someone list top books of predictive maintenance using machine learning?


texts: Splitting image dataset with few subjects but many data

texts:
I'm carrying out training/testing of a convolutional neural network for facial expression recognition with various datasets - all labelled by 7 emotion classes.

For other datasets, there are a large number of mostly unique subjects so I randomly split. In this case, however, there are only 6 subjects but a large number of images for each subject in each class. Randomly splitting seems ineffective because of the similarity in images - think of how an emotion changes per frame.

Is the best method to separate an entire subject for testing? Or something else?

I did run the network with random splitting and achieved 100% validation accuracy so I believe that is unlikely to be the best method. Thanks for your time.


texts: Deep Learning vs gradient boosting: When to use what?

texts:
I have a big data problem with a large dataset (take for example 50 million rows and 200 columns). The dataset consists of about 100 numerical columns and 100 categorical columns and a response column that represents a binary class problem. The cardinality of each of the categorical columns is less than 50.

I want to know a priori whether I should go for deep learning methods or ensemble tree based methods (for example gradient boosting, adaboost, or random forests). Are there some exploratory data analysis or some other techniques that can help me decide for one method over the other?


texts:
I'm running Keras' LSTM (not CuDNNLSTM) but I notice my GPU is under load. I need recurrent dropout, so I can only stick with LSTM. Is the 'normal' LSTM assisted by GPU? If so, how are LSTM and CuDNNLSTM different? I presume CuDNNLSTM uses the CUDNN API (and LSTM doesn't?

Similarly, is the normal LSTM supposed to be faster running on GPU or CPU?


texts:
I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like you have high levels of cholesterol, you need to have a low-salt diet that emphasizes fruits, vegetables and whole grains; limit the amount of animal fats and use good fats in moderation(this will be the suggestion; and it is an example suggestion from doctor).

So, now when i was researching on how this could be accomplished i stumbled on this research paper and i learned about something called paraphrase extraction from it. and also that i need to build a parallel corpus with tag and suggestion.

I want to do this in python. I couldn't find much information from internet on how paraphrase extraction can be done in python, but there are many articles talking about paraphrase detection and things. So, do we have any libraries in python for this purpose? or any kind of help is appreciated. How do i build a corpus for this same pupose?


texts: Am I supposed to scale binary features alongside other numerical features?

texts:
When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all together?

My fear is that scaling them all together makes the binary features seem less important than they really are.

Note: I am prepping a neural network for binary classification. I am using a sigmoid output and scaling my features from [0,1]


texts: How to explain get_weight with autoencoder in keras?


texts: What is the best approach for detecting the defect?

texts: Assuming that we have 10 same objects, they are lined up and equidistant. If any of them is rotated a very small angle(5 - 10deg), what is best method to detect them? I am using a camera to capture the image. I want to divide the image into 10 parts and then feed each part into CNN model to classify the defect. Is that a possible way? What is the recommendation for this problem?


texts: extract classifier properties from pickled file

texts:
I have *.clf file which I get from fit() of sklearn. I fit my data with SVM or KNN and want to show its properties when using it for predictions. For example I open earlier pickled classifier file and when I print it I get something like this:

SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma='scale', kernel='rbf',
    max_iter=-1, probability=True, random_state=None, shrinking=True, tol=0.001,
    verbose=True)

How can I get the value of, for example, gamma to print out it somewhere else except for traversing it as string? Because at first I have to define either it's SVM or KNN.


texts:
I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario:

Bot : Hi, what is your name?
User: My name is Edwin.

I wish to extract the name Edwin from the sentence. However, the user can response in different ways such as

User: Edwin is my name.
User: I am Edwin.
User: Edwin.

I'm tried to rely on the dependency relations between words but the result does not do well.

Any idea on what technique I could use to tackle this problem?

[UPDATED]

I tested with named entity recognition together with part of speech tagger and parser. I found out that most model is trained in a way that the first character of the entity for the person name or the proper noun must be upper case. This may be true for normal document, but it is irrelevant for a chatbot. E.g.

User: my name is edwin.

Most NER failed to recognize this.


texts: Number of filters in Convolutional Neural Networks

texts: How do I know how many filters to choose from? And how does the neural network learn and adjust the filters?


texts:
For (a) I have calculated $P(G)=\frac{5}{8}$, $P(O|G)=\frac{2}{5}$, $P(B|G)=\frac{1}{5}$, $P(C|G)=\frac{4}{5}$, and $P(A|G)=\frac{4}{5}$. Now how do I calculate the maximum likelihood estimate of these values?

And how do I go about part (b)? I get that $O,B,C,A$ are independent so I can multiply them to get joint probability. But for values like $O_i$ for sample $i=9$, that is just $0$, since sample 9 doesn't have outdoor seating. And how am I supposed to calculate $P(G_i)$ if I don't know what $G_9$ is?


texts: custom loss in keras, problem with batch size

texts: I am trying to create a custom loss function,custom_loss(y_true, y_pred). I understand that y_pred is calculated by my model but I want to deliver, two kinds of y_true, such as y_true1 and ytrue2(these are pre-calculated, not delivered in model.fit(), and used as follows: (y_pred -  y_true1/ y_pred - ytrue2). The problem is, I have batch size mis-match problem, because, my custom y_true1 and y_true2 is created on my total dataset. How to make their batch size as y_pred?


texts: please help with ValueError: Input contains NaN, infinity or a value too large for dtype('float64')

texts:
How to fix error ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Im thinking the error is from too large for dtype or an infinity.

This is an imbalanced multiclassification neural network with 15 classifications. With 78 input dimensions.
Program is here https://pastebin.com/dru0pfEH

dataframe = pandas.read_csv("C:/Users/bam88/train.csv", header=0,
dtype=object).drop_duplicates()
dataset = dataframe.values
x_train = dataset[:,0:78].astype(float)
y_train = dataset[:,78]
dataframe = pandas.read_csv("C:/Users/bam88/test.csv", header=0,
dtype=object).drop_duplicates()
dataset = dataframe.values
x_test = dataset[:,0:78].astype(float)
y_test = dataset[:,78]

encoder = LabelEncoder()
encoder.fit(y_train)
encoded_Yone = encoder.transform(y_train)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_yone = np_utils.to_categorical(encoded_Yone)
#encode our testing set
encoder = LabelEncoder()
encoder.fit(y_test)
encoded_Ytwo = encoder.transform(y_test)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_ytwo = np_utils.to_categorical(encoded_Ytwo)
(dummy_yone.sum(axis=-1) == 1).all() == True
(dummy_ytwo.sum(axis=-1) == 1).all() == True

smote = SMOTE('minority')
x_smotetrain, y_smotetest = smote.fit_sample(x_train, dummy_yone)
print(x_smotetrain.shape, y_smotetest.shape)


texts: which deep learning text classifier is good for health data

texts:
I have a data set like this:

postID  Sentence                                         drugYesOrNo
1       He went out with his friends
2       He behaved nicely while talking with me
3       He stopped using drugs after a while                 1
4       He did not meet any friend during last week
1       He slowly cut usage of drugs                         1
2       He smiled like he is good
3       He did not seem happy with his situation

As you see there is two features. the first feature is our sentence and the second feature shows is this sentence is a sign that patient has stopped his drug or not.

the first column shows that sentence which are part of a paragraph. for example HERE sentence 1-4 are one paragraph in which we have splited them to see which sentence exactly show the stopping of drugs.
so sentence 3 in first paragraph shows this.

In the second case, sentence 1-3 is part of a paragraph. here sentence one shows that this person stopped using drugs(which is not good the person should continue)

so my goal is to apply a deep learning text classifier on my text data and make a model and so when I received  A NEW PARAGRAPH, I will be able to predict if the person has stopped his drugs or not.

first question, with this case study, which deep learning text classifier may work best?

Secondly, as you see we have cut the paragraph in to series of the sentences. but in reality we will give a paragraph to test the model. in your idea what will be the best approach to deal with this?

the thing that came to my mind is that while testing and receiving a paragraph, we again split the paragraph to sentences and give those sentences to the model but I am not sure it is a good approach.

we have 900 of these sentences, again Im not sure with this much of data it will be goof to apply deep learning classifier on it.

I appreciate it if you give me your point of view:)

Update after reading comments

I asked a couple of guys to make such a dataset for me. I mean looking at paragraph, spliting, then saying which sentence has that meaning(stopping drugs or not). what if I did not ask them to explicitly say which sentence does have that meaning and just point which paragraph does have that meaning(stopping drugs or not). do you think labeling exactly which sentence does have that meaning has been a good idea rather than which paragraph does have that meaning? I hope I am clear enough :)


texts: Three different errors using external information. Which one makes sense? (Or how to interpret each?)

texts:
My goal is to compare clustering methods considering different method and different number of clusters using an external information.
Could anyone please give some opinion/ recommend book/paper about the following problem?
Let's say I have N data point and I will feed each of them to a system as one input and get a corresponding output and I consider their sum as the total output.
Now, I cluster the N datapoints into K clusters and find their centroids. I assign the number of data points in each cluster as the weight of its centroid. Now, I will feed my centroids to the system one by one and I will get K output.
How should I calculate the error?
Should I calculate the (average?) error for each cluster first and then take the average of the errors calculated for the clusters?
Or should I divide the errors in each cluster by the mean of the ground truth output and then sum the errors calculated for the clusters? Or takes the average (instead of "sum")?
Or just find the weight sum of outputs obtained from centroids and substract it from the actual total output?


texts: How to connect data-mining with machine learner process

texts:
I want to write a data-mining service in Google Go which collects data through scraping and APIs.

However as Go lacks good ML support I would like to do the ML stuff in Python.

Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.

For example most (web) protocols lack at:

buffering between processes
clustering over multiple instances

So what (type of libraries) do data scientists use to connect different languages/processes?

Bodo


texts: Prepending Input layer to pre-trained model

texts:
I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values.

I've tried several variations of the code below:

input = Input(shape=(1036800,), name='image_input')
initial_model = VGG16(weights='imagenet', include_top=False)
x = Flatten()(initial_model(input).output)
x = Dense(200, activation='relu')(x)
x = Dense(1)(x)
model = Model(inputs=input, outputs=x)

Previous variations of the above code yielded errors related to the input being the wrong dimensions, input_shape needing to have 3 channels (when using (1036800,) for that parameter in the initialization of VGG16), and the most recent error that results from running the above code is this:

Traceback (most recent call last):
  File "model_alex.py", line 57, in <module>
    model = initialize_model()
  File "model_alex.py", line 20, in initialize_model
    x = Flatten()(initial_model(input).output)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 596, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2061, in call
    output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2212, in run_internal_graph
    output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 164, in call
    dilation_rate=self.dilation_rate)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3156, in conv2d
    data_format='NHWC')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 639, in convolution
    input_channels_dim = input.get_shape()[num_spatial_dims + 1]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 500, in __getitem__
    return self._dims[key]
IndexError: list index out of range

Here is the full code. Here is the sample data file used in the script.

One of the possible approach towards fixing this might be to resize the raw image files to 224x224 and turn them into numpy arrays of shape (224, 224, 3) so they can be plugged into the pre-trained model's first layer. However, I don't want to warp the images or waste another night pre-processing data when I should already be training.

Besides that, I all I can think to do is Google my problem and try to adapt the found solutions or aimlessly tweak various shape related parameters and functions -- neither of which has gotten me very far over the past 4 hours.


texts:
What are the ways to determine the level of difficulty w.r.t. the complexity of concept for any given document ?

I am aware of Flesch Reading Ease Score, but that's generic and based on word count and syllables count.

Is there anything that works at the concept level ?

Thanks!


texts: Can I set the perposal anchor boxes to a specific size in object detection?

texts:
I'm trainig a SSD MobileNet V2 FPNLite 640x640  object detection model on custom dataset, I undorstood that to make the model train faster is to change the parameters of anchor_generator for example, I want the model to detect big objects only and the object could be at least 20% of the image size and less then 80%
how can i change the parameters to tell the model not to generate boxes smaller than the expected objects size ?
here are the defult ones:
    anchor_generator {
      multiscale_anchor_generator {
       min_level: 3
       max_level: 7
       anchor_scale: 4.0
       aspect_ratios: 1.0
       aspect_ratios: 2.0
       aspect_ratios: 0.5
       scales_per_octave: 2

A side question: my dataset contains one object per images will this effect of the training ?


texts: Clustering uncertain data with independent uncertainty per dimension

texts:
I have $n$ objects located in a $d$ dimensional space, however I do not know their exact coordinates. For each object and each dimension, I have a set of noisy measurements of the coordinate.

I would like to cluster this data in the

For example, with $n = 3$ and $d = 2$, I could have access to the following data:

object $a$, dim 1: 0.8 0.7 0.6
object $a$, dim 2: 1.0 1.0 1.0 0.9
object $b$, dim 1: 0.4 0.3
object $b$, dim 2: 0.2 0.1
object $c$, dim 1: 0.9 0.6
object $c$, dim 2: $\emptyset$

In my current approach, I take the for each (object, dimension) pair that has data the average value, and I imput the average value of the dimension for the missing data.

So I would get

$a$ [0.7 0.975]
$b$ [0.35 0.15]
$c$ [0.75 0.6] (where $0.6 = \frac{1+1+1+0.9+0.2+0.1}{6}$)

Then I use the scikit-learn python library to run the mean-shift algorithm and obtain clusters.

I am not completely satisfied with this methods for two main reasons (maybe they're the same):

an (object, dimension) pair with a single observation is treated as as reliable as one with numerous observations
there is a discontinuity between how a pair with zero observations and a pair with some observations are treated. In the second case, the value of the other objects does not influence the attribute at all.

My questions are: What would be a more principled approach to this problem? If I need to use another algorithm, are there open source decent quality libraries available that would implement it?

I currently use $d = 256$ and $n = 10$ for my testing, but I aim to use $d = 16000$ and $n = 1000$ (but maybe a smaller $n$ if it's unrealistic) in my target application.


texts: How to compare topics generated from topic modeling from different datasets?

texts:
I have two datasets of a similar theme. Let's assume Dataset A and Dataset B. Using the top2vec model (https://github.com/ddangelov/Top2Vec) (https://arxiv.org/abs/2008.09470) on each dataset, I came up with a certain number of topics. Now, I want to compare both datasets' topics.
How can I do this? Clustering or any other method will work as far as I can compare the generate topics computationally.


texts: Residual plots: why do we want to know the error?

texts:
The residual plot represents the error between the actual value.

Y-axis: residuals
X-axis: the predictor variable or fitted values.

Why do we want to know the error, what benefits us in getting the residuals?
I've uploaded an image from the video. I don't understand what is being sampled in the illustration.

Where is the independent and dependent variable(s)?
How do you identify the error?

They take one calculation and apply to the right-side of the graph and so on.

I am taking this class and don't know anything about what the instructor is talking about.

Can someone break this all the way down? (This is a beginner's course!)

Here is the transcript from the video:

  Examining the predicted value and actual value we see a difference. We
  obtain that value by subtracting the predicted value, and the actual
  target value. We then plot that value on the vertical axis with the
  dependent variable as the horizontal axis. Similarly, for the second
  sample, we repeat the process. Subtracting the target value from the
  predicted value. Then plotting the value accordingly. Looking at the
  plot gives us some insight into our data. We expect to see the results
  to have zero mean, distributed evenly around the x axis with similar
  variance. There is no curvature. This type of residual plot suggests a
  linear plot is appropriate


texts: Is it reliable to use TensorFlow (ML in general) to classify baggage bag tags based on the presence of a green stripe?

texts:
The images are identical except for the presence of the stripe on the side.
I am trying to use a classify the images into 2 classes: greenStripe, noGreenStripe.

I tried to use tensorflow retrain with a small dataset (~40 pictures in each class and batch size of 8) but the results where really bad. I am afraid to commiting to training using more data as it is time consuming.

What do you suggest? Is there a better approach or does the problem lie in the small training dataset?


texts:
I have text data in .xlsx or .txt files.
How do I convert them to a corpus file (.tab)?
I would like to use the corpus file with Orange's text add-on.


texts: Are there any available resources that contain data on LaTeX symbols usage metrics. I want to know what the most used symbols are to determine which I should account for in a project I'm working on.


texts: No module named "" error when loading the pickle file

texts:
I created a model and I saved it in a pickle file using the Algorithm SVR(Support Vector Regression)

import pickle
pickle.dump(model,open('carb patients data/Pickles/svr.pickle', 'wb'))

In jupyter notebook it gives an error

  can't pickle _thread.RLock objects

So I converted that jupyter file in to a .py file and downloaded it and executed using the Python Idle. Then it got saved in that particular location. But when I try to load my pickle file from another Jupyter Notebook it gives an error called,

  No module named 'sklearn.svm._classes'

The code how i tried to load the pickle file,

from flask import Flask, request, redirect, url_for, flash, jsonify
import numpy as np
import pickle as p
import json
from sklearn import ensemble
app = Flask(__name__)

@app.route('/api/', methods=['POST'])
def makecalc():
  data = request.get_json()
  prediction = np.array2string(model.predict(data))

  return jsonify(prediction)

if __name__ == '__main__':
   modelfile = 'Clustering Patients/carb patients data/Pickles/svr.pickle'
   model = p.load(open(modelfile, 'rb'))
   app.run(debug=True, host='0.0.0.0')

Any thing I have done wrong or is it an issue in the Jupyter Notebook? How can I load my saved pickle file?


texts: High error machine learning regressor algorithm in Python - XGBOOST Regressor

texts:
I have a dataframe with real state data from florida, it includes single apartments and buildings data:

'TRUE_SITE_CITY': The city where the building is. variable: Miami, Aventura...;
'CONDO_FLAG': If it is a condominium or not, variable: yes/no;
'BEDROOM_COUNT': Number of total bethrooms, variable: integuer,
'BUILDING_actual_AREA': The area of the entire building, or apartment in the case that there are only one apartment or house. variable: integuer;
'FLOOR_COUNT': Number of the floors that the building has;
'DOR_CODE_CUR': the type of the building. Variable: categorical;
'UNIT_COUNT': Number of apartments or houses in the building. Variable: integuer;
'YEAR_BUILT': Year that the building or house or apartment was build: Variable: categorical;
'public_transport_min_distance': I have calculated the nearest stations of the public transport;
'Price': The variable that I want to predict.Variable: integer.

I have done an exploratory data analysis and I have dropped some data that has null values and some data that was incorrect. Also I have dropped the values with outliers.

The basic statistics of the price column (targeted column):

I have checked the categorical features and they have enough variables in each one to keep they in the model.

I have done a pipeline to make a one hot encoder for the categorical values and a standard standardisation for the numerical values. In it I have include a XGBOOST regression:

from xgboost import XGBRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import cross_validate
from sklearn import metrics
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import Pipeline
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import make_pipeline, make_union
from mlxtend.feature_selection import ColumnSelector
from sklearn.preprocessing import StandardScaler
from category_encoders import OneHotEncoder

x_numeric = df_x[['BEDROOM_COUNT','BATHROOM_COUNT',
       'HALF_BATHROOM_COUNT', 'FLOOR_COUNT','UNIT_COUNT','public_transport_min_distance','BUILDING_actual_AREA']]

x_categorical = df_x[['TRUE_SITE_CITY','CONDO_FLAG','YEAR_BUILT']]

categorical_col = x_categorical.columns

numeric_col = x_numeric.columns

estimator_pipeline = Pipeline([
    ('procesador', procesing_pipeline),
    ('estimador', estimator)
])

score2 = cross_validate(estimator_pipeline, X= df_x, y= df_y, scoring=scoring,return_train_score=False, cv=5,n_jobs=2)

But I am obtaining a high error. The mean value of the price is almost 200.000 and the error that I obtain is:

I have done feature selection using RFE but I obtain a high error as well.

Also I have run it doing RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV

params = {"estimator__learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,
 "estimator__max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15],
 "estimator__min_child_weight" : [ 1, 3, 5, 7 ],
 "estimator__gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],
 "estimator__colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ] }

random_search = RandomizedSearchCV(
    estimator=estimator_pipeline,
    param_distributions=params, cv=5, refit=True,
    scoring="neg_mean_squared_error", n_jobs= 3,
    return_train_score=True,
    n_iter=50)

But I obtain a similar error value.

What could I do?


texts: How will ADA Boost be used for solving regression problems?

texts: I have an idea of how ADABOOST will be used for classification but I want to get the idea of how to re-weight and thus use ADABOOST in case of regression problems.


texts: Tensorflow.js error: must be either an HTMLVideoElement but was HTMLVideoElement

texts:
I am getting this error in the console using tensorflow.js

Uncaught (in promise) Error: pixels passed to tf.browser.fromPixels()
must be either an HTMLVideoElement, HTMLImageElement,
HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData
in webworker or {data: Uint32Array, width: number, height: number},
but was HTMLVideoElement

I am just passing a video html element after it's loaded so what does it mean?
My js code code:
...
const input_tensor = tf.browser.fromPixels(document.getElementById("video"));

My html tag:
<video id="video" autoplay muted></video>


texts: Overfitting - how to detect it and reduce it?

texts:
I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with unbalanced classes (3.5% of BAD type of clients).

I'm trying to make various models to have enough of them to make ensembles, but for this purpose, let's focus on one of them, particularly XGBoost.

I was reading a lot on how to tackle overfitting, but I haven't found any good source on how to do it in practice, step-by-step. As for now, my two best models have:

0.8 AUC on training data, around 0.799 AUC on holdout set and around 0.7355 and 0.7195 AUC on out-of-time batches.
0.764 AUC on training, 0.7785 AUC on the holdout test set and 0,7285 AUC on both out-of-time batches.

I am worried about is that drop on out-of-time batches, since I think that 0.05-0.08 drop is huge and it might be a sign that models that I did, really are overfitting and don't generalize well. To clarify, while I was tweaking models, I didn't know about those out-of-time scores.

Could anyone share the experience what is best practice to detect overfitting? And does those two models overfit, or I am just panicking, and this drop in performance is normal?

My current pipeline in general looks like this:

Data cleanup
Feature importance using xgboost package to take best 300 features from all 2400 available.
Removing highly-correlated features (0.75 cutoff) - 123 features left
Train/test split - 0.8 vs 0.2 plus two out-of-time batches
Model selection using nested CV(5-fold CV in outer) with hyperparameter tuning in inner loop(5-fold CV in inner) - all done in MLR package.
From 5 models I get from nested CV, I'm picking the best performing one (that has the closest AUC in both train and holdout test set)

And then when I was happy with the model I performed a test on out-of-time models.

How I could improve my pipeline, in a way that I could detect overfitting? Is there any list of steps that would roughly cover what it could be done to reduce it?

Also, in highly unbalanced case, choosing a good validation set means that I only need to take care of the proper distribution of the target variable? Should I take care of something else as well?


texts: GAN generator output minibatch classes order

texts:
Let's take MNIST as example. I'm interested which classes(digits) images and in which order there are in generator output.

I think it always generates [0, 1, 2, 3, 4, 5, 6, 7, 8 ,9] images is this true? If so, can we say that there is classification ability implicitly learned by generator as well (it actually knows which number should be at each position) ?


texts: CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0

texts:
According to Wikipedia, the GeForce GTX 670M has a Compute Capability of 2.1 (and a Fermi micro-architecture), which is confirmed by TensorFlow (I can read "2.1" in the error it rises).
Wikipedia says that CUDA 8.0 supports compute capabilities from 2.0 to 5.x (Fermi micro-architecture included). It even says that it's the "last version with support for compute capability 2.x (Fermi)". However, the error rised by TensorFlow says that my being-used CUDA version support at least compute capability of... 3.0... And thus my GeForce GTX 670M is ignored and my CPU is used to compute :-( . It's a big problem.
Since several CUDA versions are installed in my computer, I wanted to be sure it was CUDA 8.0 that was being used. So I typed, in the PyCharm's terminal: nvcc --version which outputs:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Jan__9_17:32:33_CST_2017
Cuda compilation tools, release 8.0, V8.0.60

Thus, CUDA 8.0 is actually being used.
The error is the following:

2019-08-01 22:04:28.366003: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
2019-08-01 22:04:28.561338: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 670M major: 2 minor: 1 memoryClockRate(GHz): 1.24
pciBusID: 0000:01:00.0
totalMemory: 3.00GiB freeMemory: 2.48GiB
2019-08-01 22:04:28.561814: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1093] Ignoring visible gpu device (device: 0, name: GeForce GTX 670M, pci bus id: 0000:01:00.0, compute capability: 2.1) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0.

My setup
I've described the problem. Now I'm going to give you more informations about my setup.

I'm working on Windows 10 with PyCharm and Keras (so TensorFlow)
Since I thought I had to use CUDA 8.0, I checked in the TensorFlow Website which version I should use. I've found that: for CUDA 8.0, I should use TensorFlow GPU version 1.4.0 with CUDNN version 6.
In reality, I'm using: CUDA V8.0.60, CUDNN V6.0 (found on the CUDNN Website for CUDA 8.0) and TensorFlow-GPU V1.4.0
Here is a screenshot of my Windows 10's %PATH%'s value. Note that the being-used CUDA version ("CUDA_PATH") is actually 8.0, as you can see it on the screenshot.

Final question
Could you please tell me what version of CUDA I should use for being able to use my GeForce GTX 670M as a compute unit for my networks training? Wikipedia seems to be wrong...


texts:
I'm doing project on subject of affinity analysis for my statistical class in college.

In order to complete it, I have to acquire sales database with at least 200-300 records, each containing list of products bought by single client.

Are there any example sales databases available for free to use for such educational purposes?


texts: Are all DNN models supervised learning (or reinforcement learning)?

texts: Are there any type of unsupervised learning models in DNN?


texts: How to create an array from the list of arrays in python

texts:
I was trying to write a python code that can set some neural network channels or neurons to zero at the inference; and I wrote the code below. The code generates 10 different arrays for different percentage of the channels or neurons that are set to zero. My challenge is how to combine these arrays into a single array or a list so that they can be individually accessed, but all I was getting was a list containing arrays with zero elements. Please help me.

import numpy as np
def three_steps(n):
    return step(n, steps=[1, 0.5, 0.25])

B = np.linspace(0.1, 1.0, 10)
print B
B = np.array(three_steps(100))
print "A = ", A

for i, value in enumerate(B):
    A[i:int(value*100)] = 0
    print A
list = [np.append(A, i) for k in range(i + 1)]
print list


texts: What is Happening in the training process when we are fitting a model to the data

texts:
In any prediction task, the process of “fitting” a model to the data observed in the training process can be best described as...

Assessing all observations available and then backsolving for the dimensionality of best fit.
Making use of all observations available to create a model, then using distance or other measures to assess how well the model characterizes or "fits" its data.
Starting with a linear model and then expanding the dimensionality of that model until it fits the data
Starting with a linear model and then regressing the shape of the line until it fits the data.

These are the four points I am confused about. Which one/ones is/are true?


texts: How are selected the features for a decision tree in CART?

texts:
Suppose I want to use CART as classification tree (I want a categorical response). I have the training set, and I split it using observation labels.

Now, to build the decision tree (classification tree) how are selected the features to decide which label apply to testing observations?

Supposing we are working on gene expression matrix, in which each element is a real number, is that done using features that are more distant between classes?


texts: What is the right way to implement the graph of validation process of Batch Normalization in tensorflow?

texts:
When using Batch Normalization we can use the batch mean and batch variance during training.
However this fails to give good enough results in validation case. We get the best results by providing the population mean and population variance.

However this means that the two graphs will be almost identical except the part where we need to subtract the mean and divide the standard deviation (we use the population ones)

Our current implementation worked ok but it includes too much boilerplate code.

Note that in other frameworks you can extract and manipulate layers of a model quite easily for such cases, but tensorflow has an only append policy to its graphs

What is the optimal way to go in such cases where one graph is slightly different from the other?


texts:
Image recognition uses deep learning, and in particular CNNs to train on and recognise faces. Usually, this entails training on lots of data.
However, recently, we have seen face recognition being deployed everywhere, and being used for passport control, for example, and some airlines have deployed in lieu of boarding pass scanners. How is this accomplished? How can they achieve such accuracy without having hundreds of pictures of everyone? How can, for example, governments, with a quick scan of my face, recognise me and identify my name and identity without having any picture of me besides the ones available from my official ids? There is a link here which, however, does not really explain it technically. Is it done using neural nets? What is their architecture. Are there papers describing it (technically) anywhere?


texts: Confusion in Policy Iteration and Value iteration in Reinforcement learning in Dynamic Programming

texts:
What I understood for value iteration while coding is that we need to have a policy fixed. According to that policy the value function of each state will be calculated. Right?

But in policy iteration the policy will change from time to time. Am I right?


texts: How to create a stacked bar chart with gaps between values of each variables in Python

texts:
I have a problem with creating a stacked bar chart in Python. I have data with 3 variables as below:

A=[3,5,7]
B=[4,5,7]
C=[2,3,4,5,6,7]

I would like to create a bar chart with gaps of values of each variable as below bar chart with gaps between values of each variable :

Could anyone know how to create it in Python?
Thank you very much.


texts: Should I save DQN parameters for performance spikes?

texts:
I am currently running a DQN on the Atari Breakout OpenAI Gym. Over time the total number of points per episode is increasing, but occasionally the performance spikes to achieve 200+ points on a single episode:

Should I capture the params for this specific run and call it a day, or is this anomalous and I'm missing something? In general, should additional code always be written to capture spikes like this?
I'm using BreakoutNoFrameskip-v4. Is something about the game random, such that this won't generalize and play well for other runs?


texts: If There is a case where decision trees are getting overfitted so by using gradient boost method do we solve that problem?

texts: I have came across a case where my decision trees are getting overfitting so by using methods like gradient boost can I solve that problem.


texts: Working with Data which is not Normal/Gaussian

texts:
What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions?

I noticed in many data sciences courses, there is always a strong assumption of using a normal/Gaussian data. I have always wonder why this is so, and most people would say that due to central limit theorem, the data is always assumed to be normal.

However, what if the data that I am dealing with is not normally distributed?
Should I perform log/ exponential transformation on the data for the sake of getting normal distributed data?
Why is Gaussian data always best suited?


texts: In performing ALS and getting an item matrix of latent features, what would be the best method for inferring the possible "meaning" of each latent factor in the item space? And as a corollary, are the any methods for successfully visualizing this data to gain some intuition or understanding?


texts: Classifying sparse binary data for int value

texts:
I'm very new to data science and still trying to get the grips. The problem I'm trying to tackle is, we have a pool of footballers from a league and data objects representing a group of 11 footballers for a given match and the number of goals scored by that team on that match.
The goal is to estimate the number of goals that are potentially to be scored given any random line up of footballers from this pool. This data set spans over a long period of time and footballers transfer between teams, so, teams don't matter.
I thought about representing each footballer as a separate binary attribute to mark if they were present on a given object or not. Then, do some pruning by dropping the attributes that occurred less than a threshold and then what? What kind of classifiers should I be considering to classify such a spare binary data set for an integer value?
From what I know, neural networks is one potential candidate but I do not understand them enough to use, so, a few other options to play with would give me some direction I hope.


texts:
I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT.
To the best of my understanding this token is used to gather knowledge of the entire class, and is then solely used to predict the class of the image. My question is — why does this token exist as input in all the transformer blocks and is treated the same as the word / patches tokens?
Treating the class token like the rest of the tokens means other tokens can attend to it. I'd expect that the class token will be able to attend other tokens while they could not attend it.
Also, specifically in ViT, why does the class token receive positional encodings? It represents the entire class and thus doesn't have any specific location.
Thanks!


texts: Why can't I use data augmentation with a pretrained convnet?

texts:
Reading Deep Learning with Python by François Chollet. In section 5.3.1, we've instantiated a pretrained convnet, VGG16, and are given two options to proceed:

  A) Running the convolutional base over your dataset, recording its
  output to a Numpy array on disk, and then using this data as input to
  a standalone, densely connected classifier similar to those you saw in
  part 1 of this book. This solution is fast and cheap to run, because
  it only requires running the convolutional base once for every input
  image, and the convolutional base is by far the most expensive part of
  the pipeline. But for the same reason, this technique won’t allow you
  to use data augmentation.

  B) Extending the model you have (conv_base) by adding Dense layers on
  top, and running the whole thing end to end on the input data. This
  will allow you to use data augmentation, because every input image
  goes through the convolutional base every time it’s seen by the model.
  But for the same reason, this technique is far more expensive than the
  first.

Why can't I use data augmentation to generate more training data from existing training samples then go with option A? Seems like I can run the VGG16 base over my augmented dataset and use the output as the input to a standalone densely connected classifier.


texts: understanding linear algebra of a forget gate

texts:
This blog covers the basics of LSTMs.

A forget gate is defined as :

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t]+ b_f)$$

At this point the linear algebra confuses me more than it should. The syntax of $W\cdot [h,x]$ is confusing in this context. I think a vector should go into the activation function since the output $f$ is a vector, but the syntax of the forget gate above implies that the input has $2$ columns because $[h,x]$ will be an $n\times 2$ matrix

For the sake of example  lets say ...

\begin{align} W &= \begin{bmatrix} 0 & 1 \\
          2 &3 \end{bmatrix}\\
    h &= \begin{bmatrix} -1 \\
          2 \end{bmatrix}\\
    x &= \begin{bmatrix} 3 \\
          0 \end{bmatrix}\\
    b &= \begin{bmatrix} 1 \\
          -2 \end{bmatrix}\end{align}

Can anyone give the final vector that goes into the sigmoid function ?

I think the math is

$$  \begin{bmatrix} 0 & 1 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} -3 & 3 \\ 2 & 0 \end{bmatrix} + \begin{bmatrix} 1 \\ -2\end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 4 & 6\end{bmatrix}+ \begin{bmatrix} 1 \\ -2\end{bmatrix} = \text{ Something wrong}$$


texts: In the context of anomaly detection, which is a better language to use, python or R?

texts: In the context of anomaly detection, which is a better language to use, python or R ?


texts: How to determine the best way to replace empty records in the dataset?

texts:
I am doing a project related to predicting the next glucose value based on his/her past records. But in some patients, some of the recordings are missing. There are 2 scenarios of how the blood glucose values are missing in my dataset. So I'll label them as Scenario 1 and Scenario 2 in my examples.

Scenario - 1

A patient has recorded the glucose values thrice a day basically before breakfast, lunch and dinner on day 30. But on day 31 he has only recorded only the breakfast and dinner. Lunch is not recorded on day 31. So how can we replace the 0 value in lunch with another value?

Scenario - 2

Another patient has recorded the blood glucose values continuously from day 1 to day 40, and then he hasn't recorded the blood glucose values for another 2 days (day 41 and day 42 recordings are not there for before breakfast, before lunch and before dinner). Again he started recording the values in the day 43. So what is the best approach to tackle this kind of scenario?

I went through many articles and majority explained on replacing the mode, median or mean values for empty records. But I think mode, median is not suitable for this kind of dataset. I highly doubt whether I can use the mean also to replace values with empty records in the Scenario - 1. Can we actually use mean for replacing the empty records in Scenario - 1 or is there any other good approach?

From my knowledge I think I can't use the above three methods in replacing the values in Scenario - 2 since about 6 records are missing. If I am correct what is the best approach for the Scenario - 2.

Thank you!!!


texts: Designing a ConvNet to facilitate game playing

texts:
For fun I want to design a convolutional neural net to recognize enemy NPCs in a first person shooter. I have captured 100 jpegs of the npcs as well as 100 jpegs of not-NPCs. I have successfully trained a really simple convNEt to identify NPCs. This was really easy because the game actually highlights the NPCs with a red marker to let humans identify them. Makes it SUPER easy for a machine learning algorthm to find them.

Great , so now I can classify a screenshot of an NPC. The next step is to identify these in a data stream at 60 frames per second. We all know that the stupid little processors inside most cameras have a face detection algorithm that operates in real time. So my i7 with 2 NVIDIA gpus can do this no sweat. So now I have to grab the screen buffer, capture a screen shot, feed it to my conVnet, get the location of the NPC, and then move the mouse cursor to the center of that NPC.

Are there any easy to follow tutorials of running a convolutional neural net on a data stream like this?


texts:
We run a online marketplace for Commercial Real Estate industry and are looking to write matching algorithms to reduce the cost of search and transaction for the property owners/tenants.

We have two groups of users - owners and tenants and would like to implement matching algorithms based on their characteristics. Without any prior transactions data to work with, how would you approach this problems? More specifically, what matching algorithm techniques would you implement?

Code is in python.

Example:

Property

"property": [
{
       "amenities": ["kitchen",
                     "conference room",
                     .....,
                     .....]

        "location": "Munich",
        "feature2: "xyz"
        ...
}

Tenants

"Tenants": [
{
       "amenities": ["kitchen",
                     "conference room",
                     .....,
                     .....]

        "location": "Munich",
        "Type": 'Retail',
        "feature3": 'xyz'
        ...
}


texts:
I want to convert a json file into a dataframe in pandas (Python). I tried with read_json() but got the error:

UnicodeDecodeError:'charmap' codec can't decode byte 0x81 in position 21596351:character maps to <undefined>

I think I have some unwanted data in the json file like noise. The data is server generated.

This is a collection from the json file:

{"_id":{"$oid":"57a30ce368fd0809ec4d1b41"},"session":{"start_timestamp":{"$numberLong":"1470151881189"},"session_id":"8356bd90-20160802-153121189"},"metrics":{},"arrival_timestamp":{"$numberLong":"1470152028294"},"event_type":"OfferViewed","event_timestamp":{"$numberLong":"1470151943271"},"event_version":"3.0","application":{"package_name":"com.think.vito","title":"Vito","version_code":"5","app_id":"7ffa58dab3c646cea642e961ff8a8070","cognito_identity_pool_id":"us-east-1:4d9cf803-0487-44ec-be27-1e160d15df74","version_name":"2.0.0.0","sdk":{"version":"2.2.2","name":"aws-sdk-android"}},"client":{"cognito_id":"us-east-1:1d507b8f-857c-42a4-a705-8db07d46bc8f","client_id":"aa092911-b9a7-498a-82da-76318356bd90"},"device":{"locale":{"country":"US","code":"en_US","language":"en"},"platform":{"version":"5.1.1","name":"ANDROID"},"make":"Xiaomi","model":"Redmi Note 3"},"attributes":{"Category":"90000","CustomerID":"4077","OfferID":"11846"}}


texts: Is there a problem of over fitting in my dataset?

texts:
I have applied the sequential forward selection to my dataset having 214 samples and 515 features (2 class problem). The feature selection algorithm has selected 8 features. Now I have applied the svm (MATLAB) on these 8 features. I have also tried to see the performance after adding more features. The table given below gives the correct rate of the algorithm (training data set) along with the feature set used. The result obtained is:
8 features = 0.9392
10 features = 0.9439
12 features = 0.9672
14 features = 0.9672
16 features = 0.9626
18 features = 0.9766
20 features = 0.9672

As visible, the accuracy seems to increase. Is it because of over fitting? Should I use the default feature set as given by the sequentialfs function of Matlab or should I force it to deliver more features to get more accuracy?
I have uploaded the validation training and testing performance (70-15-15). Now can you tell me if my data is being over-fitted or not?


texts: How to implement conditional probability distribution on set-valued Random variables

texts:
I'm trying to implement conditional probability distribution when the events of two RVs are sets. If I try to extrapolate concepts from real or categorical variables to sets things become confusing for me. In particular, for some pair of discrete RVs, $X,Y$ we can compute the conditional probability of two events by using the Bayes theorem:
$$P(Y=y|X=x)=\frac{P(X=x|Y=y)P(Y=y)}{\sum_{y'\in\mathcal{Y}}P(X=x|Y=y')P(Y=y')}=\frac{f(x,y)}{\sum_{y'\in\mathcal{Y}}f(x,y')},$$
where $f(x,y)=\sum_{(x',y')\in (\mathcal{X},\mathcal{Y})}\chi(x'=x,y'=y)$ is an accumulator function giving the total number of times the specific joint event $x,y$ is observed in the sample space (i.e. the indicator function $\chi(x'=x,y'=y)$ equals one when $x'=x,y'=y$ holds). I'd like to compute such probability when events drawn by $X,Y$ are sets $X_i,Y_j$, respectively. So, how to define the corresponding $f(X_i,Y_j)$?
I've tried with:
$$f(X_i,Y_i)=\sum_{j}|(X_i\cap Y_i)\cap(X_j\cap Y_j)|.$$
The motivation behind is that $X_i\cap Y_i$ provides a measure of how the attributes of both sets should be observed jointly (such as any intersection event). In turn intersecting this intersection event against other intersection events $X_j\cap Y_j$ should provide the measure of how much the joint attributes are observed through the sample space. However, I'm not sure whether this analogy is correct with respect to the usual notion of observing joint events in a sample space. Please, some one can help me to clarify/fix/prove this?
Thank you in advance


texts:
I am trying to run Tensorflows DeepLab using my own dataset, while following: heaversm tutorial on GitHub/Youtube.
P.S I am running Windows.
I want to run the model_test.py script (on my windows terminal - CMD) but it just returns the typical:
Traceback (most recent call last):
  File "model_test.py", line 21, in <module>
    from deeplab import common
ModuleNotFoundError: No module named 'deeplab'

I have installed all dependencies including tensorflow:
Name: tensorflow
Version: 2.3.1

I have also included (permenantly in the "Path" environment) the directories:
C:\Users\Name\Desktop\models-master\models-master\research\deeplab\
C:\Users\Name\Desktop\models-master\models-master\research\slim\
C:\Users\Name\Desktop\models-master\models-master\research\

But nothing worked when I decided to re-run model_test.py.
So I then decided to temporarily set a environment variable named PYTHONPATH with the following (as the tutorial mentions):
# Making sure I am in the models\research directory (as per the tutorial)
# In my case it is models-master\research directory
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim  # --> macOS version
set PYTHONPATH=%PYTHONPATH%;%cd%;%cd%/slim  # --> Windows version

Again nothing worked... I am running out of options and hope.
Am i doing anything wrong here?? Am i missing a certain folder maybe?
Any help would be appreciated.


texts: Is there a name for this form of sampling?

texts:
I had an idea to combine oversampling and undersampling in the following way:

Compute the average number of individuals in each class. For classes with a number of individuals greater than this average, undersample the class so that the number of individuals in this class equals the average. For classes with a number of individuals lower than this average, oversample the class so that the number of individuals in this class equals the average. Overall, all classes then have a number of individuals equal to the original average.

However, I cannot see any literature on this idea. Does this idea exist and does it have a name so that I may read about it?


texts: tanh function values are either 1 or -1, how to interpret that distribution

texts:
I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the dataset passed through the NN.
The question is why I only have the -1 and 1 values? What does it mean can we conclude something from that?

thank a lot


texts: How to determine Number of neuron in hidden layer for classification

texts:
I used Iris dataset for classification with 3 layer Neural Network
I decided to use :
3 neurons for input since it has 3 features,
3 neurons for output since it has 3 classes and
In the hidden layer
what is the best number of neurons to do classification in this case?


texts: Error while using NLTK/ How should I read paragraph using NLTK?

texts:
I am a total newbie. I have a custom text document which is a paragraph. I am trying to read it and display using NLTK.

following is my code;

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from nltk.corpus import PlaintextCorpusReader

corpus_root = ''
wordlists = PlaintextCorpusReader(corpus_root, 'evidence.txt')
wordlists.fileids()
wordlists.words('evidence.txt')
print(wordlists.words('evidence.txt'))

I am getting an error;

  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position
  19: invalid start byte

please help.


texts: How to use arctan2 function inside Keras model?

texts:
I'm trying to add arctan2 function to the end of Keras model, but it looks like it is not getting any near even local minimum. Here is my ridiculous but minimal working code with native Keras Add() function instead of arctan2 function:

import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as ss

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv1D, Add
import tensorflow as tf

kernel_size = 64
epochs = 1000

def atan2(tensors):
  Q = tensors[0]
  I = tensors[1]
  return tf.math.atan2(Q, I)

def atan2_output_shape(input_shapes):
    return tuple(input_shapes[0])

atan2_layer = Lambda(atan2, output_shape=atan2_output_shape)

## Data generation for training

x_train = np.random.randn(1024, 512)

t = np.linspace(0, x_train.shape[1], x_train.shape[1], endpoint=False)
sine = np.sin(2*np.pi*t/32)
cosine = np.cos(2*np.pi*t/32)

x_I = np.multiply(x_train, cosine)
x_Q = np.multiply(x_train, sine)

b_I = ss.tukey(kernel_size)
b_Q = ss.tukey(kernel_size)

x_I_filt = np.array([np.convolve(b_I, x_I_i, mode='valid') for x_I_i in x_I])
x_Q_filt = np.array([np.convolve(b_Q, x_Q_i, mode='valid') for x_Q_i in x_Q])

y_train = x_Q_filt + x_I_filt
# y_train = x_Q_filt * x_I_filt
# y_train = np.arctan2(x_Q_filt, x_I_filt)

x_I = np.expand_dims(x_I, axis=2)
x_Q = np.expand_dims(x_Q, axis=2)
y_train = np.expand_dims(y_train, axis=2)

## Keras model

input_I = Input(shape=(x_I.shape[1], 1))
input_Q = Input(shape=(x_Q.shape[1], 1))

conv_I_1D = Conv1D(filters=1, kernel_size=kernel_size, activation=None, padding='valid', use_bias=False)(input_I)
conv_Q_1D = Conv1D(filters=1, kernel_size=kernel_size, activation=None, padding='valid', use_bias=False)(input_Q)

out_I_Q = Add()([conv_I_1D, conv_Q_1D])
# out_I_Q = Multiply()([conv_I_1D, conv_Q_1D])
# out_I_Q = atan2_layer([conv_Q_1D, conv_I_1D])

model_1D = Model([input_I, input_Q], out_I_Q)

model_1D.compile(optimizer='sgd', loss='mean_squared_error')
history_1D = model_1D.fit([x_I, x_Q], y_train, epochs=epochs, verbose=0)

and after 100 epochs I'm getting almost perfect initial filter kernel:

plt.semilogy(history_1D.history['loss'])
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

It also works well with Multiply() function. But if I replace y_train to y_train = np.arctan2(x_Q_filt, x_I_filt) and out_I_Q to out_I_Q = atan2_layer([conv_Q_1D, conv_I_1D]) I will get this sad loss graph:

I even initialize weights as they should be but with a little bit of an offset before runningmodel_1D.fit(...). b_I and b_Q are the same arrays.

offset = 1e-5
array_for_I_weights = np.array(model_1D.layers[2].get_weights())
array_for_I_weights[0,:,0,0] = list(b_I+offset)
model_1D.layers[2].set_weights(array_for_I_weights)
model_1D.layers[3].set_weights(array_for_I_weights)

Loss graph after training stage will looks like this:

But if I change offset to 1e-4 I will get this loss graph:

It only getting worse epoch after epoch. What is wrong with arctan2 function in case of placing it inside of Keras model? Why model is jumping away from good minimum? Maybe I should use another custom loss/metric function?

Changing optimizer type won't help at all. I'm using tensorflow 2.0 and Keras 2.2.5.


texts: Why am a getting wrong prediction when combining two list of samples, which individually gives correct prediction?

texts:
So I am coding in Python. I have to set of samples. Set1 contains samples of class A and the other set, Set2 contains samples of class B. These samples taken are a part of the training dataset. When I am predicting set1 and set2 individually, the classification is perfect. Now when I am merging the two sets for prediction into one set, the prediction gives the wrong result for the samples in Set2, i.e., predicting the samples of set 2 to be in class A. However, samples belonging to Set1 are predicted to be in class A in the merged set. Why is this happening? (I have saved the model and loaded that model for further prediction, the same error persists)

model.add(Dense(newshape[1]+1, activation='relu', input_shape=(newshape[1],)))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(50, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          metrics=['binary_accuracy'])
model.fit(X_train, y_train,validation_data=(X_test, y_test),validation_split=0.2, epochs=500, batch_size=25, verbose=0)


texts: Transfer value of one column to another column into a new column based on condition

texts:
I have a dataframe with 2 relevant columns.

+-------------+---------------+
| Bezeichnung | Artikelgruppe |
+-------------+---------------+
|     A       |       1       |
|     B       |       2       |
|     C       |       3       |
|     D       |       4       |
+-------------+---------------+

I want to paste the value of Bezeichnung into a new column (new_col) for all values of the column Artikelgruppe that are equal to 0.


texts: Parsing and storing a large amount of HTML data

texts:
I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch through this data chunk and extract some attributes out of each HTML page and store it in a DB for future use. Attributes to be extracted comprises of page titles and copyright section from the HTML.
As per my understanding this data is unstructured because there is no relation per say in the folder data for now. Moreover, there is a somewhat inherent structure which is of HTML but essentially each page mutually disjoint with the rest which qualifies for unstructured. Please correct me if I am wrong here.

Manager wishes to have the data stored in an ELK stack. By storing, he is quite unclear at this point in time but so far he wants to have the whole HTML file, title and copyright for each single HTML file extracted and stored. Here comes my first concern which I need help with.

Is it a good idea to store whole HTML file into DB? I am of the
opinion that we place HTML files in a centralized storage on some
kind of FS and store the absolute paths of those files against each
entry in the DB (we are already doing the same thing for PNGs btw).

I haven't worked with ELK stack and I thought it would be a good learning opportunity. While going through online tutorials I have learned that it is essentially for logs parsing from different applications servers and storing and visualizing them in a presentable and searchable manner.

If anyone can comment about ELK, if it will work in my case, that
would be very helpful.

So far the end objective is to crunch through this data and store the attributes and when required search through the attributes and use them as per future need. For example, if there is a specific copy right text that is coming up very frequently, then get that copyright text and use it for classifying certain pattern which takes to my third and last question.

Will it help to store it in a non relational database and then query
accordingly? In my opinion RDBMS like mysql is a better contender
because it will be easy to search through the tables for a specific
type of title and then use it accordingly. End goal is not
visualization, but to have data at hand to use whenever required.


texts: How to interpret this Plot of Model Loss from a BiLSTM model?

texts:
Hi everyone,
the above graph is produced by a BiLSTM model i just trained and tested. I can't seem to interpret it while it is very different from the references that i acquired by googling online.
The graph above has a plateau appearing at the very beginning of the value loss. Shall I set my epochs to smaller than 20?
My model is like this:
prepared_model = model.fit(X_train,y_train,batch_size=32,epochs=100,validation_data=(X_test,y_test), shuffle=False)

and how do you interpret it?
thank you guys.


texts: Cash-flow prediction with machine learning

texts:
In our company we have huge incoming cash flows every day. Having a very good prediction on the incoming cash flows for the next working day would be for various reasons very useful.

As for the data: I have every incoming payment of the last 2 years. Every transaction has the actual day of the payment, the day on which we expected the payment to be received, the amount of money paid and an unique identifier for the user.

Of course we are only interested in the aggregated payments for one day. Due to our business we have a strong seasonality around the first working day of the month and an even bigger peak for the first working day of the year.

How would you try to model something like that? I know that LSTM is often used for time series forecasting, but would that also be a good idea to try that out in our case? Or from experience do you have an idea which other model might be suitable for this prediction task?


texts:
I need to test different datasets as well as different algorithm implementations. The current workflow looks like:

Perform feature extraction from train set
Train classifier on this features
Feed this classifier to production code
Run production code on a test set, feeding samples one by one
Take results of production code and convert to the same format as test dataset
Show statistics

How to automate this as much as possible? I currently use a bunch of bash/python scripts. I've looked through sklearn pipelines, but not sure they would give any benefit except for running functions in a line.


texts: How implement a feed forward network with arbitrary node connections in keras?

texts:
I'm new to using the keras framework. I have read some examples about how to construct deep learning models with the Sequential and Graph classes in keras. However, I see that, independently if I use Sequential or Graph, it is assumed that each hidden unit of a layer is fully connected with all the hidden units of the other layer, isn't it?

If I want to construct a deep feed forward network that it is not fully connected, for instance the first hidden unit of the second layer is not connected to the second unit of the third layer...etc, even I want add connections (skip connections) between hidden units that belong to non-consecutive layers, how can I implement this in keras?


texts:
I have a situation where I need to propose a solution along with stack of technologies that need to be used for the below business case

Business Case:
I am receiving a car manufacturing data set from various car manufacturing companies, the data set for each car model is different and have different number of attributes but I do receive them in csv files every 2 minutes. I need to choose a particular storage format so that it is easy to query by each car model and based on the model, i select, i should be able to display the different features of that car model. Is there a recommended architecture for this type of data analysis?


texts: Using Classification Algorithms to Interpret Clustering Reuslts

texts:
Often clustering algorithms only output a bunch of class labels, and do not provide any sort of interpretation of the classes formed by the algorithm. It seems to me not entirely unreasonable to attempt to get some sort of interpretation by using the class labels provided by the clustering algorithm as the target in a supervised classification problem.

For a concrete example, say you cluster using k-means, and use a single decision tree classifier to predict the cluster based off of other features. The decision tree then should be able to give you some way of interpreting the clusters.

However, I can't find any literature (or blog posts...) mentioning this as a technique for interpreting the results of a clustering algorithm, which leads me to believe it is problematic. So in short:

Question: Can supervised classification algorithms be used to interpret the reuslts of unsupervised clustering algorithms? If not, why?


texts:
I want to learn how a spam email detector is done. I'm not trying to build a commercial product, it'll be a serious learning exercise for me. Therefore, I'm looking for resources, such as existing projects, source code, articles, papers etc that I can follow. I want to learn by examples, I don't think I am good enough to do it from scratch. Ideally, I'd like to get my hand dirty in Bayesian.

Is there anything like that that? Programming language isn't a problem for me.


texts: XGBoost incremental training for big datasets

texts:
I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred).
I have been trying to use some libraries such as Dask to deal with this problem, without any success due to high memory consumption (I opened a question on Stackoverflow, if you can answer, you are welcome).
Due to the above failure, I decided to implement my own incremental XGBoost training, something like what has been proposed here: XGBoost incremental training
The main idea is basically the following code:
params = {} # your params here
ith_batch = 0
n_batches = 100
model = None
while ith_batch < n_batches:
    d_train = getBatchData(ith_batch)
    model = xgb.train(params, d_train, xgb_model=model)
    ith_batch += 1

However I have three important questions:

Is this method of incremental training the same as training on the whole dataset? If not, why?
How should I split my batches at each iteration? Should I basically sweep the whole dataset sequentially?
How do the most common distributed/parallel compute libraries (Dask-ML, SparkML ecc...) train these models without storing everything in memory? Do they rely on incremental training?


texts: Feature selection in regression: are those features just correlated with outcome or are they causal?

texts: When we get our variable importance plots in linear or logistic regression, we know that the features with more importance are correlated with our outcome.  Are they necessarily causal?


texts: Query about Landmark pooling layer in Fashionnet paper

texts:
I have gone through Deepfashion paper.I have query regarding Landmark pooling layer.

1) Image has 4,6 or 8 landmark points depending on cloth type. So how do you decide size of last FC layer in landmark branch? This size will depend on landmark points but we have variable landmark points.

2) I have eight landmark points for an image and feature maps of conv4. I have to do max pooling around landmark l to get local feature map, similar to Fast-RCNN. But which point on feature maps of conv4 corresponds to landmark point l (x,y). Also what is the kernel size for max pooling?


texts: Predict the probability that a user is active on a website

texts:
I have been given two different arrays, which describe if a user is active on a site or not based on some features. The first array is for training. Each row of the array is a different user and the columns are different features. The first 896 columns describes the users using-patterns and the remaining 28 are lables (These 28 columns stands for 28 different time periods, they are labled 1 if the user is active in the time period, 0 otherwise.

The second array is for testing and has the same amount of columns, but here the last 28 rows are empty. My task is to predict the probability that a user will be active during each time period. I am a bit lost and are looking for (preferably quite simple) methods to solve this problem with.


texts: What it Would be easier Building a Deep Net From Scratch or Using an existing Architecture?

texts: In Practice with CNN what would be easier: Building a CNN from scratch or using a an existing architecture with some updates?


texts: Does overfitting depend only on validation loss or both training and validation loss?

texts:
There are several scenarios that can occur while training and validating:

Both training loss and validation loss are decreasing, with the training loss lower than the validation loss.
Both training loss and validation loss are decreasing, with the training loss higher than the validation loss.
Training loss decreasing, but validation loss is increasing.

I am aware that overfitting occurs in scenario 3, but does overfitting occur in scenario 1? If so, does this mean that overfitting only occurs when either scenario 1 or scenario 3 occur? Otherwise, if overfitting only occurs in scenario 3, does this mean that overfitting only occurs when validation loss is increasing?


texts: Difference between n-dimensional of length m and m-dimensional of length n?

texts:
With respect to classifying the MNIST digits I've seen the input representation referred to as both a one-dimensional array of 748 elements and a 784-dimensional vector.

Are these perspectives always interchangeable, or is there a material difference between the two?


texts: When training a neural network on a windows machine, what can task manager tell me about how it's going?

texts:
When I trained a simple convolutional network in pytorch, I could see in task manager that my cpu was hitting 100%, and I could see my ram spike up.

When running with CUDA, I could see my GPU, go from 0% to only 1-2% utilization.

I'm just curious about several things:

I only ever see the Copy graph going up with the other 3 graphs (3D, video encode, video decode) doing nothing. Is that normal? I would've thought lots of matrix operations would make the 3D go up?
I can easily max out CPU training, but I can never go above single digit % utilization for GPU training (and I don't even have a powerful GPU). I know it's because I probably have too simple a network, but besides increasing the batch size, are there ways to "take up slack"?


texts:
I trained a LGBMClassifier model and saved in a file in this way:

clf = lgb.LGBMClassifier( ... )
clf.fit(X_train, y_train, **fit_params)
clf.booster_.save_model("model1.txt")

##Predictions
y_pred = clf.predict_proba(X_data, num_iteration=clf.best_iteration_)[:, 1]

Now what I want is to use the saved model for another prediction. But if I do this:

## new predictions:
clf_fs = lgb.Booster(model_file='model1.txt')
y_pred2 = clf_fs.predict_proba(X_data2, num_iteration=clf_fs.best_iteration_)[:, 1]

I got an error as

AttributeError: 'Booster' object has no attribute 'predict_proba'

I understand that cls_fs is an object of class Booster and not of a class LGBMClassifier, and that I can use clf_fs.predict(), but how I can get back a LGBMClassifier object from the saved booster file and all its specific attributes?


texts: Keras - Implementation of custom loss function with multiple outputs

texts:
I am trying to replicate (a way smaller version) of the AlphaGo Zero system. However, in the network model, I am having a problem. The loss function I am supposed to implement is the following:

$$l = (z - v)^2 - \pi^T log(p) + c ||\theta||^2$$

Where:

$z$ is the label (a real value between -1 and 1) of one of the two heads of network and $v$ is this value predicted by the network.
$\pi$ is the label of a distribution probability over all actions and $p$ is the distribution probability over all actions predicted by the network.
$c$ is the L2 regularization parameter.

I pass to the network a list of channels (representing the game state) and an array (same size of the $\pi$ and $p$) representing which actions are indeed valid (by putting 1 if valid, 0 otherwise).

As you can see, the loss function uses both the target and the network predictions for the calculation. But after extensive search, when implementing my custom loss function, I can only pass as parameter y_true and y_pred even though I have two "y_true's" and two "y_pred's". I have tried using indexing to get those values but I'm pretty sure it is not working.

The modeling of the network and the custom loss function is in the code below:

def custom_loss(y_true, y_pred):

    # I am pretty sure this does not work

    output_prob_dist = y_pred[0]
    output_value = y_pred[1]
    label_prob_dist = y_true[0]
    label_value = y_pred[1]

    mse_loss = K.mean(K.square(label_value - output_value), axis=-1)
    cross_entropy_loss = K.dot(K.transpose(label_prob_dist), output_prob_dist)

    return mse_loss - cross_entropy_loss

def define_model():
    """Neural Network model implementation using Keras + Tensorflow."""
    state_channels = Input(shape = (5,5,6), name='States_Channels_Input')
    valid_actions_dist = Input(shape = (32,), name='Valid_Actions_Input')

    conv = Conv2D(filters=10, kernel_size=2, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='Conv_Layer')(state_channels)
    pool = MaxPooling2D(pool_size=(2, 2), name='Pooling_Layer')(conv)
    flat = Flatten(name='Flatten_Layer')(pool)

    # Merge of the flattened channels (after pooling) and the valid action
    # distribution. Used only as input in the probability distribution head.
    merge = concatenate([flat, valid_actions_dist])

    #Probability distribution over actions
    hidden_fc_prob_dist_1 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Prob_1')(merge)
    hidden_fc_prob_dist_2 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Prob_2')(hidden_fc_prob_dist_1)
    output_prob_dist = Dense(32, kernel_regularizer=regularizers.l2(0.0001), activation='softmax', name='Output_Dist')(hidden_fc_prob_dist_2)

    #Value of a state
    hidden_fc_value_1 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Value_1')(flat)
    hidden_fc_value_2 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Value_2')(hidden_fc_value_1)
    output_value = Dense(1, kernel_regularizer=regularizers.l2(0.0001), activation='tanh', name='Output_Value')(hidden_fc_value_2)

    model = Model(inputs=[state_channels, valid_actions_dist], outputs=[output_prob_dist, output_value])

    model.compile(loss=custom_loss, optimizer='adam', metrics=['accuracy'])

    return model

# In the main method
model = define_model()
# ...
# MCTS routine to collect the data for the network input
# ...

x_train = [channels_input, valid_actions_dist_input]
y_train = [dist_probs_label, who_won_label]

model.fit(x_train, y_train, epochs=10)

In short, my question is: how do I correctly implement this custom loss function that uses both the network outputs and label values of the network?


texts:
I was wondering about the possibilities of clustering numerical data (more than 3 dimensions) into different clusters and doing curve fitting on each cluster to get much higher accuracy than using a single model.

Since linear regression is preferred, is there any way to cluster the data points based on their linear fitting?

This is because I need a result matching the input data and do not care about unseen data. I cannot hard code the data and use a lookup mechanism. Instead an approximate math function would be preferable.

Is there an existing implementation ? (Preferably in Python)


texts: Negative range for binary cross entropy loss?

texts:
So based on the website:

https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a

The binary cross entropy loss is defined as:

Which is applicable for output range {0,1}. However, what if i scale the output to be now {-1,1} instead? How would the new cross entropy loss be derived?


texts: Find changes in variables into two states


texts: Python - Converting 3D numpy array to 2D


texts: pandas: How to impute the categorical column by the nearest neighbors?

texts:
I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in the neighborhood. In the beginning of the input signal you can see nans embedded in an otherwise continuum 's' episode. My definition as to what characterizes an episode is occurrence of the corresponding symbol at least 5 times in a row. Also, in my interest to be given more weight to 'r' and 'l' when tied with 's'.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
oput = ['s','s','s','s','s','s','s','s','s','s','s','s','s','r','r','r','r','r','r','r','s','s','s','s','s','s','s','s','s','s','l','l','l','l','l','l','l','l','l']

I tried knn as following but it is rather suitable for numerical column and also imputing nans with zeros.
I was hoping for some ideas how to tackle this problem.
from fancyimpute import KNN
knnimpute = KNN(k=5)

>>>x = np.array([0,np.nan,1,1,1,np.nan,2,2,2,2,np.nan,2,3,3,3,3,np.nan,1,1,2,2,np.nan,1,3,3,3,3])
>>>x2 = knnimpute.fit_transform(x.reshape(-1,1))
>>>x2
>>>
array([[0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [2.],
           [2.],
           [2.],
           [2.],
           [0.],
           [2.],
           [3.],
           [3.],
           [3.],
           [3.],
           [0.],
           [1.],
           [1.],
           [2.],
           [2.],
           [0.],
           [1.],
           [3.],
           [3.],
           [3.],
           [3.]])


texts:
Anyone know the best packages to build a decision tree in Rstudio v3.3?

I want to look at some data-driven segmentation for my data. I was thinking of doing chaid analysis (mainly because this is why I've done in the past).

I've looked around myself, but can't find the best package for this version!


texts: unable to plot contour/surf graph of cost vs parameters for multiple linear regression.In python

texts:
After running gradient descent I have three arrays theta1 theta2 and J all three of size num_iterationx1. I tried plotting contours using the contour function in Axes3D and 3d surface plot using surf but there is always a dimension error or no figure gets plotted.

Can anyone tell me how to plot this. In python


texts: Can clustering my data first help me learn better classifiers?

texts:
I was thinking about this lately. Let's say that we have a very complex space, which makes it hard to learn a classifier that can efficiently split it. But what if this very complex space is actually made up of a bunch of "simple" subspaces. By simple, I mean that it would be easier to learn a classifier for that subspace.

In this situation, would clustering my data first, in other words finding these subspaces, help me learn a better classifier? This classifier would essentially be an ensemble of each subspace's classifier.

To clarify, I don't want to use the clusters as additional features and feed it to a big classifier, I want to train on each cluster individually.

Is this something that's already been done/proven to work/proven to not work? Are there any papers on it? I've been trying to search for things like this but couldn't find anything relevant so I thought I'd ask here.


texts: Paper proving neural networks with more data are better in NLP than traditional methods

texts:
I'm looking for a paper that does some comparisons between neural networks (deep learning) and traditional methods in order to prove that DL usually performs better with enough data.
I know this is "known" fact but I'm struggling to find a good paper doing some research in this area.

Thanks!


texts: How to choose between different types of feature scaling?

texts:
The feature set for my multi-class multi-label classification task, using the MLPClassifier from scikit learn, contains mostly features where the values are in the same range of [0,1], but there are 3 out of 45 features where this isn't the case and feature scaling is required. So far I've tried out min-max normalization, mean normalization and z-score normalization on these features. However all scaling methods result in slightly different train and test performance and z-score standardization results in the fastest convergence but worst scores overall. To measure performance, Precision, Recall, F1 and MCC were used.
What is a decent strategy when choosing a type of feature scaling?


texts:
I hope this isn't too basic of a question, I'm banking on the Data Science site description being true where it says "...and those interested in learning more about the field". I'm not looking for programming help, just validation that machine learning could help me with a problem.

I'm trying to find all customer phone numbers in our databases. One database has a field with free-form comments from our customer service center. Here is an obfuscated snippet:

  multiple #'s 123-456-7890 and 2345678901...current account #2233445566

As you can see, this record contains two phone numbers and also a 10 digit account number. One of the phone numbers has dashes while the 2nd doesn't. Looking for parenthesis helps, but only finds a small set. There are also other 10 digit numbers that could look like a phone number but in fact aren't.

If I run a query to return all records with a 10 digit number formatted with dashes, I get thousands of records. If I check for any 10 digit number, I get tens of thousands. So manually scanning these records to validate accurate matches is not practical.

I'm wondering if I could build a machine learning model that I could train to accurately find phone numbers in this mess. When I say "accurately", I don't mean 100%, just better than standard SQL queries. If I can, I would use this going forward to parse new data that is created in this database.

It seems to me this problem could be a good candidate for machine learning. But I'm new to machine learning, and the research I've done so far talks about different scenarios that don't seem quite the same.


texts:
I have this CNN architecture:

I know how to calculate error for weights based on the output and update weights between output<-->hidden and hidden<-->input layers.

The problem is that I have no idea how to calculate delta for values in input layer based on the error and then use it in convolution backpropagation.