texts: Why results of statsmodel logreg is different from scikit-learn logreg?

texts:
I am trying to do a binary classification. I have only 6 input variables and one output variables. Label 1 is 1554 records and Label 0 is 3558 records.

As you can see below, the metrics that I get from these two are different. I am not sure what other info I can share more about this issue. I just tried to do the classification through different methods

# code for statsmodel logreg

model = smm.Logit(y_train, X_train_std)    #std indicates standardized inputs
result=model.fit()
result.summary()
y_pred = result.predict(X_test_std)
y_pred[y_pred > 0.5] = 1
y_pred[y_pred < 0.5] = 0
cm = confusion_matrix(y_test, y_pred)
print(cm)
print("Accuracy is ", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))
print("ACU score is ",roc_auc_score(y_test, y_pred))
print("Recall score is",recall_score(y_test,y_pred))
print("Precision score is",precision_score(y_test,y_pred))
print("F1 score is",f1_score(y_test,y_pred))

# code for scikit-learn

#log reg optimized parameters
op_param_grid = {'C': [0.01],'class_weight':['balanced'],'penalty': ['l1'], 'solver': ['saga'],'max_iter':[200]}
logreg=LogisticRegression(random_state=41)
logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')
logreg_cv.fit(X_train_std,y_train)

May I know why is this happening and which one should I rely upon?

Is it not right to expect the same results on both approach? Do they work differently?

Can anyone help?


texts: seaborn heatmap not displaying correctly

texts:
For some reason, my heatmap is not displaying correctly anymore. It was working just fine even with 6 classes. Since the last time I used it, I've installed many packages (including plotly). I don't know what exactly has caused this. How can I make the annotations and the x/y labels centered again? In both images the exact same code is used.
    import matplotlib.pyplot as plt
    import seaborn
    conf_mat = confusion_matrix(valid_y, y_hat)
    fig, ax = plt.subplots(figsize=(8,6))
    seaborn.heatmap(conf_mat, annot=True, fmt='d',xticklabels=classes, yticklabels=classes)
    plt.ylabel('Actual')
    plt.xlabel('Predicted')
    plt.show()


texts: Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM

texts:
I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer
IterativeImputer(max_iter=<num_iter>,
                 initial_strategy = 'most_frequent',
                 verbose=True,
                 estimator=ExtraTreesRegressor(n_estimators=100,
                                               min_samples_leaf=1,
                                               min_samples_split=4,
                                               random_state=0,
                                               n_jobs=-1))
imp.fit(X_missing)
imputed = imp.transform(X_missing)

running on an 8 core (16 thread) 32G, the run completed with 1 iteration but crashed due to low memory with 2 iterations
running on a cloud machine with 16 cores 128G, when running with 4 iterations it uses up 115G of ram, anything higher than that crashes with not enough memory
Does anyone know how to reduce the memory footprint of imputer?


texts: Predicting next element of a sequence given small amount of data

texts:
I have data of bank branches and amount of revenue they have generated in a month. The data looks like this:

I am tasked to find the expected revenue for the branch for the next month using machine learning. Initially I was planning to use LSTM networks for such analysis, but I doubt its possible with such small amount of data.
I personally think machine learning is an overkill for such task. What would be the most appropriate way to predict the revenue for next month? I thought about increasing the amount of data by treating every branch as equal and using the row corresponding to each branch as separate instance for training (but I doubt that is a correct approach).
Any advice would be appreciated


texts: How do attention mechanism in CNN for images?

texts:
I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the weights?
I would like to understand the application of that mechanism on images.


texts: Predicting financial data (choosing a model)

texts:
it is my first time doing something with financial data. I have a dataset with account numbers and some other information about each client (some clients span more than one row since we have info for each month in a different row).
I managed to clean and create some models, here are the confusion matrices, the classification reports and AUC:
Logistic regression
[[185847  62897]
 [     1   1061]]

             precision    recall  f1-score   support

     not buy       1.00      0.75      0.86    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.75    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.75      0.85    249806

AUC train = 0.9168592981611143
AUC test = 0.9150300677458543

Random Forest Classifier:
[[245503   3241]
 [   960    102]]

              precision    recall  f1-score   support

     not buy       1.00      0.99      0.99    248744
         buy       0.03      0.10      0.05      1062

    accuracy                           0.98    249806
   macro avg       0.51      0.54      0.52    249806
weighted avg       0.99      0.98      0.99    249806

AUC train = 0.9996866568080237
AUC test = 0.9139101966925902

Gradient Boosting Classifier:
[[184940  63804]
 [     3   1059]]

              precision    recall  f1-score   support

     not buy       1.00      0.74      0.85    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.74    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.74      0.85    249806

AUC train = 0.8800353734759541
AUC test = 0.8657829269466372

Voting Classifier (from all the three above):
[[211316  37428]
 [   213    849]]

              precision    recall  f1-score   support

     not buy       1.00      0.85      0.92    248744
         buy       0.02      0.80      0.04      1062

    accuracy                           0.85    249806
   macro avg       0.51      0.82      0.48    249806
weighted avg       0.99      0.85      0.91    249806

AUC train = 0.9987531510931085
AUC test = 0.9160262741936392

Since I do not have any experience I am not sure which model is producing better results. Can you help me understand which one and why? Thank you!


texts: How to compute document similarities in case of source codes?

texts:
I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token based document representation (TF-IDF or embeddings) and computing similarity on these but somehow I find this approach a bit clumsy. I want to detect "handprints" (characteristic comment and abbreviation style, folder structure, used 3rd party tools, order of elements in the code etc.) that seem out of the scope of this approach.

Moreover, I cannot find place for any proper machine learning here. Clearly, finding weights for the any future quantitative features would be nice but this similarity task is not classification/regression, so how to define the target? Clustering seems to be a better tool but we cannot define as many categories as potential authors.

Could you kindly suggest any more reliable method? Does any literature exist for this topic?


texts:
I've been exploring the use of XGBoost in many different applications. Up to now, I always find the best results with shallow trees (from 1 to 3 levels), with the rest of the parameters very dependent on the problem.
On my current assignment, I found that I get a much better performance if I use >300 trees, with a depth >20 !! I understand that this is saying a lot about the complexity of the issue, but I can stop wondering if there is some feature transformation I could do to change this. EX: by doing PCA and adding the resulting component to a dataset, one can sometimes not only replace several features by a smaller number but also capture higher level relashionships


texts: Mapping a set of corrupted strings to the correct ones

texts:
I am fairly new into Data Science but encoutered it before. The following problem troubles me and i hope you guys can point me in the right direction.

The input are some strings where some carry the same information others not. An unknow number of these strings are crooked* to a warrying degree. From only one letter off to complete garbage. On the output side are the corrected strings from the input. The catch is that there are only certain, already known, combinations of valid strings possible.

In a naive approach i chained some fuzzy searches and already got some promising results. Now i don't know where to start or if there are similar problems already solved.

* (are we still allowed to say this?)


texts:
Suppose I have these elements:

a = [1, 6, 3, 4, 10, 32, 2, 54]
b = [20, 5, 14, 25, 18, 1]
c = [54, 3, 6, 12, 41, 1, 9]
d = [3, 4, 1]
e = [19, 20, 25, 5]

Each identified by a list of ids (that actually represent a string but we'll use numbers for simplicity).

How can I cluster these by finding those who have the most elements in common?
For example:
a and d have 3 elements in common
b and e have 3 elements in common
a and c have 4 elements in common

So the clusters I would like to have ar (a, c, d) and (b, e).
I would like to obtain this in a Python script


texts: XGBClassifier error! ValueError: feature_names mismatch:

texts:
I have a data set given as follows:
target shape (200000, 1)
train_data.shape (200000, 48)
test_data.shape(100000, 48)
I had used the data to predict_proba using RandomForestClassifier ExtraTreesClassifier RandomForestClassifier AdaBoostClassifier GradientBoostingClassifier and SVC
and got results without an error.
BUT
clf = XGBClassifier() is showing following errors (with the same code except clf):
ValueError: feature_names mismatch:


texts: What is slowing down classic DBSCAN algorithm

texts:
How to apply CSR Matrix on DBSCAN algorithm in python without using any libraries?

Update: Matrix size (8580, 126356)

I have given a shot and implemented the algorithm. It runs rather slow. I guess its because regionQuery function calculates the euclidean distance  between a chosen point and every other point in the dataset.

Is there any way this problem can be solved?


texts: Reading a json-file from an API, getting "AttributeError: 'Response' object has no attribute 'data' "error

texts:
Title says it all.
Im trying to read a json file im accessing via an API. But when I try, I get the error:
'AttributeError: 'Response' object has no attribute 'data'
I don't understand why, as most (not all) of my fellow classmates do not get this error.
The code I'm trying to run:
[1] import folium
[2] import geopandas as gpd
[3] import pandas as pd
[4] import numpy as np
[5] import urllib3
[6] import requests
[7] import json
[8] import os

[9] dmi_api = 'f8654190-89bb-43d2-9e08-416346b79e7b'
[10] url = 'https://dmigw.govcloud.dk/v2/metObs/collections/observation/itemsstationId=06031&parameterId=temp_dry&datetime=2021-01-01T00:00:00Z/2021-03-01T00:00:00Z&api-key='+str(dmi_api)
[11] r = requests.get(url)
[12] print(r)
[13] print(url)

[14] data = json.loads(r.data.decode('utf-8')) #This is where the error is referring to (which leads back to line 11)
[15] data

What I don't understand is why the json.loads isnt working, as the link is obviously full of data.
I'm coding in Python 3.2 using the Jupyter Notebook accessed from Anaconda, if that's any help.


texts: Which convolution should I use? Conv2d or Conv1d

texts:
I've dataset which contains dlib landmark points of the faces. I'm using keras to train a model.
The dataset shape is (length_of_dataset,68,2). I know that I've two options.

The first is using conv1d with input_shape = (68,2).
The second is using conv2d with input_shape = (1,68,2).

My question is which one is better? And why?


texts: Reading a visualization of word embeddings

texts:
For my Masters Thesis, I created a Word2Vec model. I wanted to show this image to clarify the result. But how does the mapping works to display the words in this 2D space?

All words are represented by a vector of 300 dim. How are they mapped on this 2D image? What are the x & y scales?

Code:

documents = [_text.split() for _text in df_train.text]
w2v_model = gensim.models.word2vec.Word2Vec(size=W2V_SIZE,
                                            window=W2V_WINDOW,
                                            min_count=W2V_MIN_COUNT,
                                            workers=8)

w2v_model.build_vocab(documents)

words = w2v_model.wv.vocab.keys()
vocab_size = len(words)
print("Vocab size", vocab_size)

w2v_model.train(documents, total_examples=len(documents),

epochs=W2V_EPOCH)
tokenizer = Tokenizer()
tokenizer.fit_on_texts(df_train.text)

vocab_size = len(tokenizer.word_index) + 1
print("Total words", vocab_size)

x_train = pad_sequences(tokenizer.texts_to_sequences(df_train.text), maxlen=SEQUENCE_LENGTH)
x_test = pad_sequences(tokenizer.texts_to_sequences(df_test.text), maxlen=SEQUENCE_LENGTH)

labels = df_train.target.unique().tolist()
labels.append(NEUTRAL)

encoder = LabelEncoder()
encoder.fit(df_train.target.tolist())

y_train = encoder.transform(df_train.target.tolist())
y_test = encoder.transform(df_test.target.tolist())

y_train = y_train.reshape(-1,1)
y_test = y_test.reshape(-1,1)

embedding_matrix = np.zeros((vocab_size, W2V_SIZE))
for word, i in tokenizer.word_index.items():
  if word in w2v_model.wv:
    embedding_matrix[i] = w2v_model.wv[word]
print(embedding_matrix.shape)
embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)


texts: How to choose dimensionality of the Dense layer in LSTM?

texts:
I have a task of multi-label text classification. My dataset has 1369 classes:

# data shape
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
(54629, 500)
(23413, 500)
(54629, 1369)
(23413, 1369)

For this task, I've decided to use LSTM NN with the next parameters:

# define model
maxlen = 400
inp = Input(shape=(maxlen, ))
embed_size = 128
x = Embedding(max_features, embed_size)(inp)
x = LSTM(60, return_sequences=True,name='lstm_layer')(x)
x = GlobalMaxPool1D()(x)
x = Dropout(0.1)(x)
x = Dense(2000, activation="relu")(x)
x = Dropout(0.1)(x)
x = Dense(1369, activation="sigmoid")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy']
batch_size = 32
epochs = 2
model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)

Question: Are there any scientific methods for determining Dense and LSTM dimensionality (in my example, LSTM dimension=60, I Dense dimension=2000, and II Dense dimension=1369)?

If there are no scientific methods, maybe there are some heuristics or tips on how to do this with data with similar dimension.

I randomly chose these parameters. I would like to improve the accuracy of the model and correctly approach to solving similar problems.


texts: Combining 2D Detection with Disparity Maps to Learn 3D Object Geometry

texts:
Since the disparity map above is a representation of the object's distance from the camera's origin, is it reasonable to assume that a network (perhaps a convolutional LSTM) could be trained to perform a infer the depth of the object in the Z coordinate of camera frame?

As you can see above the brightness of the object increases as the car moves closer to it, and the size of the 2D bounding box also increases.

Thus the depth in the Z direction becomes a function of two things:

the change in the diagonal/size of the 2D bounding box

1.1. More or less the object's size as the car approaches it because it can also be moving across from that car as it is stopped at a street light.  In which case the 2D box is mostly constant in size.
the change of the brightness of the pixels inside the 2D box.

Before going down this rabbit hole, I was hoping to get feedback on the feasibility of this idea, what "gotcha's" there might be that are not immediately apparent.

A couple of things are immediately noticeable about why this would be too difficult:

The occlusion of parked cars on the right hand side there makes their shapes very hard to distinguish, although maybe the 2D boxes could help with that
It will be difficult to track the same object through time steps, and would probably require a SLAM network as well.

Any help is very much appreciated!


texts: Difference between Gibbs sampling and variational Bayes inference

texts:
After reading in blogs and books, I came to the conclusion that Gibbs sampling and variation Bayes are methods for estimating or inference of posterior. Below link described but it's difficult to interpret.

https://users.ics.aalto.fi/alexilin/papers/step10.pdf

It can be a sense that both methods are most probably use in a probabilistic model. Practically Gibbs sampling is very slow compared to the variational base.

Can anyone explain how this happens? It would be interesting if one will brief about it.


texts: Does the "laplacian" kernel used in the SVM context come from a Hilbert space inner product?

texts:
The "kernel" $k(x,y) = e^{-\|x-y\|} $ is used in the context of SVM. Here $x,y \in \mathbb R^n$ and $\|.\|$ is the Euclidean norm.

Is there a Hilbert space H and a map $\varphi:\mathbb R^n \rightarrow H$ such that $k(x,y) = <\varphi(x),\varphi(y)> $ for all $x,y \in \mathbb R^n$?


texts:
I have a random created dataframe named a:

  group marks  upd
1     a     2   up
2     b     3   up
3     a     4 down
4     b     5   up
5     a     6 down
6     c     7   up
7     a     8 down

I wanted to subset it with only group='a' and marks in decreasing order of a. I used this query but the marks are not arranged in decreasing order. What is the error in my code?

Here is my code and output:

> a[a$group=="a" & a$marks[order(a$marks,decreasing=T)],]
  group marks  upd
1     a     2   up
3     a     4 down
5     a     6 down
7     a     8 down

Warning message:
In a$group == "a" & order(a$marks, decreasing = TRUE)[1:3] :
  longer object length is not a multiple of shorter object length


texts: Text classification with thousands of output classes in Keras

texts:
Task:

I have a dataset with job titles and descriptions. The task is to predict tags for job by job title and description.

There are several tags for each job posting. Therefore, the number of labels for the model will be measured in tens of thousands.

Number of job postings = 78042

Number of unique classes (tags) = 1369

Questions:

Could you advise working types of neural networks (desirable in Keras)?

Or maybe you know how to solve this problem with the help of classical machine learning algorithms?

Still I will be very grateful for links to articles where similar problems are solved.


texts: How to add a new label to a multi-label dataset (like Open Images)

texts:
Given N classes in a multi-label dataset and a trained classifier C, how would we add a new class N+1 to the dataset, and fine-tune the trained classifier C such that it now predicts N+1 labels? (lets call this fine-tuned N+1 class classifier C'). Let's use Open Images as an example, which has 9,000,000 images in the dataset.

The issue is with the data and it being a multi-label dataset. Say we have 5000 images from the N+1 class.

The first issue is that there may be other objects from any of the N classes in the images besides class N+1 (these images are taken from the web). So we run the trained classifier C on the 5000 images and have machine annotations for the other N classes in the 5000 images. We then use turkers to verify all the labels.

Now the second (unsolved) issue is, how to find instances of the N+1 class in the original 9,000,000 training set? If we don't find these 'false negatives' in the original training set, fine-tuning could have unexpected results.


texts: What methods are available to evaluate similarity between different clustering algorithms?

texts:
I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure of the clusters and well as the number (7-8). I would like to know if there is a common method to either...

compare similarity between clusters. Can you apply Adjusted Rand Index to two different clusterings of the same data (k-means clusters vs gmm)? I was under the impression ARI is used in instances where you know the truth of the data.
Find the common clusters within the clusterings. If all of the algorithms say one cluster is defined by high spending, then is there a way to determine the best centroid(s) to use for a "Master" cluster? Is it common to cluster the cluster results?


texts: Is the _error_ in the context of ML always just the difference of predictions and targets?

texts:
Simple definitional question: In the context of machine learning, is the error of a model always the difference of predictions $f(x) = \hat{y}$ and targets $y$? Or are there also other definitions of error?

I looked into other posts on this, but they are not sufficiently clear. See my comment to the answer in this post:

What's the difference between Error, Risk and Loss?


texts:
I was reading this nice tutorial about Pytorch's basics:

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

In the first example (pure Numpy), the author starts the backward phase by setting as his "first" gradient the numerical result of the loss function calculation multiplied by the constant 2:

grad_y_pred = 2.0 * (y_pred - y)

Why does he multiply by 2?


texts: Validation Accuracy greater than train accuracy, validation loss lesser than training loss MTL

texts:
I am training a multi task model using VGG16.

Datase:

Dataset contain 11K images. There are two tasks:

The dataset is imbalanced,

1) PFR classification: 10 classes

0 --- 5776

10-12 --- 1066

6-9 --- 1027

4-5 --- 680

1-3 --- 518

30-40 --- 471

20-25 --- 430

13-15 --- 304

60+ --- 267

41-56 --- 160

2) FT classification: 5 classes

1 --- 4979

2 --- 2322

3 --- 1932

4 --- 1051

5 --- 415

Initially, i trained individual CNN model for both classifications. Without loss_weights, the accuracy for PFR task was 86% in individual CNN model and for FT task, the accuracy was 92% in individual model.

Then I tried using multi task learning and it produced following results:

The validation accuracy is greater than the training accuracy through out all the epochs and the validation loss is lower than training loss.

During test, the precision and recall for each class is between 0.80 - 0.10.

Is the validation accuracy higher because the model has dropout layers?

The model code is:

baseModel = VGG16(weights="imagenet", include_top=False,input_tensor=Input(shape=(img_size, img_size, 3)))
flatLayer = baseModel.output
sharedLayer = Flatten(name="flatten")(flatLayer)
sharedLayer = Dense(1024,name="Shared")(sharedLayer)
sharedLayer = Dropout(0.5)(sharedLayer)
task1 = Dense(512, activation="relu")(sharedLayer)
task1 = Dense(10, activation="softmax",name='PFR')(task1)
task2 = Dense(512, activation="relu")(sharedLayer)
task2 = Dense(5, activation="softmax",name='FT')(task2)
model3 = Model(inputs=baseModel.input, outputs=[task1,task2])


texts: Is it possible to use NEAT networks for solving video games?

texts:
Sorry to start such an unspecific question, but I am slightly lost in the big topic.

My tutor proposed to chose neural networks for my final project, and we started by building a CNN for detecting hand written digits in python.

My own idea was to build another neural network that can play video games of ascending difficulty, for example pong -> snake -> megaman [with tools for extracting in game data] or driving in GTA San Andreas similar to what Sentdex did. (For GTA, I would try something different to Sentdex as I would train the neural network by reinforcement learning in artificial conditions, starting with staying in a straight lane and then moving on to curves, traffic etc.)
I made my own version of snake, which I could use for additional input and seems like a realistic minimal goal.

But now, I am not so sure which approach I should take for this. I took a look at Sethblings MarI/O and read the NEAT paper. Then, I read the book "Machine Learning in Python" explaining a lot of concepts, and realised that neural networks and especially GA's are only a small part of possible algorithms. Also, I am not sure how much Sethblings network actually thinks and how much it just randomly adapted to the level.

Do you think that the following idea is realistic?:
"I build my own neural network in python based on NEAT, then test it on video games / tasks of increasing difficulty and try to tune parameters for performance"

I could also imagine comparing NEAT to an implementation of Alexnet for the same tasks.
The whole goal is not to create the perfect NEAT but instead just make one and see how far it takes me. I have a lot of time and motivation at hand, but I don't want to exceed the boundaries of a high school final and my own possibilities.

Feedback welcome and thank you in advance


texts: Clustering with arrays / vectors as features?

texts:
I am trying to cluster a large set of documents of which I have a DOC2VEC representation. But I want to cluster them with more features, thus resulting in having both a vector (numpy.asarray(doc2vec_for_document)) and single values as features.

Is this possible? I would like to try clustering with both K-means and DBSCAN of the SKLearn module in Python.

My dataset would look like this:
| Document | DOC2VEC | extra_ft1 | extra_ft2 | ...

Also, what if I had multiple vectors instead of 1 vector and many values?


texts:
I tried to run a project from repo and got the following log which, I believe, tells a problem with weights load.
python3 main.py
Traceback (most recent call last):
File "main.py", line 66, in <module>
yolo = YOLO()
File "/home/matalan/venv/survalance/Traffic-Survalance/ObjectDetection.py", line 32, in __init__
self.model = self._get_model()
File "/home/matalan/venv/survalance/Traffic-Survalance/ObjectDetection.py", line 39, in _get_model
return load_model(self.model_path)
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/saving/save.py", line 182, in load_model
  return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/saving/hdf5_format.py", line 177, in load_model_from_hdf5
  model = model_config_lib.model_from_config(model_config,
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/saving/model_config.py", line 55, in model_from_config
  return deserialize(config, custom_objects=custom_objects)
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/layers/serialization.py", line 171, in deserialize
  return generic_utils.deserialize_keras_object(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in
  deserialize_keras_object
return cls.from_config(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/engine/training.py", line 2238, in from_config
return functional.Functional.from_config(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/engine/functional.py", line 616, in from_config
  input_tensors, output_tensors, created_layers = reconstruct_from_config(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/engine/functional.py", line 1204, in
  reconstruct_from_config
 process_layer(layer_data)
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/engine/functional.py", line 1186, in process_layer
layer = deserialize_layer(layer_data, custom_objects=custom_objects)
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/layers/serialization.py", line 171, in deserialize
return generic_utils.deserialize_keras_object(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in
  deserialize_keras_object
return cls.from_config(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/layers/core.py", line 1005, in from_config
function = cls._parse_function_from_config(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/layers/core.py", line 1057, in
  _parse_function_from_config
function = generic_utils.func_load(
File "/home/matalan/.local/lib/python3.8/site-
  packages/tensorflow/python/keras/utils/generic_utils.py", line 457, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

I use python 3.8.5 and latests TF & Keras for cpu. Is there a way to fix it? (load other weights or smth)


texts: Machine Learning algorithm to get X and Y points ranges values


texts:
We have a data set that is converted from signal data to video. We want to classify these images using convolution. We tried many different methods but val acc is consistently low. Training accuracy is 99% and val_acc is 40%. We need your help in this respect. Thank you

weight_decay=0.0005
input_ = Input(shape=(125, 125, 1))
# Block 1
x = Conv2D(64, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(input_)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool1')(x)

# Block 2
x = Conv2D(128, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool2')(x)

# Block 3
x = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool3')(x)

# Block 4
x = Conv2D(512, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = Conv2D(512, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool4')(x)
x = GlobalMaxPooling2D()(x)

x = Dense(800,kernel_regularizer='0.001')(x)
x = Dropout(0.5)(x)
x = BatchNormalization()(x)
x = Dense(800,kernel_regularizer='0.001')(x)
x = Dropout(0.5)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Dense(8)(x)
x = Activation('softmax')(x)
model = Model(inputs = input_, outputs=x)

enter image description here

my kernel and dataset
https://www.kaggle.com/ultrasonraporlama/video-kernel/


texts: Noisy behavior in deep learning feed-forward net

texts:
I am a bit unsure about optimizing a neural net with 3 or more layers. The input data is quite noisy and I seem to project the noise into the learning (strong bias in the data, 90% belong to one class out of five).

However, I would like to get some feedback on the interpretation (I use 50/25/8/8/8neurons with dropout (keep_prob=0,9 after first hidden layer):


texts: how to interpret this 'lift chart'? prediction and true labels

texts:
i am trying to compare the prediction from my classifcation model and it's true label either 0 or 1.
y_true = df['status'].values

y_pred = df_pred['score'].values

cut, bins = pd.qcut(y_pred, q, retbins=True, duplicates='drop')

lift_df = pd.DataFrame({
            'bin': cut,
            'prediction_prod': y_pred,
            'ground_truth': y_true
        })

        # Plot
fig, ax = plt.subplots(figsize=(8, 5))
lift_df.groupby('bin').mean().plot(kind='bar', ax=ax, color=['#BBC7FF', '#6882FF','#8168FF']
        )

here i think what this is showing is that our model (predicted mean score) is over predicting for each bucket/decile in the bar below, whereas in reality the actual score  (ground truth) is low. in an ideal model the two bars would be of equal length. would you agree on the interprettion?


texts: BERT for classification model degenerates into all-positive predictions

texts:
As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'm not sure what is going wrong with my code. Can anyone provide any help or insight into why this happens?
Technical details
I'm using hugging face's transformers library with PyTorch.
The (stripped version of the) code is as follows. Instrumentation and other details have been left out so the code is more readable.
tokenizer = transformers.AutoTokenizer.from_pretrained('bert-base-cased')

model = transformers.AutoModelForSequenceClassification.from_pretrained(
    'bert-base-cased',
    num_labels=2,
)

optimizer = transformers.AdamW(
    model.parameters(),
    lr=2e-5,
    eps=1e-8,
)

# The next lines read the CoLA dataset and split it for training and validation
training_dataloader = ...
validation_dataloader = ...

for epoch in range(4):
    train_loss = 0

    for batch in tqdm(train_dataloader):
        model.train()

        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['label'].to(device)

        model.zero_grad()

        model_output = model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            labels=labels,
            return_dict=True,
        )

        batch_loss = model_output.loss.sum()

        train_loss += batch_loss.item()

        batch_loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

        # HERE: Measure performance after learning from each batch here with validation_dataloader

I've checked and the code that reads the dataset seems correct, so it seems that the problem lies either in measuring the performance of the model or in the learning phase.
To measure the performance, I'm running the model over the validation split and then converting the logits into actual classifications as follows:
tp = fn = fp = tn = 0

for batch in validation_dataloader:
    input_ids = batch['input_ids'].to(device)
    attention_mask = batch['attention_mask'].to(device)
    labels = batch['label'].to(device)

    model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        labels=labels,
        return_dict=True,
    )

    expected = batch['label']
    predictions = output.logits[:,1] > output.logits[:,0] # Is this correct?

    tp += sum(1 for exp, pred in zip(expected, predictions) if     exp and     pred)
    fn += sum(1 for exp, pred in zip(expected, predictions) if     exp and not pred)
    fp += sum(1 for exp, pred in zip(expected, predictions) if not exp and     pred)
    tn += sum(1 for exp, pred in zip(expected, predictions) if not exp and not pred)

Is the line predictions = ... correct?


texts: Create multiple matrices from 2 bigger ones in R

texts:
I have 2 matrices, A (1000x21) and B (1000x7). Matrix A has individuals(=1000) in the rows and their consumption in 21 days at the columns. Matrix B has the SAME individuals(=1000) in the rows and some weights for each day of the week(=7) in the columns.

What I would like to have in the end is 1000 (with the dimension of 2x21) matrices  (one for each individual), lets call them $X_{i}$. In the first row of each $X_{i}$ I would like to have the consumption of the individual $i$ each of the 21 days (this will come from matrix A), and at the second row of the $X_{i}$ I would like to have the respective weight of that day (this will come from matrix B).

So matrix A looks like $[cons_{1,1} \ cons_{1,2} \ ... \ cons_{1,21} \\
                        \ cons_{2,1} \ cons_{2,2} \ ... \ cons_{2,21} \\
                                     . \\
                        . \\
                         . \\
                       \ cons_{1000,1} \ cons_{1000,2} \ ... \ cons_{1000,21}] $

Matrix B looks like $[weight_{1,1} \ weight_{1,2} \ ... \ weight_{1,7} \\ \ weight_{2,1} \ weight_{2,2} \ ... \ weight_{2,7} \\ . \\ . \\ . \\ \ weight_{1000,1} \ weight_{1000,2} \ ... \ weight_{1000,7}]$

And I would like the matrix $X_{i}$ to be like $[cons_{i,1} \ cons_{i,2} \ ... \ cons_{i,21} \\ \ weight_{i,1} \ weight_{i,2} \ ... weight_{i,k}]$

Any ideas how to do this in R in a loop ?


texts: How is an ASR's output compared to ground truth for validation?

texts:
I am curious how it is done as I am interested in doing something similar. I have some manually transcribed data that contains tags for multiple speakers. I want to compare how well the out of the box ASRs (Google, AWS Transcribe) are able to diarise speakers (or in simple words identify and transcribe audio with multiple speakers). I want to compare it to the ground truth data I have and come up with a comparison metric.
I can use Levenshtein Distance or something like Ratcliff-Obershelp similarity as a metric. But I am trying to learn if there is a more standard way of doing this?


texts: Normalizing dependent feature by one of the independent ones

texts:
I have a data set with three different features (x1, x2, x3) and I am going to use a regression model to predict y based on the features.

x3 is the total amount of money that a customer invest and y is the interest for that customer. So, for all cases y < x3. x1 and x2 are other information about the customer.

Rather than predicting y, I tried to predict y/x3 and I got much better results.

In my new model, I have tried to predict y/x3 = f(x1, x2, x3) using XGB model. The reason that I get better results is because of the reduction in the skewness in the data.

Do you think that this leads to data leakage? Please note that I did a lot of feature extraction based on x1, x2, x3, and use them for both models.


texts: Feature selection with linear interaction between variables and correlation with categorical response variable

texts:
I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable.

I found mRMR [1] but I am unsure on how it behaves with variable interactions (I think it works better on single variables than on their combinations).

Specifically I am searching for an algorithm that is able to retain all variable that are relevant when linearly combined one with another (e.g. summing/subtracting their value). I cannot combine them first and then run mRMR afterwards since the number of combinations is huge.

[1] Peng, Hanchuan, Fuhui Long, and Chris Ding. "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy," IEEE Transactions on pattern analysis and machine intelligence 27.8 (2005): 1226-1238.


texts: Any Neural Network implementations that allow for a cost function of more than just the network output?

texts:
I have an application of a straightforward MLP, for which the cost function is a function of both the network output, in addition to another value calculated from the network weights (actually the partial derivative of the output with respect to one of the inputs).

Are there any flexible NN packages where I can code this? The only ones I've seen (Keras, nnet in R) require that the cost function is just parameterised by the network output and the target value.


texts: Need some help on data mining platform recommendation

texts:
Currently I need one data mining platform (or solution) meets such requirements:

Data Source

I can import data from: CSV, MySQL, PostgreSQL, HDFS.

Optional: NoSQL, MongoDB.

Interface

I can assemble models and datas with drag & drops (like Weka).

Algorithms

I can model with common machine learning algorithms: Logistic Regression, Random Forest, SVM, word2vec, etc.

It would be awesome if it had deep learning support.

Optional: PMML

Customization

I can implement my own algorithms (under certain standard).

Optional: I can implement my own APIs (under certain standard).

Programming language is not limited.

Visualization

Simple visualization is okay.

License

Commercial solution is acceptable, open-source solution would be better (I can do some tweaks in order to meet my requirement).

There are two software I've investigated: Weka and SPSS Modeler. Weka doesn't support HDFS, while SPSS Modeler doesn't support customized algorithms.


texts: Confusion Matrix - Get Items FP/FN/TP/TN - Python

texts:
After run my python code:

print(confusion_matrix(x_test, x_pred))

I get this:
[100 32
211  21]

My question is how can I get the following list:

True positive = 100
False positive = 32
False negative = 211
True negative = 21

Is this possible?


texts: unsupervised anomaly detection on sparse data

texts:
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
 Feature_A  Feature_B  Feature_C....Feature_Z
 0.3            0       0.1            0
 0.5            0.5     0              0
 0              0       1.0            0
 1.0            0       0              0
 0.7            0       0              0
 1.0            0       0              0
 0.1            0       0.22          0.43

what is the best way to perform unsupervised anomaly detection on this kind of data? my initial idea was to perform some kind of dimensionality reduction first (e.g SVD or NMF) then do a simple anomaly detection technique on the resultant dense matrix (e.g Isolation Forest) but I'm not sure this is the best way to go.


texts: Can we generate huge dataset with Generative Adversarial Networks

texts:
I'm dealing with a problem where I couldn't find enough dataset(images) to feed into my deep neural network for training.

I was so inspired by the paper Generative Adversarial Text to Image Synthesis published by Scott Reed et al. on Generative Adversarial Networks.

I was curious to know that, can I use available small dataset as an input to a GAN model and generate a much bigger dataset to deal with deeper network models?

Will it be good enough?


texts: Neural Networks - Loss and Accuracy correlation

texts: I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the application of the two redundant in the training epochs? Moreover, why aren't they correlate?


texts: Why is my data not showing valid autocorrelation?

texts:
So i was following a kaggle notebook on weather prediction using ARIMA/SARIMA (here). I implemented the steps to look for autocorrelation on my dataset and found that the graph is in very strange format. Since, it is a acf plot it should have swings from -1 to 1 but in my case it is just a straight line like graph.
I got this:

My code implementation is here.


texts: Character recognition neural net topology/design

texts:
I'm building a neural net for classifying characters in pictures. The input can be any character a-z, lowercase and uppercase. I only care about classifying the characters, and not the case, so the neural net has an output vector of length 26; one for each character. It makes sense, intuitively, to have a hidden layer of size 26*2 just upstream of the output layer. It also makes intuitive sense for this layer not to be fully connected to the output layer, but instead having two and two hidden nodes connect to each output node.

I have some questions:

a) Does this make sense? I'm getting about 75 % success rate on a pretty hard data set with just one hidden layer, but I'm not certain on how to improve from there.

b) If so, What activation function should I use from the hidden layer with 26*2 nodes to the output layer? Maybe I should use an OR function for this, since both the lowercase and the uppercase version of a character should output for a single character.

c) Would it be wiser to have 26*2 output nodes instead, and just combine lowercase and uppercase outputs after the neural net?


texts: Word2Vec how to choose the embedding size parameter

texts:
I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions are somewhat related to different, independent "concepts" that a word could be grouped into. But beyond this I can't find any decent heuristics for how exactly to pick the number. There's some discussion here about the vocabulary size: https://stackoverflow.com/questions/45444964/python-what-is-the-size-parameter-in-gensim-word2vec-model-class  However, I suspect that vocabulary size is not most important, but more important is how many sample documents you have and how long they are. Surely each "dimension" should have sufficient examples to be learnt?

I have a collection of 200 000 documents, averaging about 20 pages in length each, covering a vocabulary of most of the English language. I'm using the word2vec embedding as a basis for finding distances between sentences and the documents. I'm using Gensim, if it matters. I'm using a size of 240. Is this reasonable? Are there any studies on what heuristics to use to choose the size parameter? Thanks.


texts: Plot a training/validation curve in Pytorch Training

texts:
I have the following training method and I'm confused how may I modify the code to plot a training and validation curve history graph with matplotlib
def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):
    """returns trained model"""
    since = time.time()

    # initialize tracker for minimum validation loss
    valid_loss_min = np.Inf
    train_loss_min = np.Inf

    for epoch in range(1, n_epochs + 1):
        print('\n')
        print('Running Epoch ', epoch, ' of ', n_epochs)
        print('\n')
        time_elapsed = time.time() - since
        print('Training for {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
        # initialize variables to monitor training and validation loss
        train_loss = 0.0
        valid_loss = 0.0

        # train the model
        model.train()
        for batch_idx, (data, target) in enumerate(loaders['train']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## find the loss and update the model parameters accordingly
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
            ## record the average training loss
            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
            if batch_idx % 10 == 0:
                    print('Epoch ', epoch, ' Training batch ', batch_idx)
                    print('Train Loss ', train_loss)

        # validate the model
        model.eval()
        print('\n')
        print('             Evaluating the model')
        print('\n')

        for batch_idx, (data, target) in enumerate(loaders['valid']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## update the average validation loss
            output = model(data)
            loss = criterion(output, target)
            valid_loss = valid_loss + ((1 / (batch_idx + 1)) * (loss.data - valid_loss))
            print('Valid Loss ', valid_loss)
        # print training/validation statistics
        print('\n')
        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch,
            train_loss,
            valid_loss
        ))
        print('\n')
        ## save the model if validation loss has decreased
        if valid_loss <= valid_loss_min:
            print('Validation loss decreased ({:.6f} --> {:.6f}). Saving model..'.format(valid_loss_min, valid_loss))
            torch.save(model.state_dict(), save_path)
            print('Model Saved')
            valid_loss_min = valid_loss

    time_elapsed = time.time() - since
    print('Training completed in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))

    return model


texts: SGD converges at exactly the same val accuracy for differnet NN model architecture

texts:
Context:
Supervised learning binary classification task with input vector of length 136.
Using MLP with Dense layers with variations in number of layers, neurons, learning rate, l1, l2 regularization, dropout, etc.
Observation:

Using SGD optimizer, validation accuracy always converge at exactly 0.7622 regardless of
configuration and hyperparameters, as shown in tensorboard below.
Experiment was repeated using ADAM optimizer. Convergence and model performance varies with said model design and hyperparameters as expected.

Question:

Is this normal? If not, any idea on what/how to debug?


texts: What to choose: an overfit model with higher evaluation score or a non-overfit model with lower one?

texts:
For lack of a better term, overfit here means a higher discrepancy between train and validation score and non-overfit means a lower discrepancy.

This "dilemma" just showed in neural network model I've recently working on. I trained the network with 10-fold cross-validation and got overfitted model (0.118 score difference):

0.967 accuracy for training set and
0.849 for validation set.

Then, I applied dropout layer with dropout rate of 0.3 after each hidden layer and got "less overfitted" model (0.057 score difference):

0.875 accuracy for training set and
0.818 for validation set

which is supposedly good since have lower discrepancy thus have better reliability for unknown data. The problem is, it has lower validation set score. My uninformed intuition says that no matter how overfitted your model is, validation set score is what matters because it indicates how well your model sees new data, so I choose the first model.

Is that a right intuition? How to go for this situation?


texts: Forecasting Foreign Exchange with Neural Network - Lag in Prediction

texts:
I have a question regarding the use of neural network. I am currently working with R (neuralnet package) and I am facing the following issue.
My testing and validation set are always late with respect to the historical data. Is there a way of correcting the result?
Maybe something is wrong in my analysis

I use the daily log return

I normalise my data with the sigmoid function (sigma and mu computed on my whole set)
I train my neural networks with 10 dates and the output is the normalised value that follows these 10 dates.

I tried to add the trend but there is no improvement, I observed 1-2 days late. My process seems ok, what do you think about it?


texts: correct value of output_dim or units parameter of dense

texts:
I have started learning deep learning and using keras library. But I am confused as to how to take a proper estimate of the value to use for units parameter of the dense method. I came across this tip that we can take it as the average of the number of input nodes and output nodes but everywhere it says that it comes from experience. I want to know if there are things to look out for to estimate it wisely or any other things I need to know. Thanks for the help.

classifier.add(Dense(units=6, kernel_initializer="uniform", activation="relu", input_dim=11))


texts: combining trained neural nets in tensorflow

texts:
I am using tensorflow to train two instances of the same neural network with two different datasets. the network itself is quite simple with an input and output layer and 6 hidden layers (each layer is a 20 meurons followed by a non-linear activation function).

I can train the network with two different datasets and that is fine. Now, what i want to do is basically create a new network which is a combination of these two trained networks. In particular, I want the input and the first 3 layers to be from one of the trained network and the last 3 layers and the output layer to be from the other network. I am very new to tensorflow and have not found a way to do this. Can someone point me to the API or some way to do this sort of hybrid networks?


texts: how can we feed both data time series and non time series data together in machine learning classification model

texts: I have a dataset(IoT wearable dataset) composed of time-series and integer data; the objective of my task is to use the dataset for classification. Whilst current libraries in sktime accept multivariate time-series data as input, I do not believe they accept integer data. What would be the best practice in this case? I am currently considering converting the integer data time-series by repeating the integer value for the same number of increments as the other multivariate inputs, then applying random noise to the manufactured integer data that I converted to Timeseries- Can anyone comment on whether this is an appropriate method or if there are other more appropriate ways to approach the issue? Thanks in advance.'


texts: Machine learning with constraints on features

texts:
I am working on a learning to rank problem. I have queries and documents related to every query which I have to rank. I used lightgbm ranker to fit the model. Some of features are very important and if they are changed the fitted model predicts a better score for that document and thus a better rank.
Lets say, for a single query id, I have a group of documents d1....d5 each having features f1...fn.
I change the features f1,f2,f3 for document d4 and it pushes the rank of d4 from 4 to 3.

What should be the values of these features f1,f2,f3 that can push the rank for document d4 from 4 to 1 subject to the condition that f1+f2+f3=45 is always satisfied ? (i.e. the sum of new modifications for f1,f2,f3 should be always equal to 45 )

As far as I understand this problem is simply a constrained optimization problem. In such a problem we have an objective function and constraints. In my case I do not have a mathematical equation that relates rank to my features r=f(f1....fn) but I do have a learned model which I can use as an objective function.
Follow up question
All this made me think of another question. In the case of OLS, which is a constrained optimization problem, the constraints present in this kind of optimization are always on the coefficients.

If I would like to have constraints on the predictors does it really make sense ? How does it relate to my problem ?

I think the objective function of a regression problem and the objective function of constrained optimization problem i mentioned above are different.
I would really appreciate some thoughts on this, thanks !


texts:
I have a regression problem and I am in doubt about how I can calculate RMSE in my life-cycle.

I deal with time-series and for every prediction, I want to look N points in the future. It is apparent how to calculate RMSE for a single iteration. My question is how to calculate RMSE for N predictions of N points to get a meaningful prediction performance metric.

I guess, I can average RMSE of all iterations though as I said I am not sure at all if this would reflect actual performance.


texts: Dynamic pricing models in freight transportation (logistics) business

texts:
I'm not sure this could be an appropriate question for here.

I'm a newbie in the field of data science.
I'm looking for keywords which can guide me to search the results to implement what I want to achieve.

Here are brief requirements that I want to implement.

I'm developing and maintaining digital logistics platform in which we
mainly be given freight transportation orders from clients and find
right freight carrier to deliver the freights with proper fare (e.g., Uber Freight).
To decide how much the fare is appropriate in terms of finding carriers on right time and maximize our revenue.
Based on our years of working experience, the fare of freight transportation is affected by lots of factors such as 'the distance between load place and unload place', 'the weight and size of the freights', 'seasons', 'the time of transportation (night transportation fare is higher than day)', 'the carrier(driver)'s preferences of load/unload places' and etc.
We have lots of data that contains final fare and factor variables to use for dynamic pricing.

What I want to ask is some keywords to research modeling techniques or methodologies to approach.

If the question is inappropriate then please let me know to revise the questions.

Thank you very much in advance.


texts:
I have a data set, in excel format, with account names, reported symptoms, a determined root cause and a date in month year format for each row. I am trying to implement a mahout like system with a purpose of determining the likelihood symptoms an account can report by doing a user based similarity kind of a thing. Technically, I am just hoping to tweak the recommendation system into a deterministic system to spot out the probable symptoms an account can report on. Instead of ratings, I can get the frequency of symptoms by accounts. Is it possible to use a programming language or any other software to build such system?

Here is an example:

Account : X Symptoms : AB, AD, AB, AB

Account : Y  Symptoms : AE, AE, AB, AB, EA

For the sake of this example, let's assume that all the dates are this month.

O/P: Account : X Symptom: AE

Here both of them have reported AB 2 or more times. I could fix such number as a threshold to look for probable symptoms.


texts: Confusion Matrix and AUC in univariate Anomaly Detection

texts:
In the code I upon a csv file which only has one column. The data in there in not that important just normal numbers.

# Pandas - data handling
import pandas as pd

# Numpy for mathematical operations
import numpy as np
import pandas as pd

# Scikit learn for the DBSCAN algorithm
from sklearn.cluster import KMeans

# Matplotlib - plots
import matplotlib.pyplot as plt

import seaborn as sns

data=pd.read_csv('C:/Users/data.csv', usecols=["meaning_txt", "valueData_value"], sep=';' )

# Add manually an anomaly value
for i in range(0, 50):
    all_data = all_data.append({all_data.column.max() + np.random.randint(1000, 2000) }, ignore_index=True)

clf = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=1)

clf.fit(all_data.column.values.reshape(-1, 1))

all_data_prd = clf.predict(all_data.column.values.reshape(-1, 1))
all_data_prd

plt.scatter(all_data_prd, all_data.valueData_value, c=all_data_prd)

My question is if it is possible to have a confusion matrix and roc curve in this case when I only  have once column.

To this column I have added some anomalies.

The goal:

Add these anomalies that are way way bigger than the maximum value
of    the column and then check how many out of the 50 added
anomalies, how    many the K-Means finds
Add these anomalies that are not that much bigger than the maximum
value of the column then check how many out of the 50 added
anomalies are found. At the end I want a Confusion matric and a ROC
graph to see how good they perform.


texts: Pre trained vehicle detection from low altitude view

texts:
I have a very noob question. I need some solution to detect all vehicles in the video (or at least in the image).
I have tried YOLOv3, however, it cannot recognize cars from the top (like that in the photo). Thought it works perfect with street view images.
Any advices on what can I do to detect cars from the top?


texts:
If anyone is there to answer these, that'll be great. I'm in the midst of a Final Year Project on LSTM.

Currently, I’m stuck and confused over LSTM codes. There are 4 hyperparameters that I can play around with:

Look back
Batch size
LSTM units
No. of Epochs

Can you explain what will happen to my results if I tune each of these hyperparameters? And also is it common if we get different results each time we run the codes?


texts: First of all thank you for taking your time to read my question. I have done a Machine Learning model with a dataset (The famous one about Cancer) and I want to know how can I do to predict the results for new variables. I think that I have to keep training the data (often) to have more accured data to use in my prediction but for predicting new data, ¿Is as simple as changing the test data (y variable) to the new data? Thank you so much for taking your time and any help would be appreciate it.


texts: What classifier is the best to determine if object was detected in the correct position?

texts:
I have a dataset which contains information about objects from 20 different categories. These objects are manually labeled on 100 images. The objects are located in 3 rows on a square image, there can be multiple instances of the same class object. For a given object, I have the following data:

its class
bounding box center coordinates
bounding box size
information whether the object is in the correct position or not

The ultimate goal is to have a classifier, which determines if the position of the detected object is correct. I am not sure if I should treat a label or "correctness" as an output, what would you suggest? Also, what are the best candidates for classifiers in case of this data, given the following:

Each training image (with labelled objects) is warped by a small angle, perspective, translations etc.
There is one correct arrangement of objects on the image

Please look below for simplified illustration of the data:
number - label,
bold - incorrect position

===================

1 1 1   3 3  5

6  7   1 1  4 4

9 5 5 5 9 7 7 7

===================


texts: location of the resampled data from SMOTE

texts: I am using SMOTE in Python to perform oversampling of the minor class in an unbalanced dataset. I would like to know the way SMOTE formats its output, that is, whether SMOTE concatenates the newly generated samples to the end of the input data and returns that as the output or whether the new synthetic data points are positioned randomly among the input data points. I'd appreciate your help.


texts: Does Tensorflow allow recurrent connections across layers?

texts:
What i mean is: Allow any arbitrary layer to receive outputs from a layer downstream from a previous timestep as input. The network graph would be cyclic. I do not mean explicitly recurrent layers like LSTMs, but recurrent connections between layers.
The Transformer model is fed its past outputs as inputs, so it must be possible to some degree. How would one implement something like that in tensorflow? And specifically for the transformer model that doesn't use recurrence except for seeing its past outputs as inputs, does backpropagation propagate errors through time when training such a model, or does it stop at the input/past output?
And beyond providing past outputs as inputs to a model, does tensorflow enable cyclic network graphs at all?


texts:
I am working with dates.

My fields are in order table:

Start date
close date
order no.

I have to show open orders in each month.

I have created relationship with date table by start date.
I need to relations like:

If start date is in this month and before this month and end date is this month means the order is closed in this month itself
it should count 0
If start date is in this month or before this month and close in month after this month
it should count 1

Now i have created measure like

if(month(startdate)<=month(date),
  if(month(closedate)>month(date),1,0),0)

It assigns 1 for the condition and then I use sum of all ones in visual for every month. Now this works fine with current start date but
for order with startdate before this month its not showing the data
as start date connected to date field in calendar table, when i put this in visual. It not showing the start date for previous months.
For example: If an order having start date in 7th month and end date in 9th month it should be in, 7th month and as well as in 8th month.
Right now its showing only in 7th month just because of relationship, I guess.

Also, its not evaluating the year changes. like its not capable for comparing data for December '19 and January '20


texts: Exploratory Data Analysis with Image Datset

texts: In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical analyses do you conduct, what kind of plots do you draw and what aims do you have in mind?


texts: Feature importance parameter in machine learning models like Naive Bayes

texts: Sorry for vague heading for the question. My question is that, is there any way to compare features (or attributes) used in machine learning algorithm? I have used Naive Bayesian classifier for binary classification which consists of total 6 features. I want to compare this features with one another and also list out features in priority order of their significance in models(how important that feature is). For example we use information gain or gini index to decide how effective feature is in CART. How can I compare this in naive bayes? I have read about confusion matrix and f measure but they are good at defining goodness of models and not features.


texts: Machine Learning: Single input to variable number of outputs

texts:
Is there a machine learning algorithm that maps a single input to an output list of variable length? If so, are there any implementations of the algorithm for public use? If not, what do you recommend as a workaround?

In my case, the input is a single scalar and the output is a list of scalars with variable length. For example, suppose I wanted to output a list of ones given the length of the list as input. Then <input, output> could be <1, [1]>, <2, [1, 1]>, etc. A small tweak would be providing the square root of the length in which case <2, [1, 1, 1, 1]> would be an answer. Note: the input need not be tied directly to the output.

For a more complex example, suppose I want to learn the look-and-say sequence. Valid <input, output> pairs would be: <1, [1]>, <2, [1, 1]>, <3, [2, 1]>, <4, [1, 2, 1, 1]>, <5, [1, 1, 1, 2, 2, 1]>, etc. My problem is also similar in that I can generate more examples; I am not restricted to a finite set of examples.


texts: How did you evaluate training loss on 100GB of data in deep learning?

texts:
Background:
In my problem, deep learning is trained on on 100GB data.
Question:
In order to monitor the training error versus epochs, we need to evaluate the training loss on our datasets, but does that mean we have to forward pass 100GB data in order to get the exact metric? What do people do in reality?
You don't want evaluate 100 GB data every 100 epochs....Maybe some random sampling?


texts:
I was going through this Udacity video (part of the ML nano-degree), where the math behind maximizing the length between the data clouds of two classes is done.

So, the line closest to the +1 class cloud would be: $w^T x_1 + b = 1$

And, the line closest to the -1 class cloud would be: $w^T x_2 + b = -1$  ($x_1$ and $x_2$ being points of both the lines respectively.)

So, for getting the distance between both of them (the lines above), the professors subtract both of the equations.

And then, while finding ($x_1-x_2$), they divide both the sides with the norm of w ($||w||$), which starts at 2:24 minutes.

From then, I didn't understand the math in the video.

Why did the professors divide the sides by the norm?  What do the norm specify/imply?

Can someone please explain the math from after 2:24?


texts: How to scale numbers according to proportions

texts:
I have some data that I'm trying to represent in a visualization with a week-over-week change. So the source can be something like:

Fruit    This Week   Last Week  Delta   Proportion
--------------------------------------------------
Bananas     10           8         2        25%
Oranges     110         100        15       15%
Apples      400         390        10      2.6%

What are some ways that I can effectively show that even though Bananas increased by 25% (technically the "biggest mover" of the week) it's an increase that is "overstated" because of the small overall numbers of bananas sold? I think I need a scaling approach of some sort but can't quite figure it out.


texts: Why exactly using a test set for model evaluation is a bad idea?

texts:
I don't understand why using the test set for model evaluation is a bad idea.

I completely understand why you should not use your test set to train your model (because in that case, you would be memorizing and you just cannot tell whether your model will generalize well or not if you don't have a separate test set). But why is it that simply using your test set to test (not train) your model is bad? You won't be changing any parameters of the model (because you are not training).

For instance, at the end of this video, Luis says we are breaking what he calls the "Golden rule" (i.e. never use your testing data for training). However, all I can see he is doing is using the test set to verify which model performs better to then be able to make a selection on which model he will use in the end.


texts: Gini index as a labeling strategy for leaf nodes

texts: Can we use the gini index to assign a class to a leaf node? If yes how? As far as I understand the gini index can only be used as a splitting metric.


texts:
I have hand writed classifiers (there are a lot of them). It's implemented as collection of rule sets IIF - THEN.
I want to optimize the % of errors. There some classifiers witch have vey big % of False Positive and False Negative results.

During my reserch about this problem i've found RIPPER alghorytm witch, seems like, was designed to solve this kind of problems.
Also there are Multi Naive Bias alghorythm that can be helpfull.

As far as i understand, usualy in EA there is Global Optimization step, withc usually/sometimes implemented via RIPPER.
So, basicly. i have manually generated rule-set witch i have optimize now, with RIPPER for example.

Is it true? Can You recomend some literature?


texts: How to choose layer from which to unfreeze image classification model

texts:
I'm wondering what steps do you take to
decide on the part of the model to unfreeze. Do you do
multiple experiments? Since the use of GPU is expensive, you must
have some guidelines.

Note: I know the relationship between size of dataset, how close
dataset is to the original dataset and how much that impacts whether
or not we train more layers. However is there a rule of thumb
involving the depth of the model to get the approximate layer?

Example:  Try unfreezing model starting from Layer number 169,
or  layers between 70-100
How much does one need to know specifics of the pretrained model? Can I use it without knowing the architecture?

Thank you for your help!


texts:
I want to train YOLO3 for a custom dataset that has raw labels in JSON format. Each bounding box in JSON is specified as [x1, y1, x2, y2].

So far, I have converted [x1, y1, x2, y2] to [cx, cy, pw, ph], that is, center x, center y of the bounding box, scaled to image width and height; pw, ph are the ratios of bounding box's width and height relative to the image's width and height. But I don't think that's complete (or even right).

As far as I understood, YOLO3 assigns N anchor boxes to each grid cell (image is divided into SxS grid cells) and thus, the prediction of a bounding box is relative to a given anchor box from a grid cell (that one anchor box that has the highest IOU with the ground truth). The formulas are below:

Therefore, how should I prepare ground truths so that YOLO3 can understand them? Do I have to, somehow, reverse those formulas? Also how to account for different number of scales and different number of anchor boxes?

For a concrete example: Suppose I have a 416 x 416 image and a configuration of 13 x 13 grid cells. The ground truth bounding box (from the dataset) is [x1=100, y1=100, x2=200, y2=200], class = c. What will be the converted values for YOLO3?

L.E.: Say we have 2 classes [car, person] and 2 anchors (1 wide, 1 tall).

Would the output be a tensor of shape 13 x 13 x (2*(5+2)) where most of the values (that vector of shape 2*(5+2)) for the grid cells are 0 except for one particular cell (the one in which the center of the ground truth bounding box falls)?

In this case, for that cell (say c[i,j]), suppose the largest IOU is given for anchor 2 and that the ground truth class is person. This means that c[i,j,:7] (anchor 1 prediction) will be ignored and c[i,j,7:] (anchor 2 prediction) will be [bx, by, bw, bh, conf, 0, 1].

Therefore, how should the ground truth for the person's bounding box should be encoded? Should it be an offset from a particular anchor of a grid cell? This is what it's still unclear for me.

Thank you!


texts: How should classification be done for a very small data set?

texts:
I am looking at data from the London Data Store based on social characteristics between London boroughs.

Since there are only about 30 London boroughs, the data sets I am looking at are naturally very small. For example, I might be fitting regression/correlations to a plot of about 30 points.

What are appropriate ways to conduct classification on such small data sets, and why? 'Why' is important.

I was thinking of something like SVM, or Naive Bayes. Or regression if the data is continuous.

What are very inappropriate ways to conduct classification here?


texts: Similarity between two sets of documents

texts:
I have 4000 tweets about a brand and 2000 comments about its products. I want to know how much the comments about the brand's products are related to the tweets it generates.
I know how to do a cosine similarity between two documents. But how can I provide a similarity score between these two sets?
My first guess is to do:
$$similarity\_between\_sets = \frac{\sum_{i=1}^n sim\_score_i}{n}$$
Is there a better way to find the similarity between two sets of documents? A way backed by scientific publications? Or even an API?
My very first thought was to:

Either do a word embedding of these two sets and compute the difference of these vectors.
Or generate their topics with Latent Dirichlet Allocation and then creating another formula to find the similarity between these docs. But it's hard to find the right number of topics. I could find the one that maximize the results of the formula though.


texts: Does Decision tree classifier calculate entropies before transforming categorical features using OneHotEncoder or transformation should be done

texts:
I am new to machine learning, and I've got to the point to drop out from it as online tutorials are pretty confusing as well.

Entropy and Decision trees

One of confusing tutorials was as the following:

Another tutorial was pretty straightforward and comprehensive in term of how entropies and information gain is being calculated, but he didn't split the data.

Where the instructor started the entropy calculations and ended up with the following tree:

I did understand the calculation as we used to calculate entropies since high school. The confusing part starts when I decided to program the same steps using python:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_excel('data.xlsx')
X = dataset.iloc[:, 0:4]
y = dataset.iloc[:, 4]

At this point, I tried to apply DecisionTreeClassifier directly on X and y. And BAM. Lots of errors appeared at the console. So I used LabelEncoder and OneHotEncoder:

from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()
X['Outlook'] = lb.fit_transform(X['Outlook'])
X['Temp'] = lb.fit_transform(X['Temp'])
X['Humidity'] = lb.fit_transform(X['Humidity'])
X['Windy'] = lb.fit_transform(X['Windy'])
y = lb.fit_transform(y)

from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

ohe.fit_transform(X)

I then figured it out, as there is categorical data that should splitted using OneHotEncoder. By that, the 4 columns (Outlook, Temp, Humidity, and Windy) will turn into:

Rainy   Overcast   Sunny   Hot   Cold   Mild     High   Normal    False    True

As independent features, and then the calculation should start from this point. So which concept is used by the model to calculate the entropies, the one that confused me even if user used OneHotEncoder or my logic is the true one.


texts:
In the following code, I first declare a, b as tensorflow placeholders for initialising iterator iter with batch size 5. Later in the code, I want to see the predicted output by network one by one on test data, so I change batch size to 1 and use iter2 with placeholders c, d, and print len(predicted), it still comes out to be 5

import tensorflow as tf
import matplotlib.pyplot as plt

EPOCHS = 500

# to reproduce error, load X as a list of input features

BATCH_SIZE = 5
batch_size = tf.placeholder(tf.int64)
n_batches = int(len(X_train) / BATCH_SIZE)
# ---------------------------------------------------------------------------------------------------------------------

a, b = tf.placeholder(tf.float32, shape=[None, len(X[0])]), tf.placeholder(tf.float32, shape=[None, 8])

dataset = tf.data.Dataset.from_tensor_slices((a, b)).batch(batch_size=BATCH_SIZE).repeat()  # creating dataset for iterating
iter = dataset.make_initializable_iterator()
features, labels = iter.get_next()

c, d = tf.placeholder(tf.float32, shape=[None, len(X[0])]), tf.placeholder(tf.float32, shape=[None, 8])

        dataset2 = tf.data.Dataset.from_tensor_slices((c, d)).batch(batch_size=BATCH_SIZE).repeat()  # creating dataset for iterating
        iter2 = dataset2.make_initializable_iterator()
        features_test, labels_test = iter2.get_next()

hidden1 = tf.layers.dense(inputs=features, units=len(X[0]), name='first', activation=tf.nn.relu)
# .... -----------------------------------------------------------------------------------------------------

# Some network architecture

# .... -----------------------------------------------------------------------------------------------------
net = tf.layers.dense(hidden1, 40, name='net1', activation=tf.nn.relu)
prediction = tf.layers.dense(net, 8, activation=None)
loss = tf.losses.mean_squared_error(prediction, labels) + tf.losses.get_regularization_loss()

reg_less_loss = tf.losses.mean_squared_error(prediction, labels)

train_op = tf.train.AdamOptimizer().minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    sess.run(iter.initializer,
             feed_dict={a: train_data[0], b: train_data[1], batch_size: BATCH_SIZE, prob: 0.5})  # prob: 1.0
    print('Training...')
    for i in range(EPOCHS):
        tot_loss = 0
        for j in range(n_batches):
            _, loss_value = sess.run([train_op, loss])
            tot_loss += loss_value
        print("Iteration: {}, Loss: {:.5f}".format(i, tot_loss / n_batches))

    sess.run(iter2.initializer, feed_dict={c: test_data[0], d: test_data[1], batch_size: 1})
    for i in range(len(test_data[1])):
        test_loss, predicted = sess.run([reg_less_loss, prediction])
        print('Test Loss: {:4f}'.format(test_loss))
        print(predicted, len(predicted))


texts: Can we use prior information to improve the results of a neural network? If so, how do we incorporate it into the learning process?

texts:
Let $f_\alpha(x)$ be the function
$$
f_\alpha(x) = x^2 + \alpha\sin(x),
$$
on the interval $[-5,5]$. Suppose $\alpha = 2$, and our goal is to learn the function $f_2(x)$ using some form of neural network. $f_2$ looks like this:

We are given a set of noisy measurements of $f_2$ from which we would like to learn $f_2$. Specifically, we are given a set of random samples $\{y_i,x_i\}_{i=1}^n$ with
$$
y_i = f_2(x_i) + N(0,\sigma),
$$
where $N(0,\sigma)$ is normally distributed noise with standard deviation $\sigma$:

Now we can use the samples $\{y_i,x_i\}_{i=1}^n$ to estimate the unknown function $f_2$. However, before doing that, note that the general behavior of $f_2$ is captured well by the function $f_0(x) = x^2$:

Now suppose that instead of just being given the set of noisy samples $\{y_i,x_i\}_{i=1}^n$ of $f_2$ we are given both the noisy samples and the function $f_0$ which we are told is a decent approximation of $f_2$. In other words, we have some prior knowledge on what the function $f_2$ generally looks like.
Can we somehow incorporate this prior knowledge of $f_2$ into our neural network learning process so that we can get a better estimate of $f_2$ than estimating it based on just the noisy samples on their own?
If so, what are our options for incorporating this knowledge into a neural network? Does the type of neural network (CNN, RNN, etc...) affect the way we incorporate the prior information?
P.S. I am coming from statistics/mathematics and while I understand the general principles of neural networks I have only just started using them.
P.P.S. Here is the Matlab code for the images
rng(123);

N = 100;
x = linspace(-5,5,N);
alpha = 2;
sigma = 4;

f_0 = x.^2;
f_alpha = x.^2 + alpha*sin(5*x);
f_sigma = f_alpha + sigma*randn(N,1).';

figure
hold on, grid on
plot(x,f_0,'k--')
plot(x,f_alpha,'b')
plot(x,f_sigma,'r.')


texts: How much features are needed for Reinforcement learning?

texts: I am trying to learn and use reinforcement learning. Now I have only 6 numeric features in my dataset. Can I still use RL? in other words can be using RL for a such a number of features sensible?


texts: Pandas pivot table, creating ad hoc columns per dimension values

texts:
I'm new to pivot tables and have the following dataset:

mydict = {'City' : ['Lexington', 'Lexington', 'Louisville', 'Hartford', 'Portland', 'Dallas'],
          'State': ['KY', 'KY', 'KY', 'CT', 'ME', 'TX'],
          'Zip': ['38293', '38293', '40207', '48488', '55849', '44930'],
          'Region': ['South', 'South', 'South', 'Northeast', 'Northeast', 'South'],
          'Sales': [1000, 2000, 3000, 1500, 2000, 2300],
          'Product Type': ['Industrial', 'Consumer', 'Consumer', 'Educational', 'Educational', 'Scientific']}
dftest = pd.DataFrame(mydict)
mypivot = pd.pivot_table(dftest, values=['Sales'], index=['State', 'City'])

This creates a single column for Sales.

                   Sales
State City
CT    Hartford     1500
KY    Lexington    1500
      Louisville   3000
ME    Portland     2000
TX    Dallas       2300

But what I want is e.g. two columns under Sales, corresponding to an ad hoc list of my Product Types for example ('Industrial', 'Consumer').

Like this:

                  Sales
State City        Industrial   Consumer
CT    Hartford     0           0
KY    Lexington    1000        2000
      Louisville   0           3000
ME    Portland     0           0
TX    Dallas       0           0

Is this possible using pivot tables?  Or do I have to manually build up such a dataframe somehow (something that I think would result in excessively complex code)?

EDIT:

I see now that mypivot.columns returns a MultiIndex.  I have heard of these but don't yet know how to manipulate them.  I sense that the solution to the problem lies in how to specify a MultiIndex filter.


texts: Modelling of an environment that is stochastic in nature

texts:
I have started learning reinforcement learning and have few doubts regarding model based and model free methods.

Is it possible to model an environment that is stochastic in nature?

Is it because its difficult to model such environment we use model free methods?


texts: For binary classification, which is best Random Forest or Neural networks?

texts:
I had to perform a binary classification, and from the beginning I started thinking about using the Random Forest classifier. But now I'm thinking, if using a neural network would've not been better.

So my question is, which model you would've chosen.? and if not the Neural Networks, what are the tasks the most suited for the Neural Network and why.?


texts: What is the "dying ReLU" problem in neural networks?

texts:
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:

  "Unfortunately, ReLU units can be fragile during training and can
  "die". For example, a large gradient flowing through a ReLU neuron
  could cause the weights to update in such a way that the neuron will
  never activate on any datapoint again. If this happens, then the
  gradient flowing through the unit will forever be zero from that point
  on. That is, the ReLU units can irreversibly die during training since
  they can get knocked off the data manifold. For example, you may find
  that as much as 40% of your network can be "dead" (i.e. neurons that
  never activate across the entire training dataset) if the learning
  rate is set too high. With a proper setting of the learning rate this
  is less frequently an issue."

What does dying of neurons here mean?

Could you please provide an intuitive explanation in simpler terms.


texts: Finding optimal threshold in multi-class classification task

texts:
In a binary classification problem, it is easy to find the optimal threshold (F1) by setting different thresholds, evaluating them and picking the one with the highest F1. Similarly is there a proper way to find optimal thresholds for all the classes in a multi-class setting.

This will be a grid search problem if we do it brute force way. Any efficient way to do this?
Is there any package that ppl use or I can use for this?
Also is it common practice to find optimal threshold in multi-class settings, I couldn't any examples?


texts: Feature extraction from web browsing history of one website

texts:
I have a dataset of web browsing histories for users visiting a particular website over a period of time (say the last 90 days). Each user has a unique ID and several records showing when he/she visited a particular page on the website.

It looks like follow:

UserID,Timestamp,Path
U_1,2017-01-24 12:05:43,/sport/rugby/article_title
U_1,2017-01-24 12:06:56,/sport/football/article_title
U_1,2017-01-24 15:26:12,/finance/local/article_title
......

I do not have access to the content of the articles, I just know the path to the article.

My goal is to build a classifier to predict if a user will take an action or not. So I need to extract features from each user data.

Suppose that I have ground-truth information associated with each user, indicating when a user did the action.

My first guess is to aggregate all the records of each user and extract frequency features (hashing TF) from each level of the path.

So for demonstration, a particular user might visit the /sport category 5 times (first level category), and the /sport/football category 3 times (second level category), and the /sport/rugby 2 times (second level category).

So for each user I will have a feature vector representing the frequency of the first level categories, and another one for the second level categories and so on.

I can now train a classifier for each feature vector and do a late fusion of the results, or I can concatenate (early fusion) the different features and train a single classifier.

I can also extract the terms from the article titles and build a TFIDF feature.

What I am trying now is to extract the features from the N days proceeding taking the actions for the positive samples, and randomly select N consequent days from negative users.

What are the possible other features that I can extract, and is there any better ML techniques to use in order to model and learn the user web browsing behaviour?


texts: How to structure the output layer of an MLP that finds the quadrant of an arbitrary point in a rectangle?

texts:
I'm trying to write a neural network that outputs the quadrant of a rectangle that an arbitrary point lies in. This rectangle has its upper left at {0, 0} and its lower right at {1, 1} (e.g. point {0.25, 0.25} should return 'upper left', and point {0.75, 0.75} should return 'lower right').
I'm confused about the structure of the output layer.
I've seen examples that suggest two nodes with Boolean "isLeft" and "isTop".
I've also seen examples that suggest four nodes with probabilities for UL, UR, LL and LR.
Is one or the other considered the proper structure in this case? Are both approaches valid?
I want to do this exercise as simply as possible, as I work to learn Multi Layer Perceptron. If there is a subtopic I need to focus on, please direct me to it.
Thanks in advance!


texts: Problem with a feature (normal distribution + peak around 0)

texts:
I have a feature that shows a characteristic of the instances. That characteristic can be present or not. If present it shows an almost normal distribution of values (actually a bit skewed to the right, but with a log transformation it becomes normalized). When the characteristic is not present in the instance, the value of the feature is just 0.
So at the end, I have a distribution with a lot of instances with value 0 and a bit far right from it the almost-normal distribution. I would like to split it in two different features: one that shows the absence/presence of the characteristic (easy), and a second that shows only a normal distribution without the annoying peak around zero.


texts: Using time series to predict house prices vs. multiple linear regression

texts:
Re-post.

Machine Learning Courses often teach house prices prediction using multiple linear regression - when we want to predict the value of a variable based on the value of two or more other variables. Indeed the one I did on coursera and that makes sense to me.

I note that others state that a techniques like time series is also a suitable method to predict the house price, given a set of input variable.

I am not clear as to why this could be so. Simply because there is no notion of selling of a house at a regular interval afaik. That is to say, the sale date (one of the variables) can be at any time, not at a reasonably regular interval. Or is it possible because it is not just one house but many houses that can contribute to such an interval?

So, looking for a clarification as to why time series could be an approach for house price prediction. To borrow from wikipedia: A time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. Examples of time series are heights of ocean tides, counts of sunspots, and the daily closing value of the Dow Jones Industrial Average.


texts: How to quantize weights in forward pass during training in Keras?

texts:
In Keras, I'd like to train a network with binary weights in the manner of Coubariaux, et al., but I can't figure out where the quantization (binarization) should occur within the code.

A core aspect of the training method is this:

At the beginning of each batch during training, the stored real (e.g., int32) weights are converted to binary values (either by rounding or in a stochastic/probabilistic manner) and stored separately from the real-valued weights.
Binary-valued weights are used in the forward pass, to compute activations.
Real-valued weights are used in the backward pass, to compute the gradients.
Weight updates are applied to the real-valued weights, not the binary ones.
Binary-valued weights are not altered until the next batch, at which time they are recomputed by binarizing the recently updated real-valued weights.

If I do the binarization in Layer.call(), I believe it will occur for every forward pass (for every sample), but it should only occur once per batch.

If I do the binarization in a Callback.on_batch_begin(), I don't think I can specify the use of binary weights for the forward pass, and real-valued weights for gradient computation.

Any suggestions?  Unfortunately my Python knowledge is not deep, so it's been somewhat challenging for me to understand the flow of the code.

**Note that when I talk about binary values, I do not mean 1-bit (e.g., 8 binary values placed into an int8).  The binary-valued weights can still be represented by int32, float32, whatever.


texts:
Since support is 0.1%, each item has to be in atleast $10^6$ many transactions to be frequent.
There are $10^9$ transactions so there can be atmost $10^3$ many size 1 frequent subsets.
Now $CANDIDATE-GEN(F_0)$ can produce upto $|F_0|^2$ many 2 sized subsets. Now I am confused how to make this bound even tighter.
In general if i wish to find k sized frequent subset, do i have to find all 1,2....k-1 sized subset or is there a direct way?


texts: Is the F1 Score sensitive to the threshold?

texts: Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?


texts: Split datetime64 column in dataframe on year

texts:
I have a column in my dataframe containing data in the datetime64 format.
so example: 2005-02-01 00:00:00 and 2007:02-02 00:00:00.

I want to split this into multiple columns which per year. Does anyone have any suggestions?


texts:
I've recently been looking at autoencoders and kernel PCA for unsupervised feature extraction.

Lets consider just for a moment linear PCA. Its my understanding that if a autoencoder (with a single hidden layer) has linear activation functions then the sum-of-squares error function (chosen for convenience rather than statistical relevance) will have a unique solution for the weights. This can be easily seen, consider the sum-of-squares error function for the autoencoder input $\boldsymbol{x}$ and the autoencoder output $\hat{\boldsymbol{x}}$, that we wish to solve:

$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \hat{\boldsymbol{x}}^{(n)}\|^2\\
\end{aligned}
$$
An autoencoder is a neural network whose input is its output. Defining the following: Let $\boldsymbol{W}$ be the weights matrix between the input layer and the hidden layer and $\boldsymbol{V}$ be the weights matrix between the hidden layer and the output layer. I'm ignoring the bias terms for now. Let $\psi$ be the activation function for each unit on the hidden layer and let $\phi$ be the activation function for each unit on the output layer. Then we have two mappings in an autoencoder:

$$
\boldsymbol{z} = \psi\left(\boldsymbol{W}\boldsymbol{x}\right), \:\:\:\:\:\:\: \hat{\boldsymbol{x}} = \phi\left(\boldsymbol{V}\boldsymbol{z}\right)
$$

$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \phi\left(\boldsymbol{V}\psi\left(\boldsymbol{W}\boldsymbol{x}^{(n)}\right)\right)\|^2\\
\end{aligned}
\label{eqn: 1}
$$
If $\psi$ and $\phi$ are linear, i.e  $\phi(\boldsymbol{s})=\boldsymbol{s}$ and $\psi(\boldsymbol{s})=\boldsymbol{s}$, then $\hat{\boldsymbol{x}} = \boldsymbol{V}\boldsymbol{W} \boldsymbol{x} = \boldsymbol{A} \boldsymbol{x}$ (i.e rotation) and
$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{V}\boldsymbol{W}\boldsymbol{x}^{(n)}\|^2\\
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{A}\boldsymbol{x}^{(n)}\|^2\\
\end{aligned}
$$

This is recognised as principle component analysis. This solution for the weights "forms a basis set which spans the [linear] principle component space" (Bishop, Pattern Recognition and Machine Learning). Bishop then goes on to say that these weight vectors aren't necessarily orthogonal or normalised. I assume that this is because we have not proposed any constraints on $\boldsymbol{A}$ as we would have done in PCA by way of proposing an orthogonal basis?

So am I right in thinking that the autoencoder I've described above does not in general give the linear PCA solution, but in some very special cases they may match?

Now bishop goes on to say that the only way to handle non-linear inputs is to add more hidden layers to the autoencoder and have some layers use non-linear activation functions. However one would assume that the same problem as above persists? i.e. that one is not guaranteed to get the same solution as PCA.

There seem to be additional issues [Bishop]:

Training requires solving a non-linear optimisation problem.
One may end up with a solution that corresponds to a local minima, not a global one.
One must specify the number of subspace dimensions as part of the network architecture.

Therefore my question is, why use autoencoders at all for unsupervised feature extraction? Why not just use kernal PCA? Which will reduce to linear PCA when the data permits it. Is there some other advantage that I'm missing? Under what conditions should an autoencoder be chosen over kernel PCA?


texts:
I have a table like this; all columns are numeric:

TableTest

Col1  Col2  Col3  Col4
  5    6     7     8
 12    6     5     6
  2    3.5   6     1

And I want to find the closest row with these values: 4, 5.75, 7.2, 6. A human can find it (row 1 is answer), but how can I write a query for a computer to find it?

I'm using SQL Server but I think it's relevant to machine learning.


texts: Censored output data, which activation function for the output layer and which loss function to use?

texts:
I am building a neural network to solve a regression problem. The output is a single numerical value.

Unfortunately, the output is censored: the values below 0 were recorded as 0, and postive values remained unchanged.

What activation function should I use for the output layer (maybe ReLU)? How to define the loss function, should I just use RMSE? (because the output is censored, we want the neural network to be able to generate 0 output, and positive values).

Edit 1:

The problem is to predict the electricity demand time series based on multiple input variables. Only the values above a certain threshold are being recorded, hence the output is censored. We have a lot of numerical/categorical input variables: time of the day, air temperature, days in a week, holiday/workday, etc. We want to build a neural network model to predict electricity demand (0 or positive) based on the input variables.