texts: Why is my MLP with 2 features is doing worse than MLP with 1 feature where the one feature is a combination of feature1*feature2?

texts:
I have programmed a MLP for a dataset (~500 rows) containing the length (L) and width (W) of an organism and the output of biomass (the organisms weight in pounds, B).

            mlp = MLPRegressor((5, 5), max_iter=1000)

I have trained the model with features

# Model 1
# Input = Feature 1: Length, Feature 2: Width. Output = Biomass
df = {'length': [60.1, 59.2, 59.4, 58.5], 'width': [15.4, 16.2, 14.9, 15.7], 'weight': [8.34, 7,65, 7.89, 7.14]}

# Model 2
# Input =  Feature 1: Length * Width^2. Output = Biomass
df = {'length*height^2': [60.1, 59.2, 59.4, 58.5], 'weight': [14253.31, 15536.44, 13187.39, 14419.66]}

The overall accuracy of my model with one feature is over 95%, however the accuracy with the features separated is about 85%.

My understanding of an MLP is that Model 1 should do better than model 2 as it will basically find the best combination of length and height to biomass, However my 1 feature model is doing significantly better. I have also tried standardizing the dataset with a scaler with no luck.

scaler = StandardScaler()


texts: keras flow_from_directory returns 0 images

texts:
When I try to use the following snippet of code to try to predict on a batch of images, I get a message saying that no image were found.

test_datagen = ImageDataGenerator()

test_generator = test_datagen.flow_from_directory(
        directory='test/',
        target_size=(300, 300),
        color_mode="rgb",
        shuffle = False,
        class_mode='binary',
        batch_size=1)

filenames = test_generator.filenames

My directory structure is as follows.
I first have a main directory named dogs_vs_cats in which I have two sub directories train and test containing the respective images and also the notebook which contains this code.


texts: how bidirectional neural networks can be applied on time series while we do not know the future data?

texts:
I have read about bidirectional neural networks. It seems that they need input from both past and future. so lets say we are going to predict the energy use of one hour ahead having the energy use of last 168 hours as inputs to the network. So using bidirectional RNNs it also needs the future data, which is what we are going to predict! Can you please guide me about my confusion?

Thanks


texts:
I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set.

Is it common to include the response as a grouping variable or not and what are the implications of including/excluding it?


texts:
I am working on a ranking question, recommending k out of m items to the users. The evaluation metric is average precision at K.

Both R and Python have xgboost can be used for pairwise comparison and can be adapted for ranking problems.

Are there other algorithms or approaches that can be applied to ranking problems?


texts: How can I randomly sample the space of consistent neural networks for given data?

texts: Suppose I have a dataset $X$ and target labels $Y$. For a fixed neural network architecture, how can I randomly and uniformly sample from the space of all possible assignments of weights such that the neural network maps $X$ to $Y$?


texts: Solutions for Labelling Training Data for Binary Classification Problems

texts:
I have a huge dataset for which I am trying to use an 80-20 (Holdout method) approach to train and test my model. However, the dataset I have been given has 6m rows. The objective is to train+test+validate the model before using live data traffic for real-time predictions.
The expected result here is "It's not corrupted with 97% accuracy" which is implementation details and output of some Jupyter notebook etc.
My Question is - Is there any alternatives than manually labelling such a big dataset?
By manually labelling - I mean a human (or a group) going through all the 6m rows(!). Also, not all input strings have identical contents so it's hard to just push it through some script/csv and automate it. But I am trying to understand if this is the ONLY way.


texts: Doubt regarding the number of weights in 2 layer neural network

texts:
Considering a hypothetical scenario , where we have 10 input layers, and 5 output layers.

How many weights are there in the neural network?

If this is implemented in pytorch, the answer will be 50. But shouldn't it be 15 since 10 weights from input layer, and 5 weights from output layer? Please resolve my doubt.

Please forgive my english, not a native speaker.


texts: Looking for a web based tool for natural language to PostgreSQL query for custom database

texts: I need to build a web-based solution that will take a natural language query in English, convert it into an SQL query that I need to execute on a PostgreSQL database. I tried ln2sql/nl2sql but it only supports MySQL database. Is there any alternative that I could use directly on my PostgreSQL schema?


texts: How to print a Confusion matrix from Random Forests in Python

texts:
I applied this random forest algorithm to predict a specific crime type. The example I took from this article here.

import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder
import random
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
import matplotlib
import matplotlib.pyplot as plt
import sklearn
from scipy import stats
from sklearn.cluster import KMeans
import seaborn as sns
# Using Skicit-learn to split data into training and testing sets
from sklearn.model_selection import train_test_split
# Import the model we are using
from sklearn.ensemble import RandomForestRegressor
import os
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

features = pd.read_csv('prueba2.csv',sep=';')
print (features.head(5))

# Labels are the values we want to predict
labels = np.array(features['target'])
# Remove the labels from the features
# axis 1 refers to the columns
features= features.drop('target', axis = 1)
# Saving feature names for later use
feature_list = list(features.columns)
# Convert to numpy array
features = np.array(features)

# Split the data into training and testing sets
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42)

baseline_preds = test_features[:, feature_list.index('Violent crime')]
# Baseline errors, and display average baseline error
baseline_errors = abs(baseline_preds - test_labels)
print('Error: ', round(np.mean(baseline_errors), 2))

# Instantiate model with 1000 decision trees
rf = RandomForestRegressor(n_estimators = 1000, random_state = 42)
# Train the model on training data
rf.fit(train_features, train_labels);

# Use the forest's predict method on the test data
predictions = rf.predict(test_features)
# Calculate the absolute errors
errors = abs(predictions - test_labels)
# Print out the mean absolute error (mae)
print('Promedio del error absoluto:', round(np.mean(errors), 2), ' Porcentaje.')

# Calculate mean absolute percentage error (MAPE)
mape = 100 * (errors / test_labels)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Precision:', round(accuracy, 2), '%.')

# Get numerical feature importances
importances = list(rf.feature_importances_)
# List of tuples with variable and importance
feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(feature_list, importances)]
# Sort the feature importances by most important first
feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)
# Print out the feature and importances
[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];

# Import tools needed for visualization
from sklearn.tree import export_graphviz
import pydot
# Pull out one tree from the forest
tree = rf.estimators_[5]
# Import tools needed for visualization
from sklearn.tree import export_graphviz
import pydot
# Pull out one tree from the forest
tree = rf.estimators_[5]
# Export the image to a dot file
export_graphviz(tree, out_file = 'tree.dot', feature_names = feature_list, rounded = True, precision = 1)
# Use dot file to create a graph
(graph, ) = pydot.graph_from_dot_file('tree.dot')
# Write graph to a png file
graph.write_png('tree.png')

So my question is: how can I add a Confusion matrix to measure accuracy? I tried this example from this here, but it doesn't work. The following error appears:

Any advice?


texts: Categorical and ordinal feature data representation in regression analysis?

texts:
I am trying to fully understand difference between categorical and ordinal data when doing regression analysis. For now, what is clear:

Categorical feature and data example:
Color: red, white, black
Why categorical: red < white < black is logically incorrect

Ordinal feature and data example:
Condition: old, renovated, new
Why ordinal: old < renovated < new is logically correct

Categorical-to-numeric and ordinal-to-numeric encoding methods:
One-Hot encoding for categorical data
Arbitrary numbers for ordinal data

Categorical data to numeric:

data = {'color': ['blue', 'green', 'green', 'red']}

Numeric format after One-Hot encoding:

   color_blue  color_green  color_red
0           1            0          0
1           0            1          0
2           0            1          0
3           0            0          1

Ordinal data to numeric:

data = {'con': ['old', 'new', 'new', 'renovated']}

Numeric format after using mapping: Old < renovated < new → 0, 1, 2

0    0
1    2
2    2
3    1

In my data I have 'color' feature. As color changes from white to black price increases. From above mentioned rules I probably have to use one-hot encoding for categorical 'color' data. But why I cannot use ordinal representation. Below I provided my observations from where my question arised.

Let me start with introducing formula for linear regression:

Let have a look at data representations for color:

Let's predict price for 1-st and 2-nd item using formula for both data representations:
One-hot encoding:
In this case different thetas for different colors will exist. I assume that thetas already derived from regression (20, 50 and 100). Prediction will be:

Price (1 item) = 0 + 20*1 + 50*0 + 100*0 = 20$  (thetas are assumed for example)
Price (2 item) = 0 + 20*0 + 50*1 + 100*0 = 50$

Ordinal encoding for color:
In this case all colors will have 1 common theta but  my assigned multipliers (10, 20, 30) differ:

Price (1 item) = 0 + 20*10 = 200$  (theta assumed for example)
Price (2 item) = 0 + 20*20 = 400$  (theta assumed for example)

In my model White < Red < Black in prices. Seem to be that correlation works correctly and it is logical predictions in both cases. For ordinal and categorical representations. So I can use any encoding for my regression regardless of the data type (categorical or ordinal)? This division in data representations is just a matter of conventions and software-oriented representations rather than a matter of regression logic?


texts: Running multiple random forest and combining them

texts:
I am trying to build a random forest model in R (RStudio). My training dataset has around 2 million rows and 38 variables. When I tested 5000 rows from this dataset I was able to build the random forest but when I run on the whole dataset I get the following error:

Error in randomForest.default(m, y, ...) :
  long vectors (argument 24) are not supported in .C

Can anyone please suggest, apart from removing the number of rows, how can I fix this? Can I run multiple random forests and then combine them into one? If yes, can someone please recommend how can I try this?

Many thanks in advance.


texts: What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors?

texts:
I understand that the models are only as good as the data you get, and bad design can generate really bad data. Nonrandom sampling, unbalanced/incomplete designs, confounding, can make data analysis really hard.

At what point should one be confident that they ran a useful model? Do you just do a cross-validation with a training/test dataset and call it a day? Obviously "all models are wrong, some are useful" but at some point the tradeoffs with excluding too many parameters by LASSOing and strange transformations by getting BIC down become glaring.

tl;dr at the end of the day what makes you go "I did the right thing for my company/project, and this should work"


texts: Advice on using Recurrent Neural Networks for non-time series dataset

texts:
I'm testing different machine learning algorithms for predicting week-to-week fantasy football scores for individual players. For those who don't know, fantasy football is a game in which players pick a "team" of American football players each week and based on their statistical performance that week they are asigned a score. Each position scores points based on their own statistics, so for instance a QB gets 4 points for each touchdown pass it throws, a RB gets 1 point for each 10 yards it runs, etc.
What I am doing is taking the last 5 years of stats for each player in each position (filtering for those who started more than 5 games each season) and calculate the 1, 5 and 10 moving average of each particular statistic. So, for instance, for each game of player X who is a QB I have the previous game number of touchdown passes thrown, the 5 game moving average of touchdown passes thrown and the 10 game moving average of touchdown passes thrown. Repeat for every statistic. I also added each of their rivals' defensive statistics for each game, in the same vein: 1, 5 and 10 moving averages of every relevant statistic.
I used this dataset to run linear regression, SVM, Bayesian regression and MLP models, with results similar to the benchmark of "fantasy football experts" that give their predictions each week. I split the datasets 70-30 randomly for train-test, and used 20% of the train set for validation. The problem is that the error is very large anyway: MRE is over 50%, with a std of 30-50% too (it's the same for experts, but still).
I was wondering if RNNs wouldn't perform better. The problem is that I don't know if I should feed the NN with the same dataset (with the moving averages instead of each game's statistics) or if I should build a time series for each player instead. In which case: with 5 seasons for each player (each season 16 games or less) I have at most 90 datapoints for each player. Would that be enough for a RNN? And would it be fair to compair the performance of the RNN with the performance of the other models, which I trained with a different training set and validated with a different validation set and test set, since in this case I'll have to set apart one season (or a number of seasons) for prediction?
I'm new to deep learning models, so I'm basing this on what I read on the subject: I'll have to build a time series of each game of each player and starting with one game I'll let the model predict the performance of each subsequent game in the series like this?
    t_1             t_2                    t_i            t_(i+1)        t_(i+2)                    t_n
 _________      _________               _________      _________      _________               _________
| full    |    | full    |             | full    |    | empty   |    | empty   |             | empty   |
| game    |----| game    |---- ... ----| game    |----| game    |----|  game   |---- ... ----|  game   |
|_________|    |_________|             |_________|    |_________|    |_________|             |_________|
     ^              ^                       ^              ^              ^                       ^
     |              |                       |              |              |                       |
  use to          use to                 use to         predict        predict                 predict
  train           train                  train

Or is this not necessary and I can feed the neural network with my samples the same way I fed the other models their samples? I'm mostly confused because I've seen these sort of models applied to samples that don't necessarily fit a "time series" frame, like image processing where each sample doesn't have a time condition relation with other samples.


texts: How does Excel obtain the R² determination coefficient in an exponetial regression? This may seem a silly question but Excel actually shows a R² coefficient for non-linear regressions. How does Excel calculate it? Does it have any sense?


texts: Use tf.image.ssim_multiscale for a batch of images in sequential model

texts:
I am trying to use SSIM as loss value for my Keras Sequential model. The output value is a set of 6 images - (6, 32, 28, 3). I want to implement a custom loss function for the model.fit() function but cannot understand how to implement this. I am attaching what I have tried before-
def ssim_loss(y_true, y_pred):
    return 1-tf.reduce_mean(tf.image.ssim_multiscale(y_true, y_pred, 2.0))

def ssim_loss(y_true, y_pred):
  loss = 0
  for i in range(6):
    loss += tf.reduce_mean(tf.image.ssim_multiscale(y_true[i], y_pred[i], 2.0))
  return loss

y_true shape is same as y_pred which I have checked. These both solutions do not work. I do not know how to access each element of y_pred. My input shape is  - (6, 192, 56, 3). Please help. TIA


texts: How to select a bunch of optimized data from a larger data set? I have a data set containing the products to be sold to the customers. And the products were sold before, it has some information, like the review, ratings, how many were sold per product from the customers. Now, I want to select a bunch of optimized data from a larger data set based on the number of units sold, ratings, review etc all the information. The purpose is to increase the number of units to be sold in the future. How to do this? What kind of model, method, algorithms should be used here? If it is related to machine learning, what kind of machine learning tools should I use here? Thank you so much.


texts: 2 Phase Prediction and Refinement Pipeline - Deep RL Design Question

texts:
At a high level, the problem I am currently facing is a coverage maximization problem within a mobile sensing network. I am working with decentralized mobile sensors that aim to maximize their coverage of mobile units (let's call them "nodes"). The sensors have a limited sensing radius in which they can observe nodes and other sensors within that radius and a node is considered to be "covered" if they are within the radius of a sensor. I desire to use a protocol where each sensor acts as an independent agent and uses a neural network (or a pipeline of multiple networks) to predict where to move during the next timestep such that total coverage of the entire node system is maximized.
I have encountered a research paper that discusses the principle of maximizing total reward (total coverage in this case) within a system of decentralized agents. It mentions a 2 phase method:
Phase 1. Each sensor observes the current state of its surroundings (sensors and nodes included). It predicts where it should move during the next timestep using a reward function that measures the total observable coverage within its sensing radius.
Phase 2. Each sensor shares its move prediction with neighboring sensors. Then each sensor uses the move predictions of all neighboring sensors as well as current node positions to refine the move prediction it made within phase 1.
I am relatively new to the RL field and am a little dubious/torn between ideas of using a separate network for each phase, using a system of intermediate inputs and outputs on one network, or using a time series style input structure (notably phase 1 and phase 2 have the same input shape). I seek recommendations on a neural network or system of multiple neural networks that would satisfy the following requirements:

I would like phase 2 to "refine" the proposed move prediction generated in phase 1 to a limitable/configurable extent as opposed to "generate" a completely independent prediction. In other words, I would like phase 2's output to be a configurable level of similar to phase 1's output because if each sensor moves completely disjointly to its move prediction within phase 1, the implication is that the phase 2 output was generated using disjoint/incorrect data.
I would like the sensors to have an ability to base its move prediction on a pattern of node movement, as opposed to a static node position (the nodes move by a heuristic-based distribution and not randomly). I'm considering achieving this requirement by passing in state from previous timesteps such that the sensor model can observe the "trend" of past node movements.
I really only need phase 1 because I want the sensors to have a semblance of where other sensors are moving in the next timestep when it generates its actual move action within phase 2. Right now the problem seems intuitively implementable with two networks, one for each phase. Is there a cleaner/more artful way for agents to generate action predictions, pass these predictions amongst themselves, and update their actions based on other agents' initial predictions?


texts: Adding extra variables to XGboost model is worsening the train and test accuracy

texts:
I am fitting a multi class model using Xgboost. I am getting an accuracy of 96% on Train and 95% on test. I am using the 80-20 train/test split.
However, when I am adding two new features , the accuracy drops down to 92% for train and 89% for test.

Doesnt XGBoost:

Pick the most important variable that can be used to split a node and leave out the rest?
Handle multi colinearity?

I have not used cross validation. Could it be that I am still overfitting the data ?

This is the code I used

from sklearn.model_selection import train_test_split
df_new_train, df_new_test, y_train, y_test = train_test_split(df, labels2, test_size = 0.2)

dtrain = xgb.DMatrix(df_new_train, label=y_train)
dtest = xgb.DMatrix(df_new_test, label=y_test)

param = {
        'max_depth': 10,
        'early_stopping_rounds': 10,
        'eta': 0.01,
        'subsample': 0.6,
         'colsample_bytree': 0.5,
        #'alpha': 0.5,x`
        #'lambda': 0.5,
        'gamma': 10,
        'min_child_weight': 1,
        'watchlist': [(dtrain, 'train'), (dtest, 'valid')],
        'objective': 'multi:softprob',  # error evaluation for multiclass training
        'num_class': 4}  # the number of classes that exist in this datset
num_round = 1500

bst = xgb.train(param, dtrain,  num_round)

preds = bst.predict(dtest)
preds_train = bst.predict(dtrain)

best_preds_train = np.asarray([np.argmax(line) for line in preds_train])

best_preds = np.asarray([np.argmax(line) for line in preds])

print(classification_report(y_test,best_preds,target_names=label_encoder.classes_ ))
```


texts:
i am thinking about using a CNN to classify certain images from industrial production, like scratches, stain, particles etc.

the problem is we don't have many images. We only get around 10 parts with certain defects to detect. I thought about using a database with different defects on it.

I use the example of scratches, there are around 100 images of scratches. Now my question:

Does it make sense to rotate the images in all 4 orientation, and to mirror them as well? So i would get 8 images (the original + 7 uniformly transformed images), i know the "value" of the rotated and mirrored images is not as high as from real images, but it should still help the CNN to abstract and find features.

What do you think about it?

Thanks in advance!

Dog_Fish


texts: How can I test the relationship between events and categorical data

texts:
I have a dataset comprised of mostly categorical data, in particular, different name tags of events that happen within a process. Those are accompanied of their timestamps and there are hundreds of unique tags.

I also have some shutdown events that happen regularly within said process, in my data, I have around 50 registered shutdowns. Which method can I use to try to identify which tags are influencing the shutdown?


texts: Why use both validation set and test set?

texts:
Consider a neural network:

For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the network by checking it on validation set. Then what is the need to test it on the test set to check its performance?

Won't the error on the test set be somewhat same as the validation set as for the network it is an unseen data just like the validation set and also both of them are same in number?

Instead can't we increase the training set by merging the test set to it so that we have more training data and the network trains better and then use validation set to prevent overfitting?
Why don't we do this?


texts: Creating an easy but not trivial dataset

texts:
I am working on the problem of automatic punctuation: given a stream of words, decide for each word whether there should be a punctuation mark after it (in future work I also want to distinguish between different punctuation marks, but currently it's a binary classification problem).
My classifier uses a bidirectional LSTM whose output is fed into a multi-layer perceptron. The accuracy of the current classifier on the entire dataset is not good enough. In order to understand why, I want to create an "easier" dataset - a dataset on which even a simple (but not trivial) classifier would give good results. Then, I will make the dataset more and more difficult and see what improvements have to be made to the classifier. What is a good technique for creating synthetic datasets that are easier (but not trivial) to classify correctly?

In case it is relevant, here are some details on my system:

Each word is represented by an embedding vector of length 200.
The embedding vector is fed into a bidirectional LSTM with an output length of 50.
The LSTM output is fed into a multi-layer perceptron with two layers, the hidden layer has size 60.
The dataset has about 2000 paragraphs, each of which contains about 200 words.
Since the classes are imbalanced (only 20% of the words are punctuated), I check the system accuracy using Cohen's kappa coefficient - a number between -1 and 1. A random classifier has  kappa around 0. A good classifier has kappa above 0.7. My current classifier has kappa around 0.4 (average of 10-fold cross-validation), that is, better than random but not very good.
I train the classifier in 80 iterations. After each iteration I check the kappa of the model on a validation set. Finally I return the model with the highest kappa.


texts: How can I build a seq2seq model , which is topic aware

texts:
I have developed a chatbot, which is basically a seq2seq LSTM network. Which can generate text based on input text. But the problem I am having right now is it is not topic aware.
As an example :

Input : Wanna go to the stadium ? Topic: Football

Answer: No , I don't like football.

What I am planning to do , is to tokenize each possible topic

As an example {1: "Football",2: "Food" ..........}

But don't know how to embed this tokenized topic in training .


texts:
I'm clustering texts based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can spread very randomly, I found Kernel Density Estimators relevant.

However, the scores of KDE are very sensitive to the choice of bandwitdh hyperparameter, which I couldn't pre-estimate. Most bandwidth values end up with either infinite score for points outside the cluster, of zero for the points in the cluster. I need a way to 'automatically' choose the bandwidth, so that my results will yield scores that make sense for the points in the cluster (larger values) and the points outside of it (smaller values).

I tried:

Both Silverman and Scott factors methods to evaluate the bandwidth depending on #points and #features, both were far from relevant in my case
GridSearchCV returns the minimal bandwidth in the grid
Different kernel types (all the relevant ones are similarly sensitive)
Reduce dimension, but as expected this severely hurt the KDE results without making the bandwidth that less sensitive

from sklearn.neighbors import KernelDensity

# indexes of points in cluster3
docs = np.where(y_pred == 3)[0]

kde = KernelDensity(kernel='gaussian',bandwidth=0.399).fit(X_tfidf[docs].todense())

# evaluate scores on all points
scores = np.exp(kde.score_samples(X_tfidf.todense()))

Note that there are ~2200 features in the TF-IDF, a few dozen points (40-120) in each cluster the KDE was fitted to, and about 4000 points in total.

Any ideas on anything (even beyond KDE) are welcome, thank you


texts: Bag of words and word2vec clarifications

texts:
I have documents,  and I calculated the word vectors using word2vec for all the terms in my corpus.

Now I want to compute similarity between documents using the bag-of-words model.  After  creating the Bag-of-words ( consisting of  vector representations).

how do I now compute similarity between a sequence of vectors?

Do I simply just take the mean? and calculate the cosine similarity between the mean vectors?

or is there a better approach for computing similarity between two bags?


texts: Do Recurrent Neural Networks assume stationarity or just a general kind of sequential dependence?

texts:
Just when I thought I had convinced myself that RNNs make no other assumption about a sequence other than that there are dependencies between the inputs and that (in the case of monodirectional RNNs) the past affects the present, Goodfellow, Bengio and Courville (2016) hit me with this:
"The parameter sharing used in recurrent networks relies on the assumption that the same parameters can be used for different time-steps. Equivalently, the assumption is that the conditional probability distribution over the variables at time t + 1 give the variables at time t is stationary, meaning that the relationship between the previous time step and the next time step does not depend on t."
Could someone elaborate on what this means with regards to the stationarity assumption for the time series?
(I have the feeling that there is a notion of local or "input-conditioned" stationarity here but that is just my intuition.)
Thanks!


texts: Do models learn a test set after multiple predictions upon it?

texts:
In machine learning, when we train a model, such as

vect = CountVectorizer(min_df=5, ngram_range=(1,2)).fit(X_train)

X_train_vectorized = vect.transform(X_train)
model = LogisticRegression()
model.fit(X_train_vectorized, y_train)

predictions = model.predict(vect.transform(X_test))

and make predictions with a test set, does our model learn the test set after multiple predictions are made?


texts: what's this approach to spatiotemporal data named as?

texts:
I have some sequential data (e.g. audio, video, text etc.) and I am using this approach to classify sequences. I am sure there's a name for it, but I can't think of it:
vectors =
    t1,[v1_0....v1_n]
    t2,[v2_0....v2_n]
        :
        :
    tm,[vm_0....vm_n]

where t1..tm are the time offsets and the VM are the feature vectors.
Out of this data, I create oversampling by using different window_size, and step_size
[t0,win_size,step_size, np.median(vectors[frame_id:frame_id+win_size],axis=0)]

win_size = the size of the window to mean/median ahead
step_size = how many columns to move with each step
np.median(vectors[frame_id:frame_id+win_size],axis=0) = column0wise mean or median across the array as a resultant of the vectors

and use the above to train a classifier and should be to generate predictions at multiple scales (e.g. at a large window size).
Edit: for predictions, from the query clip extract a similar set of vectors and then predictions can be aggregated somehow to generate the "consensus".


texts: Extending a trained neural network for a larger input

texts:
I have a seq2seq conversational model (based on this implementation) trained on the Cornell movie dialogs.
Now I want to fine-tune it on a much smaller dataset. The new data comes with the new words, and I want UNKs for as few new words as possible. So I'm going to create a new network with respect to the new input/output sizes, and I'm going to initialize its submatrices with learned weights I have at hand.

Could you say if this method can cause problems with the resulting model's performance? E.g. are the softmaxes likely to be affected significantly with these new initially untrained weights?
And if it's OK, do you have some examples on how to do it with the least pain in tensorflow's seq2seq setup?


texts: Does training of neural networks follow the same order in each epoch?

texts: Each epoch uses the weight from the end of the previous epoch(correct me if I am wrong). Is the updating of parameters after each batch always in the same order? To rephrase, are the batches always in the same order? Could this bias the learning and are there any adaptations that deal with this.


texts: Time-series decomposition to a base level and an effect of another feature

texts: I've got a time-series data (let's denote it as y) and some feature (let's denote it as x). y is dependent on x, but x is often equal to 0. Even then, y is not 0, so we can assume that there's a base level in y which is independent of x. Additionally, we can observe some seasonality in y. I need to decompose y into base level and an effect of x. And I need some hint about methodology. I have googled and found plenty of methods to decompose time-series data into trend, seasonality and random noise. However, my case is different, because I have an additional feature x and I would like just to extract its effect, and leave trend, seasonality and noise alltogether. What I have in mind can be represented on a plot below, where turquoise area represents base level of y and red area represents an effect of x. What method would allow to make such a split? I would also appreciate any links or materials.


texts: Extending DTW 1-NN Classification to On-line Scenario

texts:
I am familiar with Dynamic Time Warping classification using a 1-nearest neighbour approach. However, in most benchmark datasets and applications, it used ex-post, i.e. classifying a time series after you have observed it.

Is there a way to use the dynamic time warping similarity distance in an on-line setting before I have fully observed the series I am comparing?

Furthermore, if there is a way to probabilistically categorize as the series comes in that is even better. E.g. After observing 1/10 of the time series, the model predicts this sub-series has a 20% of being in group A, 10% of being in group B, etc


texts: Decision tree not using all features from training dataset

texts:
I have built CART model using sklearn. I'm having total 6 features in training dataset and passing all of them in fit function. I've tested both criteria Gini and entropy. But whenever I plot tree using graphviz, the tree uses only 3 features in case of Gini and 4 features in case of entropy. I've also implemented CART from scratch for cross check purpose and still, Gini uses 3 features and entropy uses 4 features.

Everything else is working fine. I'm getting the result as expected in test dataset with accuracy 97%. I just want to confirm that, is this normal? Does cart chooses best features only and ignore other features to avoid the wrong classification? Also, my dataset is quite small in size, only 220 records.


texts: How to perform a reggression on 3 functions using a Neural Network

texts:
I am currently building a neural network using Keras to perform a regression.

I have 4 independent variables W,X,Y,Z. They are used to predict 3 different functions f1(W,X,Y,Z), f2(W,X,Y,Z), f3(W,X,Y,Z).

Should my output layer have 1 or 3 neurons? Also, should I be using a relu or linear activation function for the output layer? I'm currently using MSE for my loss function and adam for my optimizer.

For my metrics, should I use 'accuracy' or 'r2'?

Any suggestions?  I'm sorry, I'm new to deep learning...


texts: Parameters calibration for a general DNN project

texts:
I am working on an RNN for a task of emotion recognition from speech. In order to set some parameters, I am following this post, and running a grid search over:

The number of units in each RNN  layer.
The number of RNN layers.
The number of units in each dense layer.
The number of dense layers.
The learning rate.
The dropout.
The number of epochs.
The batch size.

I have neglected some parameters, such as optimizer, distributing a different number of units in each layer, different dropouts, and more. The remaining grid search is very long as it is (Two days now).
For the dropout level, as an example, I am testing the values 0.1,0.2,0.3,0.4. I have no reason for choosing these values other than typical values familiar to me from previous projects. This, to me, is not a very educated way to pick my values, hence my question.
What is the best way (even just a good way) in order to tune parameters for a general DNN project? Specifically:

Should I use grid search for groups of parameters, separately, to shorten the time? For example, search for the best combination of units and layers with a separate grid search, and search for the best combination of batch size and number of epochs using a second grid search? I have found this question, but the answers are not definitive, in my opinion.
If the former is true, is there any grid search that should be established before another grid search? For example, should I establish the architecture before establishing the training parameters (seems logical)?
Are there any reasonable values I should start with for each parameter? For example, I have just found a project where they are using dropout=0.5, which I am not even searching for. What are the reasonable values for the dropout level?

I could not find a tutorial summarizing a good practice of these. I am looking for a practical answer. Papers and research support are most welcome but I think it is more important for beginners to have some guidelines on a good project work order.


texts: Measuring similarity for sets with same cardinality

texts:
The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.

I had 100 sets all of same cardinality. By mistake I calculated the similarity measures as the ratio of intersection with total elements in a set (i.e 100).

This gives different similarity values than the original Jaccard formula.

I was wondering if the original formula has considered the union of two sets to handle cases where there might be sets with different cardinalities.

I think though numerically my values are different, they repersent the same idea.

If anybody could verify/disverify what I am trying to do ?


texts: Convert a pandas column of int to timestamp datatype

texts:
I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately convert it to a column of datetime data by adding the timestamp column series to a series that consists entirely of datetime values for 1970-1-1.

I know how to convert a series of strings to datetime data (pandas.to_datetime), but I can't find or come up with any solution to convert the entire column of ints to datetime data OR to timestamp data.


texts: Validation Loss does not decrease but validation average precision improves

texts:
I am training three neural networks (2CNNs, 1LSTM) on an EHR dataset, which is to predict diabetes according to (100 labs * 360 days) data. However, when I trained them, there is a common phenomenon shown in the following pictures: training loss is decreasing, validation loss (seems) not decreasing at all, but both average precision score (PR-AUC) are increasing.

I am using the scheduler in Pytorch to adjust learning rate, so the training loss is like steps. I know there is some overfitting happening here, so I used dropout (p=0.5) and weight decay (1e-3). I am wondering

why the validation loss is not decreasing but the validation average precision score still goes up,
whether I need to further do some thing with it to combat the overfitting until make the validation loss decreases like the training loss
if not, what is the best way to select the best epoch to do the testing? Based on lowest loss function or best average precision score?


texts:
I'm using FEBRL to do some entity resolution, but I can't figure out a way to add the ground truth to the program in order to evaluate the results.

Can someone with experience on this program help me on this?

(download link to FEBRL)


texts: Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

texts:
I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the precision_recall_curve(), I get a different macro-metric than when I use the builtin average_precision_score() function. As per the docs, this makes sense, due to auc() using a trapezoidal approximation, and thus average precision being more accurate. However, when I compute the micro-metrics, the values are the same. I'm trying to understand this.

Here's an example:

yhat_raw = np.array([[0.511, 0.62, 0.3],[0.11, 0.76, 0.945],[0.002, 0.671, 0.452],[0.9102, 0.783, 0.2]])
y = np.array([[0, 1, 1],[0, 0, 1],[0, 0, 0],[0, 1, 1]])
yhatmic = yhat_raw.ravel()
ymic = y.ravel()
prec["micro"], rec["micro"], _ = precision_recall_curve(ymic, yhatmic,
pos_label=1)

auc(rec["micro"], prec["micro"])
returns 0.58305555555555544.
average_precision_score(ymic, yhatmic) returns 0.62222222222222223.

Upon actually deploying the model, these metrics are coming to the same thing. I think this makes sense, considering that at a larger scale (e.g. along a dimension of thousands of datapoints rather than len(rec['micro']) = 11 datapoints in this example), the trapezoidal approximation is much closer to the true value.

My question is, why do the metrics then differ in the macro case? Is it fair to say that in a real-world deployment, the micro-evaluation has enough datapoints for the trapezoidal approximation to converge to the true value, but the macro- does not (computed across ~3000 test examples, with a label space of ~2000 labels)?


texts: How to construct a self learning process for an ensemble model?

texts:
I have two different (regression) models spitting out predictions on a daily basis for the same dependent variable. My intention is to assign weights to those two predictions and calculate a weighted average. To this end, I have developed a simple system where the MSEs of the models are calculated and used as weights everyday. As a result, the higher the MSE a model has, the lower the weight it is assigned to the model. However, this is a very lame approach and I haven't observed any improvement compared to taking a simple average of the predictions. So what are the ways that I can use to assign weights to those predictions dynamically. That is, I want to update the weights everyday. From where should I start?

Note: I am aware that with ensemble models, I can get the weights automatically (i.e. H2O stacked ensemble). However, I am not allowed to retrain the predictions everyday. So applying an ensemble method and retrain it everyday is not applicable in my case.

Thanks in advance!


texts: Merging and deduplicating mass amounts of data in ORC on a 24 hour interval

texts:
I have the following problem I'm looking to ideate on and looking for advice on how to think about it from a data science perspective. I'll keep the numbers small for simplicity.
Day 1 I get 100 records that sync into N ORC files into S3.
Day 2 I get 40 records, where 30 of them were updates to 30 records from day 1 and the other 10 are new records.
The problem here is the bucket now has duplicates. The 30 from day 2 will also exist in day 1, but day 2 will just have updated versions of day 1.
P represents a person where N people can exist within a single ORC file.

The way I'm thinking about solving this problem is finding the disjoint set to get everything that didn't update today and merging it with the updates today like so:


texts: Do we need to tune same model differently for different window sizes in time series data classification?

texts:
I am currently working on the time series data classification problem using deep learning. As we all know that in time series, we process the time-series data sequentially for some time steps at a time through the model which is called as a window. We slide the window and the next window is our input again. Since I have just started working in deep learning and time-series domain, my question is if we tune some deep learning model for some window size , let's say 10 then should the same tuned model be used to get the near accuracies for some different window(say 15) on the same data or the model should be tuned again for the latter window.

Currently,if I am using the same model for a different window then I get the accuracy decreased by 4-5% from the previous window. From this I believe that window size again is a hyperparameter which when changed requires the model to be tuned again.

So, what is the right thing in this case? Should the same model be used or should it be tuned again? I tried searching this online but I couldn't get any help on this. Help is appreciated as I am new to this domain.


texts: Scripting code for class imbalance in Biolabs Orange

texts:
I'm trying to manipulate some data in Biolabs Orange, using the built in Python Script widget and information at Biolabs Orange tutorial on scripting.
However, I'm struggling with taking the results of SMOTE and putting them into a format for Orange:
This is the code I'm using in the Python Script widget:
# Get libraries
import Orange
import numpy as np
from Orange.data import Domain, Table
from imblearn.over_sampling import SMOTE

#in_data = Orange.data.Table('WORKING_temp.csv')

df = in_data.copy()

# set variables for SMOTE
sm = SMOTE(random_state=42)

# get table of data (X) and class variables (y)
X, y = df.X, df.Y

# resample data and classes
X_res, y_res = sm.fit_sample(X, y)

df.X = X_res
df.Y = y_res

temp = Orange.data.Table(df.X, df.Y)
temp.domain = df.domain

out_data = Orange.data.Table(temp)

The result is a ValueError, which I think is related to changing the length of the class variables and data table, while leaving the original index length?
"ValueError: could not broadcast input array from shape (3724,10) into shape (3724)"


texts: Anomaly Detection in Time Series: How to label the data

texts:
How to label time series so that we can train it on machine learning models to classify data point as anomaly or not?

If I have time series, and anomaly occurs at time t, should I label that point 1 and other points as 0, or there are any other methods to label the time series?


texts: Is reseating passengers a reinforcement learning problem?

texts:
Requirement is to optimally move passengers from one seat map to another which has a different configuration.

Move should be based on many rules like -
1) Families should be sitting together
2) Those who were at windows seat should be moved to window seat if available.
3) Kids should be seated at middle seat.
4) and many more..

There are many seat map configurations available and at a time we need to move passengers from one to another as per the requirement.

Is it a reinforcement learning problem ? If yes, how it can be approached.


texts:
I am working with data that requires classifying if a patient will develop cancer or not in the future, based on medical tests done over time. The tests have a sequential relationship. A, then B, then C, etc. For example:

| Patient ID | Test ID | RBC Count | WBC Count | Label
|     1      |    A    |     4.2   |    7000   |   0
|     1      |    B    |     5.3   |    12000  |   0
|     1      |    C    |     2.4   |    15000  |   1
|     2      |    A    |     7.6   |    8000   |   0
|     2      |    B    |     7.4   |    7500   |   0

Each point is not taken at a regular time interval, so this may not be considered a time-series data. How can I apply Keras Sequential model to this? Especially since each patient can have a different number of tests, up to 5, eg. Patient ID 1 has 3 tests, and Patient ID 2 has 2 tests.


texts: How would knowing spammers email address improve spam detection algorithms?

texts:
I am new into Artificial Intelligence field and I am working on the classic example of Spam detection using classification. I am using Naive Bayes algorithm as well as SVM.

While working on them it came to my mind that having the email addresses of who has sent the emails as an extra feature would improve the accuracy of my algoithm as we will have some extra info besides the content of the email itself. Would it be true? And how could I use this extra feature? Should I just add it to my email content?


texts: How to encode an array of categories to feed into sklearn

texts:
I'm working on a recommendation problem, broadly following the Youtube paper on theirs. Their surrogate problem is to recommend the next video a user will watch. One feature they include in their model is a representation of watch history (I.E. which videos a user has watched), which they achieve by embedding the Video ID's and then averaging that embedding.

So for data, they might have...

[
    [1234, 5678, 9012],
    [1234],
    [9012, 1234, 8245],
]

and each of those IDs is embedded into say 8 features, those 8 features are averaged for every observation in the training data and those 8 average features are input to the deep layers of the network.

Now I can replicate that in keras using a combination of tf.keras.preprocessing.sequence.pad_sequences plus Embedding() and GlobalAveragePooling1D(), and that works reasonably well. My problem is that I'd like to evaluate other algorithms on this same problem (after all, I have WAY less data than Youtube does so I'm not sure I need an NN at all). I can't really conceptualise how to deal with this encoding of historical "watches" though in a way that would work in a more traditional algorithm like XGBoost or RandomForests in sklearn. Ordinal Encoding and taking the average seems plain silly, I obviously can't OneHot encode the 1st to nth video watch and so on. Any ideas of how to tackle this would be greatly appreciated.

EDIT:

I guess possible FeatureHasher fits the bill? Rather than an average, it produces a sum of each of the hashed input categories. That's still (be default) gonna lead to 10s of thousands of features though so I'm not sure that it's going to be a significant improvement over OneHotEncoding!

EDIT 2:

Alternatively, perhaps I could treat this as a text embedding problem by concatenating the array of categories into a single string and use CountVectorizer to create an array of (n_samples, n_categories), then use TruncatedSVD to reduce that to a more manageable number of components.

I guess I've a couple of ideas there, anyone know if they're worth trying?


texts:
I am a newbie in machine learning topic and I need to create model from music data.

It contains features of the songs but it is not labeled. How can I create a model from that ?

Do I need to use unsupervised learning algorithms ? Which one is better or is it better if I use deep learning methods.

Data is looking like this:

      danceability  loudness  valence  energy  instrumentalness  acousticness
136         0.795    -8.334    0.578   0.409          0.000000      0.684000
442         0.502    -4.556    0.720   0.912          0.000173      0.000025
92          0.713   -14.590    0.560   0.258          0.006060      0.877000
67          0.505   -14.951    0.723   0.782          0.930000      0.921000
127         0.470    -6.740    0.490   0.809          0.006710      0.272000


texts: Best NN for image pattern recognition without classifier

texts:
For images like this

I'm trying to split up the cells (the different rectangles). The cells, can vary in size, brightness and so forth.
I tried multiple ways using OpenCV for about 5 days, I almost got it working, but there are always edge-cases for which it fails, so I want to give it a shot using ML.
I believe the best way to do so, is to find the intersection patterns, upon which I can know the boundaries of the cells using a Mask R-CNN trained over images like this:

etc..
Once the position of these objects is known (the crossing point) I can create a grid to split up the cells, a 70-80% accuracy would be enough to generate the grid.
Although, most of the R-CNN includes a classification layer, which I don't given I try to detect a multitude of single objects.
Which type of NN should I be using for that purpose?
enter image description here


texts: ML method suggestion for regression task

texts:
I'm working with a brushed DC motor current, voltage, calculated rotations and measured rotations and a couple of state signals(direction of rotation and the operation state).
My objective is to approximate a function which gives the error in measurement of rotations-
H(x) = (measured rotations - calculated rotations)
Here is an example of how my data looks and the difference is determined by the difference of the signals that you see in the top subplot srefpos(measured) and slppos(calculated). The calculated rotations are already pretty accurate but there are obvious instances where it doesn't work correctly and this is the shortcoming that I am trying to improve.
The error(bottom most subplot) is my target(ideally) after removing noise and so far I have already tried-

Regression through Multi Layer Perceptron
1D-CNN
LSTM and
Decision Trees (both simple and ensemble)

The best results I have gotten so far are with decision trees but I am not happy with the generalization achieved(performs not up to mark on test data). With the other methods I have mentioned above, the network/the algorithm doesn't even learn anything and yields a constant output of either 0 or the approximately mean of data that it has seen during training.
Based on experience can you suggest a method might yield me best results or refer me to a paper?

(sub-question) the target i.e., the error values fall in range -1 < error < +1, does this somehow play a crucial rule?

x-axis is number of samples
Edit: details about graph

In the $1^{st}$ subplot, srefpos is not visible because it is very close to the signal slppos. Here is a closer look of the same signal

The reason why I think $2^{nd}$ subplot is important is because the motor rotations (i.e. slppos-sensorless positioning) are calculated based on the ripples in the current signal
The $3^{rd}$ subplot if a the difference of srefpos and slppos, but the error occured in $t_{-n}$ gets carried forward and hence I take another difference to obtain the $4^{th}$ subplot
The signal sampling frequency is 400 Hz(originally 4 kHz but I have down-sampled it here) and this what I meant when the x-axis represents samples(they do occur sequentially)


texts:
I want to choose a clustering algorithm for which I can define the value of centroids (and not only the number of them).

Which algorithm should I look at?

My understanding of k-means is that I can specify 'K' (the number of centroids) but not the value itself.

Instead, I have a given number of centroids that I want to "test". Some of them will be indeed the actual centroid, whilst others will have only few to none values and therefore I will drop them.
However, I don't want to create new values for the centroids.


texts: Can any one tell me what the effects of $L_2$ loss and smooth $L_1$ loss (i.e. Huber loss with $\alpha = 1$) are, and when to use each of them ?


texts: color compression using k-means algorithm

texts:
I am reading about k means algorithm at this link.

At ln[22] here author mentioned that Input color space is 16 million possible colors. How author came up with 16 million number here. Kindly explain.

Additionally at the end it is mentioned as below

  Some detail is certainly lost in the rightmost panel, but the overall
  image is still easily recognizable. This image on the right achieves a
  compression factor of around 1 million! While this is an interesting
  application of k-means, there are certainly better way to compress
  information in images. But the example shows the power of thinking
  outside of the box with unsupervised methods like k-means.

How compression is done here as we are still using data of shape (273280, 3) after compression. How compression of 1 million is achieved.

Thanks


texts: Should features be correlated or uncorrelated for classification?

texts:
I have seen researchers using pearson's correlation coefficient to find out the relevant features --  to keep the features that have a high correlation value with the target. The implication is that the correlated features contribute more information in finding out the target in classification problems. Whereas, we remove the features which are redundant and have very negligible correlation value.

Q1) Should highly correlated features with the target variable be included or removed from classification problems ? Is there a better/elegant explanation to this step?

Q2) How do we know that the dataset is linear when there are multiple variables involved? What does it mean by dataset being linear?

Q3) How to check for feature importance for non-linear case?


texts: Best way to combine two similar document

texts:
I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "relevant" information.

I was thinking to check the texts paragraph wise on its "information value" and then only combine the paragraphs that are the most "relevant".

Any suggestions on an approach how to do this? Any research papers about this topic?


texts: Cannot interpret feed_dict key as Tensor: Tensor Tensor(“Placeholder:0”, shape=(3, 3, 3, 32), dtype=float32) is not an element of this graph

texts:
I have a working Keras model that makes predictions great in the repl but fails to load in a Flask app on accessing multiple times..

K.clear_session() and graph = tf.get_default_graph() did not work for me.

model1 = Sequential()

model1.add(Conv2D(32, 3, padding='same', input_shape=(256,256,3), activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("First layer...")
model1.add(Conv2D(32, 3, padding='same', activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("Second layer...")
model1.add(Conv2D(64, 3, padding='same', activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

model1.add(Flatten())
model1.add(Dense(64))
model1.add(Activation('relu'))
model1.add(Dropout(0.5))
model1.add(Dense(1))
model1.add(Activation('sigmoid'))

model1.load_weights("Weights_model1.h5")

model2 = Sequential()

model2.add(Conv2D(32, 3, padding='same', input_shape=(256,256,3), activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("First layer...")
model2.add(Conv2D(32, 3, padding='same', activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("Second layer...")
model2.add(Conv2D(64, 3, padding='same', activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

model2.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model2.add(Dense(64))
model2.add(Activation('relu'))
model2.add(Dropout(0.5))
model2.add(Dense(1))
model2.add(Activation('sigmoid'))

model2.load_weights("Weights_model2.h5")

Training_water_data = 'C:\\Users\\ysmehta\\'+folder+'\\'
training_water_data = [x for x in sorted(os.listdir(Training_water_data))]
x_train_water_data = np.empty((len(training_water_data),256,256,3),dtype = 'float32')
for i,name  in enumerate(training_water_data):
    im = cv2.imread(Training_water_data + name).astype('int16').astype('float32')/255.
    im = im[:,:,:]
    im = cv2.resize(im,dsize = (256,256),interpolation = cv2.INTER_NEAREST)
    x_train_water_data[i] = im

x_train_water_data = x_train_water_data.reshape(x_train_water_data.shape[0],256,256,3)

for i in range(0,x_train_water_data.shape[0]):
    A = x_train_water_data[i]
    B = A.reshape(1,256,256,3)
    prediction1 = model1.predict(B,verbose=0)
    if (prediction1 >= 0.7):
    prediction2 = model2.predict(B, verbose=0)
else:
    returnstring="No manhole found"
    prediction2 = -1
if(prediction2 >= 0.6):
    returnstring="Water Leakage"

else:
    if(prediction1 >= 0.7):
        returnstring="Not a Leakage"
return returnstring;

if __name__ == "__main__":
    app.run(host='131.163.144.196',port=5000)`


texts:
The training set contains $p$ papers. Each paper is annotated as research or non-research. To develop the research paper filter, we consider the $W$ most frequent phrases in a paper. The research paper filter will use the presence/absence of these $W$ phrases to decide if the paper is indeed a research paper or not.

How many possible hypothesis are there?
I have $2^W$ possible hypothesis, but if we are including conjunctive hypothesis symbols, then I would say $4^W$. Is my logic correct?

How many of them are consistent?
For this question, I have no idea where to begin. Any insight?


texts: Feature Selection with one-hot-encoded categorical data

texts:
I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem.

My professors asked me to perform feature selection using sequential forward selection (mlxtend).

Is there really a point of doing this since it is also very time consuming? Is it logical to remove categorical data? If so, what would be the k_features number for sfs that I should use? Or is the method(sfs) even suited for this?


texts: Difference between rulefit & skope-rules Python Packages

texts:
RuleFit is an implementation of a rule-based prediction algorithm based on the rulefit algorithm from Friedman and Popescu (PDF).
skope-rules
Skope-rules is a Python machine learning module built on top of scikit-learn and distributed under the 3-Clause BSD license.
Both the packages generate rules. Do both the packages rely on The RuleFit algorithm by Friedman and Popescu (2008), and implement the same algorithm?


texts: Best way to store large data set using R from Twitter?

texts: I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R.  have difficulty storing tweets because my machine has only 8 GB of memory. It ran out of memory even before I set it to retrieve for one day. Is there a way where I can store the tweets straight to my disk without storing into RAM? I am not using the streaming API as I need to get old tweets.


texts: Using GRU with FeedForward layers in Python

texts:
I'm trying to reproduce the codes in this paper here for the multi-labeling problem (11 classes), which is using

1- Embedding layer
2- GRU
3- two Feed forward Layers with the ReLU activation function
4- sigmoid unit.

I've tried to run the codes, but it is showing the following error:

  ValueError: Error when checking target: expected dense_5 to have 3
  dimensions, but got array with shape (6838, 11)

Edit: The error is fixed. I changed the "return_sequences" to False, and removed flatten() to fix the error.

My code: i'm not sure if 2 Feedforward layers are correct.
in the paper it stated FF1:1024 units, and FF2: 512 units. with mini-batch size of 32. How can I state it in the code?

target_input=Input(shape=(max_length, ))

target_embedding=Embedding(input_dim=vocabulary_size, output_dim=embedding_dims, #embedding_matrix]
                           input_length=max_length, weights=[embedding_matrix] , trainable=False)(target_input)

#target_embedding=Dropout(0.3)(target_embedding)

target_gru1=Bidirectional(GRU(units=200, return_sequences=True, dropout=0.3, recurrent_dropout=0.3))(target_embedding)
target_gru=Bidirectional(GRU(units=200, return_sequences=False, dropout=0.3, recurrent_dropout=0.3))(target_gru1)

# target_gru=Dropout(0.3)(target_gru)

#2 feedforward layers
# target_output1=Activation("relu")(target_gru)
# target_output2=Activation("relu")(target_output1)

FF1 = Dense(1024)(target_gru)
target_output1=Activation("relu")(FF1)
FF2 = Dense(512)(target_output1)

target_output=Dense(units=11, activation="sigmoid")(FF2)#target_output2)
target_model=Model(target_input, target_output)
## configuring model for training:
opt = Adam(lr=0.0001)#lr=0.001,decay=0.5
target_model.compile(optimizer=opt,loss="binary_crossentropy", metrics=["categorical_accuracy"])

and here is the layers


texts: sckit-learn Cross validation and model retrain

texts:
I want to train a model and also perform cross validation in scikit-learn, If i want to access the model (For instance to see the parameter's selected and weights or to predict) i will need to fit it again.

I believe and according to my understanding if CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data.

I have seen cross_val_predict method and it is not exactly what i need as it is for predicting score and i want model object.

My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step


texts: How to interpret the graph representing the fit provided by the ARIMA model?

texts:
I'm following this tutorial here to build an ARIMA model in R.

I've done a Forecast using a fitted model in R. I specified the forecast horizon h periods ahead for predictions to be made and used the fitted model to generate those predictions. Then I plotted them to see the results and this is what I got:

The light blue line above is supposed to show the fit provided by the model.

In the Tutorial this is what they got (their Dataset is different from mine):

In their case it is clear in contrast with mine. Can somebody provide me with an explanation regarding the graph that I got. Does it make any sense? Does it tell something or is what I got completely wrong (knowing that I followed the tutorial as it is)?

Any insights would be much appreciated.

Thank you.

Here's my code :

  #Removing Outliers

  count_ts = ts(df[, c('qty')])

  df$clean_qty = tsclean(count_ts)

   df$cnt_ma = ma(df$clean_qty, order=7) # using the clean count with no outliers
   df$cnt_ma30 = ma(df$clean_qty, order=30)
   #Data Decomposition
   count_ma = ts(na.omit(df$cnt_ma), frequency=30)
   decomp = stl(count_ma, s.window="periodic")
   deseasonal_cnt <- seasadj(decomp)

   fit2 = arima(deseasonal_cnt, order=c(1,1,7))
   fcast <- forecast(fit2, h=30)
   plot(fcast)


texts: Can I get numeric data from a color map?

texts:
In my class I often need to work with color map images. I would show the image and try to make inferences/observations about different subjects. Often times I need to actually quantify some aspects, but it is always very approximate and somehow vague because the images are provided "as is" and I do not necessarily know their content a priori.
Let's imagine I'm working with two images (*). Is it possible to indicate the computer "learn" the color scale bar so I can click at any point in the map and get the value of altitude at that point? Image 1 is a discrete color scale bar, while Image 2 has a continuous color mapping. I included both examples because maybe there are differences in the approach of how to do this.
So, is there a software/way to do this? Preferably open source. I tried ImageJ and couldn't come up with a decent solution.
Image 1:

Image 2:

(source: topocreator.com)
As you can see, the color scale is part of the image in all cases.
(*) I do not own any of these images, just examples I found online to illustrate my point.


texts: Term for calculated values that lose pertinence when changing scale

texts:
I'm trying to find the term for a type of calculations or values that cannot be simpply added or multiplied when zooming in or out from a temporal scale. I know it's not very clear, if it were I would just google the question.

Here's an example: say you're comparing 2014 vs 2015 sales of ice cream and temperature at day's level. If you want to do an analysis by week, you can just add the sales for seven days, and compare weeks between them, the information still makes sense: there's a fact (amount of money deposited in the bank that week) that is grounded in reality that relate to the new figure. You could also calculate an average by day from the total by week.

But, adding temperatures for a total would not make any sense in reality, you need to calculate an average for that kind of data.

So is there a term for this type of values that can only make sense as average and never be added or multiplied (except as an intermediate step for calculating an average)?


texts: How to improve machine learning model using 2+ datasets

texts:
I am building a supervised machine learning model which (for example) predicts heart failure (yes/no). I have two datasets from 2 different labs A and B, which both have decent distribution, aka it's not like A has way more young people than B, but somehow A has much lower rate of heart failure.
Having created a separate model for each, they each achieve around 90% or higher accuracy, but my aim is to make one model which can use a concatenated dataset containing information from A and B.
Currently this combined model gets 75% accuracy for samples from B and 90% for A.
What I have tried:

sample in different ways / create 'even' datasets
add feature indicating which lab
normalise numerical features
create categories for numerical features

I'm aware the overall results would of course not become as good as the individual models, but how can I make the platforms at least get more similar results?
I appreciate any advice / methods I could try out to tackle this type of problem!!


texts: What is difference between detrend and normalization?

texts:
matlab function detrend subtracts the mean from data. If data contains several data columns, detrend treats each data column separately.

One of the normalization technique is subtracting the mean and dividing it by standard deviation.

Since the normalization already subtract the mean from the data, in such case, is it essential to perform (before or after) detrend operation?

What is the significance of each operation?


texts: Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers?

texts:
In Scikit-Learn, the Local Outlier Factor (LOF) algorithm is defined as an unsupervised anomaly detection method.

So then I don't understand why this algorithm requires pre-filtered training data. Perhaps "training data" here simply means "data to start with?" But the example code provided by SciKit-Learn clearly shows training data which explicitly contains NO anomalies. Does that mean that this model would NOT work if the training data contains anomalies? And more importantly, how do I find anomalies in the training data using this algorithm?

Here is the website.
https://scikit-learn.org/stable/auto_examples/neighbors/plot_lof_novelty_detection.html#sphx-glr-auto-examples-neighbors-plot-lof-novelty-detection-py

I changed the sample code to include anomaly data in the training data set and the model still found a decision boundary that looked correct. So am I just getting confused because of the way the documentation and sample code is written? Or is does this model really need a clean training data set?


texts:
Provided a set ($m$ no. of) of n-dimensional vectors what would be the correct unsupervised approach to cluster them? The vectors essentially represent patterns.

For example:
Set of vector is represented as $V$. Let a vector $v1$ represents a pattern similar to $y = sin(x)$ curve. The $y$ values are stored in $v1$ and the $x$ intervals are same for all the vectors. Similarly there is a vector $v2$ representing pattern similar to $y=log(x)$.

The problem: Does a group of vectors exist, exhibiting similar (not exactly same) pattern as $v1$, similarly for $v2$ and so on?

Therefore these patterns are required to be clustered appropriately. There are methods such as Vector Quantization, but I am not sure if those methods are appropriate in this case.


texts: Mapping column values of one DataFrame to another DataFrame using a key with different header names

texts:
I have two data frames df1 and df2 which look something like this.

    cat1    cat2  cat3
0   10       25     12
1   11       22     14
2   12       30     15

   all_cats  cat_codes
0   10       A
1   11       B
2   12       C
3   25       D
4   22       E
5   30       F
6   14       G

I would like a DataFrame where each column in df1 is created but replaced with cat_codes. Column header names are different. I have tried join and merge but my number of rows are inconsistent. I am dealing with huge number of samples (100,000). My output should ideally be this:

    cat1    cat2  cat3
0    A        D     C
1    B        E     Y
2    C        F     Z

The resulting columns should be appended to df1.


texts: Calculating Confidence Interval at a certain confidence level

texts:
def get_ci(mean, cl, sd):
  loc = stats.norm.ppf(1 - cl/2)
  rng_val = stats.norm.cdf(loc - mean/sd)

  lwr_bnd = value - rng_val
  upr_bnd = value + rng_val

  return_val = (lwr_bnd, upr_bnd)
  return(return_val)

This function takes three parameters which are the following:

mean --> It is the mean
cl --> Confidence Level
sd --> Standard Deviation

Can someone explain how this function is working as for example if we are calculating 95% confidence interval, why can't we just return (-2*sd+mean,2*sd+mean)?


texts: Improve test accuracy for TensorFlow CNN

texts:
I'm trying to use Tensorflow for signal classification. The signals are either normal or high-risk signals. For this purpose, I used convolutional neural networks. The length of signals are 685 and the architecture is:

Convolution layer with 27 channels and 1 by 16 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 14 channels and 1 by 32 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 4 channels and 1 by 32 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 3 channels and 1 by 10 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Fully connected layer with 20 neurons and dropout layer.
Fully connected layer with 10 neurons and dropout layer.
And finally Soft max layer.

After training the network using AdamOptimizer with learning rate 0.001 with 150000 signals the training accuracy is near 95 percent (batch training with 16 batch size is used), however testing accuracy using 20000 new signals is almost 50 percent. Since there is just 2 classes, this accuracy is no better than random guess.

model="conv1d-27-16-1,maxpool-2,conv1d-14-32-1,maxpool-2,conv1d-4-32-1,maxpool-2,conv1d-3-10-1,maxpool-2,full-20,full-10,softmax"

cnn=CNN_1D(model,input_size=685,n_classes=2,num_epochs=20,batch_size=16,dropout=0.75)
cnn.read_data('train_input','train_targets','test_input','test_targets')
cnn.build_model()
cnn.training(validation_set='all')

How can I improve the testing accuracy in my network?


texts: What is the minimum amount of data required for sales prediction with ML

texts:
I have historical data from the MySQL DB which contains 33 months of data.
The features in the data are state, depot, product type, purchase date, sales person name, volume, and price.

Using this information, I would like to predict/forecast product sales weekly, monthly, and yearly.

Since I only have 34 months of data, I do not think that I can apply statistical modeling techniques like Exponential Smoothing or ARIMA.

So, I just wanted to check if this data is sufficient for Machine learning modeling such as Linear Regression, SVM, or Random forest? Or any kind of feature engineering.

Note: I am unable to get more data, as DB has only 34 months of data.

Please suggest.


texts: I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling django.setup() somewhere, I did it in init of my app., but this is not acceptable because it breaks app execution (like runserver)


texts: What are the applications of Solr in Big Data?

texts:
Why is solr used in Big Data?

What is it's purpose?


texts: How to preprocess an ordered categorical variable to feed a machine learning algorithm?

texts:
I have a categorical variable that measures the income of a family:
A: no income
B: Up to $500
C: $500-$700
…
P: $5000-$6000
Q: More than \\\$6000

It seems odd to me that I have to get dummies for this variable, since it's ordered. I wonder if it's better to map the values: {'A': 0, 'B': 1, …, 'Q': 17} so I can input it into the algorithm this values as integer numbers.
What's the proper way of preprocessing this variable to feed an algorithm such as Random Forest or a simple neural network?


texts: Is there a way to measure the "sharpness" of a decision boundary of a CNN?

texts:
It is commonly seen as something bad if the decision boundary of a neural network is too sharp, meaning if slight changes in the input change the class prediction completely.

Given a trained CNN, is it possible to measure / calculate the "sharpness" of its decision boundaries? Did somebody do that already?


texts: what metrics to evaluate rank order results?

texts:
I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem).

Anyway, say I have two sets of prediction results, as show by df1 and df2.

y_truth = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
y_predicted_rank1 = [6, 1, 7, 2, 8, 3, 9, 4, 10, 5]
y_predicted_rank2 = [4, 1, 7, 2, 8, 3, 9, 6, 10, 5]
df1 = pd.DataFrame({'tag': yy_truth, 'predicted_rank': y_predicted_rank1}).sort_values('predicted_rank')
df2 = pd.DataFrame({'tag': yy_truth, 'predicted_rank': y_predicted_rank2}).sort_values('predicted_rank')

print(df1)

#   tag predicted_rank
#1  1   1
#3  1   2
#5  1   3
#7  1   4
#9  1   5
#0  0   6
#2  0   7
#4  0   8
#6  0   9
#8  0   10

print(df2)
#   tag predicted_rank
#1  1   1
#3  1   2
#5  1   3
#0  0   4
#9  1   5
#7  1   6
#2  0   7
#4  0   8
#6  0   9
#8  0   10

By looking at them, I know df1 is better than df2, since in df2, a negative sample (zero) was predicted to have rank #4. So my question is,  what metric can be used here so that I can (mathematically) tell df1 is better than df2?


texts:
I have been reading through this blog in order to find what mAP is .In the sub heading of AP, they give the example of 5 apple images and finding out the average precision.As far I understand false positive is when the object is localised and classified but IOU < 0.5 (in the blog) and false negative is when the model fails to identify an object itself.So what about objects which are misclassified dont they belong to false postives?.

Also what does the table in the blog really respresent.The 'correct?' is for one particluar example or 5 examples together.Could you just brief me what is going on in your own terms or just what the blog says?


texts: Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

texts:
I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one position for one office. I am storing this information into a preset "key" for what the offices are... so when there are two individuals for one position in one office it just duplicates the information except for where there was the NAME1/NAME2.  The first line will use NAME1 and the second line will use NAME2.
For example:
key1  ...    NAME1
key1  ...    NAME2
key2  ...    NAME3
key3  ...    NAME4

I'd like to make this end up as:
key1  ...    NAME1/NAME2
key2  ...    NAME3
key3  ...    NAME4

So that the duplicates are merged by adding a / ... This should be as simple as string concatenation. But I am struggling without making a bunch of loops to do this.


texts: How to control for Co-variate shift in test data set compared to train data for regression task?

texts:
I am working on a regression project. But I am facing the problem of covariate shift in features due to time delay.Test data was collected a year later due to which there has been some change in distribution. The research paper that I am working on also mentioned this shift in feature values but didn't mention anything about rectifying it.
I found this but it talks about changing training set distribution.
What I need is a way to make test data distribution closer to training data or some other way in which I could control for it.
I am using sklearn and Elastic-net for regression.


texts: Getting TypeError: expected bytes, Descriptor found while importing tensorflow

texts:
I am trying to use tensorflow and keras for one of the tasks but while importing the tensorflow I am getting the below error. Till now I have removed the virtual which I have created previously and then created the new virtual environment again, but still I am getting the same error. Any leads would be appreciated. Here is the command below which I am running.

import tensorflow as tf

Here is the error I am getting

runfile('C:/Users/Lenovo/.spyder-py3/temp.py', wdir='C:/Users/Lenovo/.spyder-py3')
Traceback (most recent call last):

  File "C:\Users\Lenovo\.spyder-py3\temp.py", line 1, in <module>
    import tensorflow as tf

File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow__init__.py", line 28, in
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\python\__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2

      File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 16, in <module>
   from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 16, in <module>
   from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 92, in <module>
__module__ = 'tensorflow.core.framework.resource_handle_pb2'

TypeError: expected bytes, Descriptor found

Below is the configuration of my system.

(myenv) C:\Users\Lenovo>conda list tensorflow
# packages in environment at C:\Users\Lenovo\Anaconda3\envs\myenv:
#
# Name                    Version                   Build  Channel
tensorflow                1.14.0                   pypi_0    pypi
tensorflow-estimator      1.14.0                   pypi_0    pypi

(myenv) C:\Users\Lenovo>conda list keras
# packages in environment at C:\Users\Lenovo\Anaconda3\envs\myenv:
#
# Name                    Version                   Build  Channel
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.0                    pypi_0    pypi

(myenv) C:\Users\Lenovo>


texts: How to get the number of syllables in a word?

texts:
I have already gone through this post which uses nltk's cmudict for counting the number of syllables in a word:

from nltk.corpus import cmudict
d = cmudict.dict()
def nsyl(word):
  return [len(list(y for y in x if y[-1].isdigit())) for x in d[word.lower()]]

However, for words outside the cmu's dictionary like names for example: Rohit, it doesn't give a result.

So, is there any other/better way to count syllables for a word?


texts: Concat two tensors of different dimensions

texts:
I have two tensors. For example -
a = torch.randn((500, 200, 10))
b = torch.randn((500, 5))

I have to concat each of b tensor to all elements of corresponding a tensor i.e., each 200 tensors of a[0] should get concatenated with b[0] - final dimension should be (500, 200, 15).
Without using explicit for loop, how can I achieve this in Pytorch efficiently?


texts:
This is my first NLP project. I'm trying to use SMOTE for a classifier with 14 classes. I need to convert the classes into an array before using SMOTE. I tried using MultiLinearBinarizer but it does not seem to be working. From the stack trace, it seems like everything is getting converted.  Do I need to convert something else to an array? How would I do that?

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.preprocessing import MultiLabelBinarizer

nb = Pipeline([('vect', CountVectorizer()),
               ('tfidf', TfidfTransformer()),
               ('clf', MultinomialNB()),
              ])
nb.fit(X_train, y_train)

mlb = MultiLabelBinarizer()
print(mlb.fit_transform(df["Technique"].str.split(",")))
print(mlb.classes_)

import imblearn
from imblearn.over_sampling import SMOTE

smote = SMOTE('minority')

x_sm, y_sm = smote.fit_sample(X_train, y_train)
#print(x_sm.shape, y_sm.shape)
pd.DataFrame(x_sm.todense(), columns=tv.get_feature_names())

I'm getting the error ValueError: could not convert string to float: 'left left center'

Here is the stack trace

[[1 0 0 ... 0 0 0]
 [1 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
['Appeal_to_Authority' 'Appeal_to_fear-prejudice' 'Bandwagon'
 'Black-and-White_Fallacy' 'Causal_Oversimplification' 'Doubt'
 'Exaggeration' 'Flag-Waving' 'Labeling' 'Loaded_Language' 'Minimisation'
 'Name_Calling' 'Red_Herring' 'Reductio_ad_hitlerum' 'Repetition'
 'Slogans' 'Straw_Men' 'Thought-terminating_Cliches' 'Whataboutism']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-45-68681190410f> in <module>()
     10 smote = SMOTE('minority')
     11
---> 12 x_sm, y_sm = smote.fit_sample(X_train, y_train)
     13 #print(x_sm.shape, y_sm.shape)
     14 pd.DataFrame(x_sm.todense(), columns=tv.get_feature_names())

8 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a, dtype, order)
     83
     84     """
---> 85     return array(a, dtype, copy=False, order=order)
     86
     87

ValueError: could not convert string to float: 'left left center'
```


texts: Extracting rules in regression tree in Python

texts:
I don't know if this question is a complicated question or not. I wanna train a regression tree in which in leaves, linear regression is applied to predict. Then, when the tree and linear regression models are trained, extract all rules and regression models.
Is there any package or library to do this?


texts: Which features can help to differentiate two similar images in deep learning?

texts:
I have used the Inception V3 model. There were 11 classes in training out of 5 classes that are (Visually look) Similar and that 5 classes were misclassified with other classes when testing the images.
So, What Should I do to differentiate those classes from others? Or any other parameters should I consider for differentiating?


texts: The effect of the image type and the image conversion on deep learning CNN model

texts:
Does the type of the image affects (jpg, png, bmp) on the CNN deep learning algorithm?

Dose converting the image type affects on the CNN deep learning algorithm (ex. converting bmp to jpg or ppm to jpg)?

Thanks


texts: How to detect payment's fraud using Machine Learning?

texts:
While browsing Machine Learning problems, I came across this question:
"Given a month’s worth of login data from Netflix such as account_id, device_id, and metadata concerning payments, how would you detect payment fraud?"
Now, I would approach this problem as a supervised learning exercise    and leverage past data as training data where hopefully, a response was provided for each sample and determined to be fraudulent or not.

I would train a classification model on this training data with the
objective of minimizing the False Positives (Type I error), i.e. after obtaining a model, I
would set the probability threshold to be low so as to minimize the
False Negatives.
I am thinking that most of these problems are usually highly
imbalanced (with very few cases of fraud), and so, I would probably
not use accuracy as a performance metric to evaluate the performance
of the model (and use precision instead, or perhaps precision-recall AUC).

Now I am wondering if my approach is correct, is there anything missing from my Logic?
In the past I have solved similar 'fraud' problems, however, the question were regarding credit card transactions, and the challenge was to detect whether a particular transaction was fraudulent or not.
And so, for this particular Netflix problem, I am not sure if the fact that, here, the challenge is to detect whether payments are fraudulent (instead of transactions) makes any difference or not.
I am curious to hear what other people think.


texts: XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

texts:
So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used with scikit's RF regressor and xgboost's Regressor. Specifically the part about tuning for the number of trees/iterations/boosting rounds. From my understanding each of those terms are the same. They determine how many times a decision tree should be calculated based on the algorithm. However, should I be referring to them as ntrees or n_estimators? Or should I simply use early stopping rounds for my xgboost and tune the number of trees only for my rf?

My Random Forest:

rf = RandomForestRegressor(random_state = 13)
param_grid = dict(model__n_estimators = [250,500,750,1000,1500,2000],
                  model__max_depth = [5,7,10,12,15,20,25],
                  model__min_samples_split= [2,5,10],
                  model__min_samples_leaf= [1,3,5]
                  )
gs = GridSearchCV(rf
                  ,param_grid = param_grid
                  ,scoring = 'neg_mean_squared_error'
                  ,n_jobs = -1
                  ,cv = 5
                  ,refit = 'neg_mean_squared_error'
                  )

My xgboost

model = XGBRegressor(random_state = 13)
param_grid = dict(model__ntrees = [500,750,1000,1500,2000],
                  model__max_depth = [1,3,5,7,10],
                  model__learning_rate= [0.01,0.025,0.05,0.1,0.15,0.2],
                  model__min_child_weight= [1,3,5,7,10],
                  model__colsample_bytree=[0.80,1]
                  )
gs = GridSearchCV(model
                  ,param_grid = param_grid
                  ,scoring = 'neg_mean_squared_error'
                  ,n_jobs = -1
                  ,cv = 5
                  ,refit = 'neg_mean_squared_error'
                  )


texts: Linear Regression bad results after log transformation

texts:
I have a dataset that has the following columns:

The variable I'm trying to predict is "rent".
My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent column and the area column using log transformation since both columns had a positive skewness. Here's the rent column and area column distribution before and after the log transformation.
Before:

After:

I thought after these changes my regression models would improve and in fact they did, except for Linear Regression.
If I don't do any type of transformations the models underperform. When I only transform the rent column all models improve including Linear Regression, but when I transform the rent column and the area column Linear Regression has a terrible result with a MAPE of 2521729.47.
Not transforming area MAPE results:

Transforming area MAPE results:

Can anyone tell me what's probably happening or guide me through any type of testing or verifications to understand what's happening to linear regression? Am I wrong to transform those columns even if the models are improving?
Edit:
After testing the models by removing and adding columns, I found that linear regression goes crazy after I insert the neighborhood column (which contains 66 neighborhoods) and create dummy columns. When I create this dummy variables the number of columns goes to 77, while the dataset has only around 3000 rows.
My thoughts are that after transforming the column into dummy columns the data becomes very sparse and with too many features for only 3000 rows, and that's why Linear Regression has this bad performance and Lasso Regression doesn't. Besides that, I should probably still use the other models since they perform well after the changes.
Am I correct?


texts: Why we can't Remove features with missing values in Data Preprocessing

texts:
In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values preprocessing.
So, Can we use mean, median, standerd deviation or we can remove those whole records?
Why many people denying to remove complete record from dataset, Why?


texts: "Pip install tensorflow" only installs version 1.14 – how to install version 2 (with eager mode enabled?)

texts:
The question is already in the title: when installing tensorflow in command line (in a virtual environment or not...), it is always tensorflow version 1.14.0 which is going to be installed. Consequently, eager execution does not work, I get error messages ("Tensor Object has no attribute numpy..." etc.). However, the tensorflow documentation states that it should the recent tensorflow 2 version which is in fact the one I need. I would be glad to read your advice on this issue.

Kehrwert


texts: How to classify test objects with this ruleset in order of priority?

texts:
I'm coding a program that tests several classifiers over a database weather.arff, I found rules below, I want classify test objects.

I do not understand how the classification, it is described:
"In classification, let R be the set of generated rules and T the training data. The basic idea of the proposed method is to choose a set of high confidence rules in R to cover T. In classifying a test object, the first rule in the set of rules that matches the test object condition classifies it. This process ensures that only the highest ranked rules classify test objects. "

How to classify test objects?

No. outlook temperature humidity    windy   play
1   sunny       hot     high        FALSE   no
2   sunny       hot     high        TRUE    no
3   overcast    hot     high        FALSE   yes
4   rainy       mild    high        FALSE   yes
5   rainy       cool    normal      FALSE   yes
6   rainy       cool    normal      TRUE    no
7   overcast    cool    normal      TRUE    yes
8   sunny       mild    high        FALSE   no
9   sunny       cool    normal      FALSE   yes
10  rainy       mild    normal      FALSE   yes
11  sunny       mild    normal      TRUE    yes
12  overcast    mild    high        TRUE    yes
13  overcast    hot     normal      FALSE   yes
14  rainy       mild    high        TRUE    no

Rule found:

1: (outlook,overcast) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 3, 7, 12, 13]

2: (humidity,normal), (windy,FALSE) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 5, 9, 10]

3: (outlook,sunny), (humidity,high) -> (play,no)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 1, 2, 8]

4: (outlook,rainy), (windy,FALSE) -> (play,yes)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 4]

5: (outlook,sunny), (humidity,normal) -> (play,yes)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 11]

6: (outlook,rainy), (windy,TRUE) -> (play,no)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 6, 14]

Thanks,
Dung


texts:
I put a dropbox link to the URL box in the File widget and the data got imported perfectly. I saved the workflow that I would like to share with my team. But when I reopen the workflow, the URL link is gone. Every time you save and close the workflow, the URL link get erased.

Is there a way that I can save the URL link info in the .ows file?


texts: Best ML approach for huge number of classes

texts:
I have an problem where the dataset consists of:

400k observations
40k classes (mutually exclusive)

The problem is about predicting what is the supplier of an invoice (from which supplier/shop a purchase was made) based on the text of the invoice.
As you can understand there are hundreds of thousands if not millions of suppliers in a country hence the so big number of (mutually exclusive) classes.
In terms of features, I use a TF-IDF algorithm so I end up with 1.5M features. I apply feature selection and they are eventually relatively reduced in number. In any case, the features approach is something that I can and I am open to change so the number of features should not be considered as fixed.
However, this huge number of classes (along with the number of observations, features) results in me having to use huge amount of hardware to make things run and in many cases the VMs just die because of lack of memory.
All these happen when using ML algorithms such as random forest or SGD.
I am thinking that may be Tensorflow and GPUs can be a solution (?) although again this is a kind of hardware solution so I am wondering if I should reframe somehow the problem to make its solution more scalable.
How would you approach this problem IN AN ML WAY to make things scalable given also that the number of classes will be significantly growing in the future (finally there will be around 1M classes)?