texts: Multiclass Classification on live sensor data

texts:
I want to use an accelerometer to detect which way a train is heading.
You could do this by just setting thresholds manually to detect the direction of acceleration on the x-axis (which would be aligned to the trains driving axis) after a standstill.

I still want to solve this problem using machine learning, because setting the thresholds takes quite some time.
The sensor data has to be in a context, otherwise its useless. The driving direction can only be estimated if you know the n sensor readings before.

So now I could probably use a RNN to create this context.
My ideas was it to not use a RNN (since it adds complexity) and just put several measurments in one sample. It would look like this:
accelX,accelY,accelZ,accelX1,accelY2,AccelZ2....AccelXn,AccelYn,AccelZn, label

I also measure the Y- and Z-axis because they are important to tell if you are standing still or just driving very smoothly without much acceleration (you have peaks in the Z-axis because you drive over bumps and in curves you have peaks on the Y-axis)

These samples would be taken one after another. This would create the context the net needs to judge it the sample was either standstill,accelerating positive on the X-Axis, accelerating negative on the X-Axis or if the train is just driving normally.

After my net would be trained I could just measure, pack and feed these samples in my net and in the end get a classification. If now one sample is classified as standstill and the next following one as accelerating+, I'd now that we are now driving forward. If a driving then standstill then accelerating- follows I then know, that we are now driving backward.

My dataset will be quite unbalanced since the train is mostly driving and only accelerating once after every station. In addition, my dataset won't be too big since I can only record data when I'm in the train.

I would like to use keras for building my model since its easy to understand.

Now my questions:

Do you think my approach will work?
What model architecture do you suggest me?
Should I manually classify my samples or use unsupervised learning?
Should I also include other features (variance of the data ..)

Thanks for the help


texts: How many model parameters did Microsoft's 2015 ImageNet-winner network have?

texts:
In 2015 a team of researchers from Microsoft won the ImageNet contest with a network of 152 layers.

How many parameters did their model have?


texts: Continuous variable not supported in confusion matrix

texts:
I am using linear regression algorithm for a data set. And trying to compute confusion matrix between y_pred and y_test. I am getting "ValueError : continuous is not supported" error.

I have included the code below. Help to solve this problem.

x = data_set.iloc[:, :-1].values
y = data_set.iloc[:, 7].values

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 0)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import confusion_matrix

regression = LinearRegression()
regression.fit(x_train, y_train)

y_pred = regression.predict(x_test)

cm = confusion_matrix(y_test, y_pred)


texts: Serializing a trained classification model into a set of actionable insights

texts:
I'm looking for ways to convert a trained classification model into a list of insights based on the resulting parameters of the model.

To make an example, let's assume we trained a decision tree to predict a binary outcome (0 or 1). We ended up with a simple tree that has 7 nodes (A - root, B / C - level 1, D / E - level 2 from B, F / G level 2 from C).

From this model, I want to build a list with all combinations of decisions (e.g. A -> B, A -> C, A -> B -> D, etc.) and a couple of parameters to prioritize the classification power of each decision:

Confidence: The purity of the current decision (e.g., what percentage of the classified data falls under the predicted class, "0" or "1")
Support: The fraction of the entire data set covered by the current decision

Here are the sample rows of my list, assuming that:

A -> B is "Feature N > 5", 80% - 0 / 20% - 1, covers 100% of records

B -> D is "Feature K <= 100", 70% - 1 / 30% - 0, covers 40% of records

(column separator is "|")
Decision    | Conditions                               | Confidence | Support
A -> B      | Feature N > 5                            | 80% - 0    | 100%
A -> B -> D | Feature N > 5 & Feature K <= 100         | 70% - 1    | 40%

Such a list would allow me to quickly find the most impactful decisions by comparing them across confidence and support, and understand exactly what conditions are in play for the prioritized decisions. The second part gives me great insight to act on the findings.

The use case seems common enough to hope for some packaged solutions, but I would appreciate any guidance on how a Python beginner can approach this task from scratch.

A bonus question I have is whether we can extend this to a random forest model?


texts:
I tried to cluster data points by k-medoids where k-centroids/medoids are randomly picked from (X), how those centroids are recalculated when there are some clusters changing by the updating x1? without reclustering
lets assume that there is a number if round for updating x1 of X, How are the new centroids picked when the cluster of user 5 is changed to cluster 1? and the cluster of user 0 is changed to cluster 2? (when x1 values are updated) and so on.


texts: I have time series for product usage over an year on daily basis. Product usage exhibits seasonality i.e. it usage increase/decreases by more than the normal usage during that time. When i get the current day's data and its more than normal, i want to check if its exhibits seasonality or not and take some actions accordingly. I am thinking of using Double Exponential Smoothing to achieve this. Is this an right approach or is there an better approach to do it.


texts:
I want to build a fast binary classifier that decides if an image belongs to a given class (e.g. if it is a picture of a person). I want to do this by training a network on the RGB of pixels at a predetermined set of coordinates (e.g. 4 points, one near each corner of the image) and I want to achieve at least 75% accuracy.

How many points and what architecture should I use? Or if this is a very bad method what is another way to build a classifier that makes training and classification as fast as possible maybe on the account of lowering accuracy?


texts:
I was reading through many blogs and understood the relevance and scenario of having merging two model. But, how to decide the Merge mode for two models. ie. concat, sum, dot, etc.

For eg. I am working on the task of Auto Image captioning. So, captions and Images are 2 kinds of input that I need to handle and merge them at certain point for model to let know which caption is for which image. I am learning the text representation and Image representation by 2 different network designs. Now, at the state after learning representation for both the inputs, how do i decide what ways(concat, add, etc) to use to join/merge two representations.


texts: Approximating multi-variable function with neural network in python

texts:
I'm trying to use 2-5-1 neural network to approximate function $$x_1 \in[-3,3], x_2\in[-1,3],  f(x_1,x_2)=\sin(2x_1+x_2)$$

I used code from Implementing a flexible neural network with backpropagation from scratch, to avoid using any complex libraries and tried teaching my network to approximate with following data:

# Define dataset
n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
for ee in range(0, 4):
    for e in range(1, 4):
        alpha = e / 10 ** ee
        nn = NeuralNetwork()
        nn.add_layer(Layer(2, 5, 'tanh'))
        nn.add_layer(Layer(5, 1, 'sigmoid'))
        errors = nn.train(Xs, Zs, alpha, 300)
        print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))
        # Plot changes in mse
        plt.plot(errors)
        plt.ylim([0, 1])
        plt.title(str('Changes in MSE - alpha ' + str(alpha)))
        plt.xlabel('Epoch (every 10th)')
        plt.ylabel('MSE')
        plt.show()

But I can't seem to have MSE lower than $0.4$. What can I do here to make it more accurate?


texts: How to understand the return values of scipy.interpolate.splrep

texts:
Background

Continuation of Spline interpolation - why cube with 2nd derivative
as following Cubic Spline Interpolation in youtube. The example in the youtube is below.

Implemented using scipy.interpolate.splrep and try to understand what the returns of the splrep function are.

  Given the set of data points (x[i], y[i]) determine a smooth spline approximation of degree k on the interval xb <= x <= xe.
  Returns
  tck : tuple
  A tuple (t,c,k) containing the vector of knots, the B-spline coefficients, and the degree of the spline.

import numpy as np
from pylab import plt, mpl

plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'
%matplotlib inline

def create_plot(x, y, styles, labels, axlabels):
    plt.figure(figsize=(10, 6))
    for i in range(len(x)):
        plt.plot(x[i], y[i], styles[i], label=labels[i])
        plt.xlabel(axlabels[0])
        plt.ylabel(axlabels[1])
    plt.legend(loc=0)

x = np.array([3.0, 4.5, 7.0, 9.0])
y = np.array([2.5, 1.0, 2.5, 0.5])
create_plot([x], [y], ['b'], ['y'], ['x', 'y'])

import scipy.interpolate as spi
interpolation = spi.splrep(x, y, k=3)

IX = np.linspace(3, 9, 100)
IY = spi.splev(IX, interpolation)

create_plot(
    [x, IX],
    [y, IY],
    ['b', 'ro'],
    ['x', 'IY:interpolation'],
    ['x', 'y']
)

Questions

How to interpret and understand the return values and which resources to look into to understand?

  A tuple (t,c,k) containing the vector of knots, the B-spline coefficients, and the degree of the spline.

The return value on Knots

interpolation[0]

array([3., 3., 3., 3., 9., 9., 9., 9.])

I thought the first tuple element would be the knots which would be the x, but not. What are these 3., 3. ... values?

The return values on B-spline co-efficient

interpolation[1]

array([ 2.5 , -2.21111111, 6.18888889, 0.5 , 0. , 0. , 0. , 0. ])

Please help or suggest where I should look into and what to understand about "B-spline coefficient" to be able to interpret these values?

The solution of the first interval is (0.186566, 1.6667, 0.24689), hence I thought these values would be in the 2nd element, but not. How the solution values would relate to the return values?


texts: How can it be that a xgboost.cv cross-validation operation where n-folds are evaluated is quicker than a single XGBoostClassifier.fit(X,y) of the xgboost.sklearn API?


texts: Python sklearn - average classification reports

texts:
Using Python's sklearn module,

 from sklearn.metrics import classification_report
 y1_predict = [0, 1, 1, 0]
 y1_dev = [0, 1, 1, 0]
 report_1 = classification_report(y1_dev, y1_predict)
 y2_predict = [1, 0, 1, 0]
 y2_dev = [1, 1, 0, 0]
 report_2 = classification_report(y2_dev, y2_predict)

Is there a way to combine (maybe just an average) report_1 and report_2 I'm looking for an implementation like:

 report_average = average(report_1,report_2)

Or does this have to be done manually? I was hoping that printing the report_average would have average values between the two reports.

Here's a MWE of the accepted answer:

    from sklearn.metrics import classification_report
    import pandas as pd
    import numpy as np
    from functools import reduce
    def report_average(*args):
        report_list = list()
        for report in args:
            splited = [' '.join(x.split()) for x in report.split('\n\n')]
            header = [x for x in splited[0].split(' ')]
            data = np.array(splited[1].split(' ')).reshape(-1, len(header) + 1)
            data = np.delete(data, 0, 1).astype(float)
            avg_total = np.array([x for x in splited[2].split(' ')][3:]).astype(float).reshape(-1, len(header))
            df = pd.DataFrame(np.concatenate((data, avg_total)), columns=header)
            report_list.append(df)
        res = reduce(lambda x, y: x.add(y, fill_value=0), report_list) / len(report_list)
        return res.rename(index={res.index[-1]: 'avg / total'})

    y1_predict = [0, 1, 1, 0]
    y1_dev = [0, 1, 1, 0]
    report_1 = classification_report(y1_dev, y1_predict)
    y2_predict = [1, 0, 1, 0]
    y2_dev = [1, 1, 0, 0]
    report_2 = classification_report(y2_dev, y2_predict)

    report_ave = report_average(report_1,report_2)

    print(report_ave)

Which yields

             precision  recall  f1-score  support
0                 0.75    0.75      0.75      2.0
1                 0.75    0.75      0.75      2.0
avg / total       0.75    0.75      0.75      4.0


texts: The TF-IDF is not matching with the bags of words

texts:
I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different.
For bag of words I am using:
total_words = df['description'].str.split(expand = True).stack().value_counts()
total_words = total_words.index.tolist()

For vectorizer I am using:
vect = TfidfVectorizer()
tfidf = vect.fit_transform(df['description'])
vect.get_feature_names()

I am removing words that have length < 3. However, I am seeing 2, 3 letter word in the feature_names, but when I check this in the total_words everything is fine.


texts: classification algorithms for anomaly detection when there is class imbalance

texts:
My target variable is boolean and has 0.001% of NO records and remaining YES records.
Can anyone suggest algorithms which are best for anomaly detection in R when there is severe class imbalance.


texts: NLP Emotion Detection - Model fails to learn to recognize negations

texts:
I am working on a nlp emotion detection project. The emotions that I try to predict are 'joy', 'fear', 'anger', 'sadness'. I use some publicly available labeled datasets to train my model e.g. ISEAR, WASSA etc. I have tried the following approaches:

Traditional ML approached using bigrams and trigrams.
CNN with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (CNN 1D) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)
LSTM with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (LSTM/GRU) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)

The NN models achieve more than 80% accuracy but still when I use the trained model to predict the emotion on text that includes some negation I get the wrong results. For example:
Text : "I am happy with easy jet, it is a great company!"
Predicts Happy
Text: I am not happy with easyjet #unhappy_customer
Predicts Happy
Any suggestions on how to overcome this problem?


texts: Final Model fitting - subset vs entire training data

texts:
If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training data?
For example, if I have 1M samples available and I took a 100K random samples as a test holdout and 200K random samples as a training dataset for model tuning, should the tuned hyperparamaters used to fit the final model on the 1) 200K training dataset, or 2) 900K available data (excluding test holdout)? In other words, can the hyperparameters by generalized for the entire population?
I am assuming that both the holdout and training datasets are selected randomly and follow the class distribution in the original data.


texts: Time series forecasting dilemma. Could feature engineering overcome time dependency?

texts:
I keep reading articles about time series forecasting.

They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.

The motivation often comes with some example data showing some trend/seasonality. Also, other supporting points are:

The data distribution (mean, variance) changes over time.
Traditional train/test splits won't make sense since what is the point of forecasting January data with September data?

Fair enough. But let me try to point out this example. Let's say we have a simple timestamp, variable dataset and we are trying to predict the value at t+1

  |  timestamp |  value |
  |  01/01/2019|   10   |
  |  01/02/2019|   12   |
  ...
  |  31/12/2019|   ???   |

What we know is that there is no trend it's very weekly-cyclic instead, which means the value at t will probably depend on its value at t-7 days.
We also know that depending on whether it's a day during the week or during the weekend, data will change accordingly.

What prevents me to use some basic feature engineering and transform the example data as follow?

  |  timestamp | value_at_t_minus_7 | day_of_week |  value |
  |  01/01/2019|   11   |   02   |   10   |
  |  01/02/2019|   12   |   03   |   12   |
  ...
  |  31/12/2019|   10   |   02   |   ???   |

It is not time dependant from a formal perspective but the correlation between its lagged values and the information about the day of the week should take me where I want to, being able to use now classic and flexible methods such as Random Forest, XGB and also splitting train and test (of course leaving the holdout set for validation) to get a good sense of how my model is performing.

Could anyone offer their input supporting with some proper motivation?

Thanks!


texts: Get Gate Activation of GRU-Cells in TensorFlow

texts:
I implemented a standard RNN in Tensorflow (0.12) using the classes

tf.python.ops.rnn_cell.GRUCell
tf.nn.dynamic_rnn

I am interested in the gates and which values they take during. Unfortunately the function dynamic_rnn doesn't support this and just gives the 'output' and 'state' (The final hidden state). In the code of the GRUCell the gates are called 'r' and 'u' for reset and update.

Is there a clever way to save the values of the gates over time or do I have to write my own GRU cell?

Thanks!


texts: What is the difference between AI, ML, NN and DL?

texts:
What is the difference between the following four categories:

Artificial Intelligence (AI)
Machine Learning (ML)
Neural Network (NN)
Deep Learning (DL)
Data Science

My current understanding is that each of these encapsulates some set of algorithms. I feel like, amongst ML, NN, and DL, Machine Learning (ML) is the largest set, which contains NN, which in turn contains DL. See figure below. Data Science is the largest set amongst all 5. Is this correct? Where would AI be on this figure?


texts: Classify if someone is home based on time

texts:
I have a dataset with locations and a timestamp of a subject. For each location and timestamp I determined by comparing the location to the home address if the subject was at home or not (0/1) and added this value to the dataset.

Now, I want to train a model to learn based on the timestamp when it is most likely that the subject is at home. Thus, if you give the model some timestamp, it will classify if the subject was at home at this time. The model learns the "best time" for someone being at home so to say.

Obviously people are not at home at the same time every day but over a long period of time there should be some pattern and I want the model to classify based on this pattern.

What would be a fitting algorithm to do this?


texts: Generate predictions that are orthogonal (uncorrelated) to a given variable

texts:
I have an X matrix, a y variable, and another variable ORTHO_VAR.  I need to predict the y variable using X, however, the predictions from that model need to be orthogonal to ORTHO_VAR while being as correlated with y as possible.

I would prefer that the predictions are generated with a non-parametric method such as xgboost.XGBRegressor but I could use a linear method if absolutely necessary.

This code:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
TARGET = 'target'
PRED = 'yhat'

# Create regression dataset with two correlated targets
X, y = make_regression(n_features=20, random_state=245, n_targets=2)
indep_vars = ['var{}'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df[PRED] = xgb.predict(df[indep_vars])

# Correlation should be low or preferably zero
pred_corr_w_ortho = df.corr().abs()[PRED][ORTHO_VAR]
assert pred_corr_w_ortho < 0.01, "Correlation score: {0} is superior to the given threshold.".format(pred_corr_w_ortho)

Returns this:

---------------------------------------------------------------------------
AssertionError
      1 pred_corr_w_ortho = df.corr().abs()[PRED][ORTHO_VAR]
----> 2 assert pred_corr_w_ortho < 0.01, "Correlation score: {0} is superior to the given threshold.".format(pred_corr_w_ortho)

AssertionError: Correlation score: 0.5895885756753665 is superior to the given threshold.

...and I would like something that maintains as much predictive accuracy as possible while remaining orthogonal to ORTHO_VAR


texts: Using Lift charts with multinomial classification model

texts:
I’m trying to understand the use of Lift charts with multinomial classification model in the evaluation phase. I can see only one category can be selected to use in ‘x’ axis e.g drugA .

What kind of insight we can get by using a target at a time to create a Lift chart to understand how good is the model?

This is the example I’m trying to understand https://www.ibm.com/support/knowledgecenter/en/SS3RA7_sub/modeler_tutorial_ddita/clementine/clemtut_intro_buildingstreams.html


texts: What are the basic approaches for balancing a dataset for machine learning?

texts: Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?


texts: Why might a neural network consistently underestimate its target?

texts:
I have a neural network (MLP) that is consistently underestimating the target variable on the validation set, test set, and on the training set (by about the same amount as on the validation set and test set). In other words, the sum of the entries in the neural network regression output column (Y_hat) is something like 10% less than the sum of the entries in the target variable column (Y). The entries in the target column are greater than or equal to zero, with a decent number of zero entries. The right-side tail of the distribution of the target variable is long.

Fitting the neural network using many random seeds consistently leads to similar results (the neural networks are all biased in the same direction and by a similar amount).

The problem seems to be fairly robust to changes in important hyperparameters, including:
- early stopping
- learning rate schedule
- model complexity
- regularization (dropout, batch norm)

The problem is less severe for low-capacity neural networks.

Does anyone have any ideas as to why this persistent underestimation might be happening?


texts: Training a Supervised Classifier using Continuous Data

texts:
I am trying to build a NLP classification model using methods such as XGBoost, SVM, logistic regression. The features I am trying to include are cosine similarity and LDA topic models, all of which are continuous data. The response variable is also a continuous variable where the values range from 5 - 20 (they are performance scores). The output of the classifier will produce the dependent variable Y which will be a number between 0 and 1 indicating the percent chance that the replication of the original text is a 1. Below is a table representation of my input data type. Is there a way to build a classifier using this type of data?

Script ID
Cosine Similarity
LDA Topic 1
topic 2
topic 3
Target output

1
0.634
0.6
0.4
0
1

2
0.245
0.7
0.1
0.4
0

3
0.975
0.9
0.3
0.3
1


texts: Breaking down a column in Pandas into a separate CSV for display in Tableau

texts:
My data is coming from a CSV, which should be visualized in Tableau.

However, the data contains the column category_list, which consists of values separated by a vertical bar (|).

Since Tableau can't handle arrays inside of attributes, I used Python (Pandas) to load the CSV and manipulate the data:

import pandas as pd
companies = pd.read_csv("companies.csv")

I assume that the category_list column needs to be broken down and stored into another CSV (containing the permalink (unique ID) and category pairs).

Something like this:

permalink,category
/organization/-qounter,Application Platforms
/organization/-qounter,Real Time
/organization/-qounter,Social Network Media
/organization/-the-one-of-them-inc-,Apps
/organization/-the-one-of-them-inc-,Games
/organization/-the-one-of-them-inc-,Mobile
/organization/1-4-all,Entertainment
/organization/1-4-all,Games
/organization/1-4-all,Software
/organization/1-800-publicrelations-inc-,Internet
/organization/1-800-publicrelations-inc-,Marketing
/organization/1-800-publicrelations-inc-,Media
/organization/1-800-publicrelations-inc-,Public Relations
/organization/1-mainstream,Apps
/organization/1-mainstream,Cable
/organization/1-mainstream,Distribution
/organization/1-mainstream,Software
...

How to achieve it?

Excerpt of the original CSV:

permalink,category_list,...
/organization/-qounter,Application Platforms|Real Time|Social Network Media,...
/organization/-the-one-of-them-inc-,Apps|Games|Mobile,...
/organization/1-4-all,Entertainment|Games|Software,...
/organization/1-800-publicrelations-inc-,Internet|Marketing|Media|Public Relations,...
/organization/1-mainstream,Apps|Cable|Distribution|Software,...
...


texts:
I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict).
Maybe something is not working with the code or I missing something?

Train data is 20.000 rows (more or less), test data is 2000 rows. Each row is a match where 0 is lost and 1 is won.

y_train_binary = keras.utils.to_categorical(Y, 2)
y_test_binary = keras.utils.to_categorical(t_Y, 2)
model = Sequential()
model.add(Dense(40, input_dim=45, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(30, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(20, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(10, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(2, activation='softmax'))
keras.optimizers.Adam(lr=0.5, beta_1=0.9, beta_2=0.999, epsilon=0.3)
model.compile(loss='categorical_crossentropy', optimizer='Adam', metrics=['accuracy'])
model.fit(X, y_train_binary, epochs=300, validation_split=0.2, batch_size=10, verbose=0)
prediction_classes = model.predict_proba(t_X)
numpy.savetxt("C:/Users/Megaport/Desktop/foo.csv", prediction_classes, delimiter=",")

I've tried so many parameters (learning rate, epochs, batch size, epsilon, add layer, less layer, different value of dropout) but the problem is the same: probs are not working.

the probabilities of the class are 1 or 0 and not a value between 1 and 0. This happen after 100 epoches.

data test with prediction after 100 epoches is like this:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           1          0
   0           7          4          5           0          1
   1           6          7          6           0          1
   1           2          3          4           0          1

What I'm looking for:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           0.65          0.35
   0           7          4          5           0.25          0.75
   1           6          7          6           0.20          0.80
   1           2          3          4           0.30          0.70

Plus, someone could please give me some advice looking at accuracy and loss graph?

https://i.stack.imgur.com/vTnz4.png
https://i.stack.imgur.com/Zz5cv.png


texts:
In data science you sometimes encounter a scenario where you have meta data on a given process and the process data itself.
For example you have a mechanical component that is tested over time. So you have a (multi value-)timeseries (e.g. air pressure, temperature, mechanical movement, electric resistance and so on) on the one hand and the product metadata (age, CAD design parameters, used materials and so on) on the other hand.
In this scenario you can't really just use timeseries methods since you're ignoring the metadata, but just focusing on the metadata is also wrong, thus I'm looking for ways to use both types of data in my ML-system to better predict an anomaly or the likelihood of the test failing.
Do you know any models, tutorials or papers for this kind of data?
What are some good starting points to get experience with this kind of data?


texts: Does MLPClassifier (sklearn) support different activations for different layers?

texts:
According to the documentation, it says the 'activation' argument specifies:

  "Activation function for the hidden layer"

Does that mean that you cannot use a different activation function in different layers?


texts: In batch normalization, shouldn't using DropConnect harm test accuracy?

texts:
In my understanding of batch normalization, mean and variance are calculated over the entire batch and then added to the population average. This average is then applied to the test set to estimate the mean and variance of the overall test set.
However, DropConnect removes connections during training as a form of network regularization. The model calculates batch statistics through a network that is missing usually half its connections. At test time, all connections are used. Shouldn't this affect the mean and variance of the population and/or the test set, throwing the estimated mean and variance off, or will the network stabilize itself over time?


texts: Cross-validation of a cross-validated stacking ensemble?

texts:
let me begin by saying that I understand how to build a stacked ensemble by using cross-validation to generate out-of-fold predictions for the base learners to generate meta-features. My question is about the methodology when cross-validating the entire stacked ensemble to check generalization error.

To eliminate any confusion, I'm going to call the cross-validation to generate out of fold predictions for the base learner CV A, while I'll call the cross-validation of the entire stacking ensemble CV B.

When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process? Or do I have to keep doing CV A and generate new out of fold predictions during each fold of CV B?

Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of. The main reason I'm asking this is because doing the second method would surely remove any data leakage but there would be an order of magnitude of additional computational complexity involved.


texts: Does high regression coefficient for Principal components that don't explain much variance imply that my data is not a good predictor?

texts:
There isn't much to add to the question.
Essentially i had some data that I reduced to 4 principal components, the first two components of which explain 99% of the variance in my data.
Upon building a linear regression model using these 4 components, the coefficients for component 3 and 4 is significantly higher (by an order of 20) in magnitude than the components 1 and 2.

Without having to look at accuracy of the model, is this information sufficient to conclude that my data is a very weak linear predictor of the dependent variable?


texts: Clustering with geolocation (lat/long pairs) attributes

texts:
I am trying to cluster customer behavior based on where they shop given by lat/long pairs. I also have other numeric attributes such as volume, average amount spent, etc. I am considering using HDBSCAN to create clusters. However, I'm not sure whether to feed the dataframe directly to the clustering algorithm or whether I would need to scale/normalize the data.

Is it wise to scale the geolocation pairs? Or would important location information be lost?

Any help would be much appreciated.

https://stats.stackexchange.com/questions/89809/is-it-important-to-scale-data-before-clustering

This page explains a lot. However, in the answer by @Anony-Mousse, he mentions not to scale lat/long pairs. That's good but what about other continuous variables?


texts:
I found a good model for a time series forecasting problem I have but it doesn't allow for covariates, which I need to include a few of (date-based events).
The approach I came up with to deal with this was to run a regular linear regression on the events and the dependent variable and then feed the residual training errors from the regression into a time series forecasting model (Theta to be exact). The time series model would then project the "predicted" error and I would simply adjust my linear regression forecast using that forecast.
This seems to work in theory and somewhat in practice, but I'm having issues transforming the residuals to conform to the requirements of the Theta model implementation (it requires non-negative numbers). My approach was simply to scale the residuals (min/max), forecast them, then inverse scale them and add to the regression predictions based only the on event features. The issue seems to be the scale I pick actually effects the forecast results for some reason (see code below). Any reason why this is the case?
trian = nc[:wl]
test = nc[wl:]
wl = wl + stride

lr = LinearRegression()
model = lr.fit(trian[['business_builder_bonus']], trian['new_clients'])

preds = model.predict(trian[['business_builder_bonus']])
actuals = trian['new_clients']

residuals = actuals - preds

#Residuals
mms = MinMaxScaler(feature_range=(1, 2))
r = mms.fit_transform(residuals.values.reshape(-1, 1))
r = pd.Series(r.flatten())

tf = ThetaForecaster(sp = 12)
tf.fit(r)
t = tf.predict(1)

pred = model.predict(test[['business_builder_bonus']])[0]
adj = mms.inverse_transform(t.values.reshape(-1, 1))[0]
adj_pred = pred + adj
print(adj_pred)


texts: How do I generate a laplacian matrix for a graph dataset?

texts:
If I have a dataset in a csv that looks like the one shown below.

How do I convert this into a laplacian matrix using Python?


texts: How to Present All Categories in All Samples

texts: I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with different column counts. Because not all categories in columns preserved in samples and different samples includes different subset of categories for each column. Is there a way to ensure all categorical columns in all samples contains all possible categories?


texts: What approach should I take to model forecasting problem in machine learning?

texts:
I have a dataset which contains 4000k rows and 6 columns. The goal is to predict travel time demand of a taxi. I have read many articles regarding how to approach the problem. So, every writer tell his own way. The thing which I have concluded from all my readings is that I have to use multiple algorithms and check the accuracy of each one. Then I can ensemble them by averaging or any other approach.

Which algorithms will be best for my problem accuracy-wise? Some links to code will be helpful for me.

I currently only have training set of data. After I work on it, it will be evaluated on any testing set by my professor. So, what should I do now? Either split data I have into my own testing and training set or separately generate dummy data as a testing set?


texts: Multiple output for multi step ahead prediction using LSTM with keras

texts:
I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training an LSTM feeding 3 steps back data points to predict the next 3 steps in the future.
The input data looks like:

X = [[[A0, B0, C0],[A1, B1, C1],[A2, B2, C2]],[[ ...]]]

with dimensions: (1000, 3, 3). The output is:

y = [[C3, C4, C5],[C4, C5, C6],...]

with dimensions: (1000, 3).

I am using a simple LSTM with 1 hidden layer (50 neurons).
I set up an LSTM with keras as:

n_features = 3
neurons = 50
ahead = 3
model = Sequential()
model.add(LSTM(input_dim=n_features, output_dim=neurons))
model.add(Dropout(.2))
model.add(Dense(input_dim=neurons, output_dim=ahead))
model.add(Activation('linear'))
model.compile(loss='mae', optimizer='adam')
model.fit(X, y, epochs=50)

This model works fine. Now, I'd like to predict the values of B as well (using the same input). So I tried to reshape the output in a similar way as I did for the training that has multiple features:

 y = [[[B3, C3],[B4, C4],[B5, C5]],[[ ...]]]

so that it has dimensions: (1000, 3, 2). However, this gives me an error:

Error when checking target: expected activation_5 to have 2 dimensions,
but got array with shape (1000, 3, 2)

I guess the structure of the network needs to change. I tried to modify model.add(Dense(input_dim=neurons, output_dim=ahead)) with no success. Should I reshape the y differently? Is the structure of the network wrong?


texts: Feature selection where adding features are deteriorating model

texts:
k

I am training a kNN classifier with 144 features and graphed the accuracy vs number of features used and got this. What might be the reason for the drops in the accuracy at some points of the graph? I am using accelerometer-gyroscope-magnetometer fusion to recognize human activities.

The one presented is validation accuracy. Should I use training accuracy instead? And why?

I ranked the features using ReliefF feature selection algorithm.

I used time domain features such as mean, standard deviation, rms, median, variance, iqr, mad, zcr and mcr, and frequency domain features such as skewness, kurtosis and pca

Here are the top 8 features chosen. Peak accuracy occurs at 8 features.


texts: Using pandas, check a column for matching text and update new column if TRUE

texts:
My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE.

From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, are checked for a matching text pattern - 'PEA'. Based on whether pattern matches, a new column on the data frame is created with YES or NO.

I have the following data in file DATA2.csv

ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY
GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness)
RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure
RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure
RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access
RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales

And using this code....

import pandas as pd

df=pd.read_csv('DATA2.csv')

Search_for_These_values = ['PEA', 'DEF', 'XYZ'] #creating list

pattern = '|'.join(Search_for_These_values)     # joining list for comparision

IScritical=df['COLUMN_to_Check'].str.contains(pattern)
for CHECK in IScritical:
    if not CHECK:
        print CHECK
        df['NEWcolumn']='NO'
    else:
        print CHECK
        df['NEWcolumn']='YES'

df.to_csv('OUPUT.csv')

Printing the value of 'CHECK' returns correct values, i.e., first row returns false.

C:\Users\ME\Documents\SandBox (master)
λ python numpytest_pub.py
False
True
True
True
True

But the output csv file shows all values of 'NEWColumn' as 'YES', where on 'NEWcolumn', row[0], value should be 'NO' as the 'COLUMN_to_Check' here should not match the pattern.

,ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY,NEWcolumn
0,GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness),YES
1,RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure,YES
2,RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure,YES
3,RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access,YES
4,RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales,YES

I can sense that something is missing in the CHECK part, but not able to figure out what. Can anyone help ?

Let me know if the question needs rephrasing for better understanding or future community use.


texts:
I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me.
so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case).  I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC.
so what do you suggest?


texts: Career advice for a data scientist - Am i in the right place?

texts:
I need a career advice from you.

I'm 26 and working as a data scientist for less than 2 years. I have a statistics and economics background mainly. My tech skills can be summed up by a good knowledge of SQL,Python and R.

My first experience was a 6 month internship for a software house company that aims at building product for banks and financial institution. Here I worked with other 2 unexperienced data scientist, all with a statistics background like me.

I left and currently I'm working in a big company, one of the major wholesaler in Europe.

The problem I'm struggling with , it's that I'm the only data scientist in the company and in a digital marketing dept, so with non tech people.

After almost a year, I'm quite tired of working alone and i don't feel I have been growing much professionally.

A thought I had a while ago, was to ask a temporary shift to Data Wharehouse department in order to acquire more Database related knowledge.
Another idea was obviously to ask for another data scientist.

My doubt can be summed up like this:

Regarding  first idea: As a data scientist, what DB knowledge should i master? I'm quite good with SQL, mainly to perform data extraction and also create some simple table structure. It could be benificial to know more on this topic, for example data migration, ETL procedure, etc etc? I'm planning to learn more about Hadoop/Spark/Hive/Cassandra, but we don't have this tech stack at work. Just an oracle db.
For budget reason, I'm aware that if they'r gonna hire a new data scientist it couldn't be someone with much of experience. In this case it's very unclear to me if I could benefit from working with another unexperienced ds or i should prefer the first option.

Thanks for any opinion


texts:
I am trying to fit my data into my model which takes numpy as input, so I feed the model with the dataframe values

stacked_averaged_models.fit(train.values, y_train1)

I am getting the following error

ValueError                                Traceback (most recent call last)
<ipython-input-145-9ba69af8df05> in <module>()
      1 X_traintrain = train.as_matrix().astype(np.float)
      2 from sklearn.metrics import r2_score
----> 3 stacked_averaged_models.fit(train.values, y_train1)
      4 stacked_train_pred = stacked_averaged_models.predict(train.values)
      5 stacked_pred = np.expm1(stacked_averaged_models.predict(test.values))

<ipython-input-140-dfca4af6e9d1> in fit(self, X, y)
     18                 instance = clone(model)
     19                 self.base_models_[i].append(instance)
---> 20                 instance.fit(X[train_index], y[train_index])
     21                 y_pred = instance.predict(X[holdout_index])
     22                 out_of_fold_predictions[holdout_index, i] = y_pred

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params)
    248         Xt, fit_params = self._fit(X, y, **fit_params)
    249         if self._final_estimator is not None:
--> 250             self._final_estimator.fit(Xt, y, **fit_params)
    251         return self
    252

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\linear_model\coordinate_descent.py in fit(self, X, y, check_input)
    705                              order='F', dtype=[np.float64, np.float32],
    706                              copy=self.copy_X and self.fit_intercept,
--> 707                              multi_output=True, y_numeric=True)
    708             y = check_array(y, order='F', copy=False, dtype=X.dtype.type,
    709                             ensure_2d=False)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    574     if multi_output:
    575         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,
--> 576                         dtype=None)
    577     else:
    578         y = column_or_1d(y, warn=True)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    451                              % (array.ndim, estimator_name))
    452         if force_all_finite:
--> 453             _assert_all_finite(array)
    454
    455     shape_repr = _shape_repr(array.shape)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in _assert_all_finite(X)
     42             and not np.isfinite(X).all()):
     43         raise ValueError("Input contains NaN, infinity"
---> 44                          " or a value too large for %r." % X.dtype)
     45
     46

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

I did a check on NaN and infinity, it did pass the test

X_traintrain = train.as_matrix().astype(np.float)
print(np.any(np.isnan(X_traintrain)))
print(np.all(np.isfinite(X_traintrain)))

Output:

False
True

How else can I solve, or at least, debug this?

X1      X2       X3     X4       X5    X6   X7     X8   Y1      Y2
0.64    784.00  343.00  220.50  3.50    5   0.00    0   10.56   16.67
0.62    808.50  367.50  220.50  3.50    2   0.00    0   8.60    12.07
0.62    808.50  367.50  220.50  3.50    5   0.00    0   8.50    12.04
0.98    514.50  294.00  110.25  7.00    2   0.10    1   24.58   26.47

This is few rows of my dataset


texts: In random forest, what happens if I add features that are correlated?

texts:
I'm training a random forest, trying to predict market shares of future stores on geographical areas. I have many features for these areas, some of which tell similar but different things about one thing.

For example, I know the total number of $accommodations$ in the area, and I also have 5 others columns which are all linked in the following way:

$main \space accommodations + secondary \space accommodations + holiday \space accommodations = houses + flats = accommodations$

I have the feeling that including them all in my model would be wrong... but including them might be important... Any hint on how I should handle this?

Would it be a good idea to include $accommodations$ as absolute value and include all the other five but as % (of $accommodations$) and not as absolute values?

In a similar fashion, I also have the total number of $households$ of the area, the $total \space income$ of the area, and the $average \space income$ of households in the area (so that $households * average \space income = total \space income$). I have the feeling using the average and not the total income would be a better idea, but how can I be sure I'm right ?

(I guess I could train three random forests using the average income only, the total income only, and both, and see how they perform on cross validation, but is there a rule of thumb that I should know of which can make me go faster ?)

(In case it's relevant, I'm using R and the randomForest package)


texts: In K-means, what happens if a centroid is never the closest to any point?

texts:
I am implementing K-Means from scratch and that exercise raised a question.

To update my centroids, for each centroid, I have to find the points for which that centroid is the closest.

In some cases, especially when the number of centroids is high and the number of instances is low (i.e. k=20 and 100 instances), I find centroids for which no point has them as their closest centroid. In other words, they become "orphans" as no instances are allocated to them.

In the example above, the two centroids (X) at the top clearly both have points for which they are the closest centroid. But the centroid at the bottom is never the closest centroid for any instance.

How do I deal with this?

Should the lone centroid remain unmoved?
Should I move that centroid? If yes, how?
Should I remove it?

Is there a standard way to deal with this?

------------ EDIT: --------------

On Github, I found this:

https://github.com/klebenowm/cs540/blob/master/hw1/KMeans.java

This suggests that if a centroid becomes an "orphan", it should be assigned to the point that is the furthest from its centroid.

This seems like a sound method, is there any paper or theory supporting this?


texts: AUC ROC in keras is different when using tensorflow or scikit functions.

texts:
Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

and

def auc(y_true, y_pred):
    return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)

Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.

Shouldn't I get about the same results using both functions?


texts: Predicting binary target value based on unlabelled features

texts:
I have a dataset of around 15 stocks having the following data format :

TimeStamp | F1 | F2 | F3 | ...Fn | Y

TimeStamp: Date and Time (masked)

Fi : A random feature(numeric) which contains some information about the target Variable

Y: A binary 0/1 target variable to predict.

The time column is meta data, not a feature. All other columns are features.

The dataset is sufficiently large. I need to make a predictive model for predicting binary values using the random unlabelled numeric features( 71 total features) . Can you help me how to proceed with this problem ?


texts: Predictive model to maximize sum of dependent variable?

texts:
I am trying to classify cars for a towing company. Junky  cars earn more when  sent to the junkyard, and the more valuable cars should earn more at the auction, despite the auction fee.  Creating a logistic regression that takes into account Make, Model, Mileage,  Year and Run status helps us improve the accuracy of which cars should go where, but a difficulty arises: Sometimes, a car that would be classified as junk can actually be an outlier, and sell for a lot of money. So to optimize our model, we don't really care that much whether we are right or wrong on an individual car, so much as we maximize our bottom line.
All of the models I have seen (Logistic regression, RF, linear regression) make predictions on a line by line basis.
What would be a good model to try and maximize the aggregate sum of the predictions?
Below is a reprex of my data, as well as basic code I used.
What I actually tried until now is to look at past data, and classify, in hindsight, what should have been done, based on prices that were earned in the auction vs available junk prices.  I then ran a glm against that classification to predict the future. As mentioned above, my code improved the accuracy of our decisions, and would have sent more cars to junk correctly, but  some  that we classified as junk sold for so much in the auction that  it wasn't worth sending any to junk.
What is the proper way to approach this?
cars <- structure(list(YearOfCar = c(2009L, 2009L, 2003L, 2004L),
    Make = c("Hyundai", "Lexus", "Ford", "Toyota"), Model =
    c("Sonata", "GS 350", "F-250 Super Duty","Camry"), PickUpState =
    c("MN", "LA", "MA", "NJ"), Auction_Result = c(650,625,425, 1500),
    Auction_Fee = c(144.25, 373.54, 213.5, 187), Mileage = c(116120L,
    198900L, 140241L, 312927L), Runs = structure(c(1L, 1L, 1L, 2L),
    .Label = c("No", "Yes"), class = "factor"),   junkyard_Offer =
    c(230L, 235L, 140L, 300L), Date = structure(c(17592,  17707,
    17674, 17583), class = "Date")), row.names = 3:6, class =
    "data.frame")
cars$hindsight <- ifelse(cars$Auction_Result-
    cars$Auction_Fee>cars$junkyard_Offer,1,0)
glmodel <- glm(hindsight~Make+Model+Mileage+Runs, data = cars,
    family="binomial")
prediction <- predict(glmodel, cars, type = 'response')
prediction_classifier <-  if_else(prediction>.501,1,0)
cars$prediction_results <- ifelse(prediction_classifier==1,
    cars$Auction_Result-cars$Auction_Fee,cars$junkyard_Offer)


texts:
In the context of face recognition I have the following histogram:

blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the distances for cross matches (different persons).

The distance is the value returned by the neural networks as the result of a comparison between two faces, how much two "faces vectors" (embeddings) differ.

I'm looking for a function that, given a distance, can tell how likely it is for two pictures to be from the same person.

This function should be look like this red line (with a different y-scale):

So with distance 0.5 it is extremely likely, with distance 1.4 it is close to a 50% chance.

Is there such a function? How is it called?

It is similar to this question but also very different. In my case a very small z-value (distance) still means high confidence even if it is many standard deviations away from the mean. The same is true for a value like 1.0.

This question is very similar too but I'd like to extract the distance to probability mapping function from actual measurements, on a reference dataset, and not from the distance by itself. So this should not depend on the loss or distance definition but on the data distribution alone.


texts:
We currently manage a large volume of economic data in Excel in my organisation. All of the data is downloaded from different online databases into Excel spreadsheets (one for each data frequency including annual, monthly, quarterly) - and then one main spreadsheet organises everything and creates tables that we need regularly. By organise, I mean that many of the things we need are simply identities ($Z=X+Y$ where we would have only downloaded data on $X$ and $Y$)

My view is that this could be done much more efficiently in R - where we'd automate the updating of the data and then spit out the tables that we need. But I am not trained at all in data management.

Would you all recommend a better way of doing this, or are there pitfalls to using R that I am not considering.


texts:
I have 2000 samples with 50 sensors signal every millisecond during 600 milliseconds. This means a 2D record for each sample.

I want to predict a continuous value.

I'd like to try autokeras for this regression problem. Could someone point to a tutorial or some advise to start with ?


texts: Count number of cards stacked one over the other

texts: I have a stack of ATM cards and I want to count the number of cards available in the stack. How to proceed through it? I'm using Python 3.6.0 and opencv2. I'm attaching PNG file of the images. Kindly provide help in this direction.[1


texts: Multitask learning model with variable number of tasks

texts:
This is my basic multitask learning model, and it has 2 tasks. Since there are only 2 tasks, maybe I can duplicate the code for each task as self.tower1 and self.tower2.

class Multitask_Network(nn.Module):
    def __init__(self):
        super(Multitask_Network, self).__init__()

        in_ch = 20
        shared_layer_size = 10
        tower_hidden_size = 10
        output_size = 2

        self.sharedlayer = nn.Sequential(
            nn.Linear(in_ch, shared_layer_size),
            nn.ReLU(),
            nn.Dropout()
        )
        self.tower1 = nn.Sequential(
            nn.Linear(shared_layer_size, tower_hidden_size),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(tower_hidden_size, output_size)
        )
        self.tower2 = nn.Sequential(
            nn.Linear(shared_layer_size, tower_hidden_size),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(tower_hidden_size, output_size)
        )

    def forward(self, x):
        x = self.sharedlayer(x)
        out1 = self.tower1(x)
        out2 = self.tower2(x)
        return out1, out2

model = Multitask_Network()
x = torch.ones((4, 20))
y = model(x)
y

But for my use case, I need to define the model's number of tasks during run time. I can do it with for-loop, but I think it is inefficient to run on GPU. I think it does the forward pass for task #1, then followed by task #2, to task #n, one after another.

How can I achieve this without a for loop in the forward pass?

class Multitask_Network(nn.Module):
    def __init__(self, num_tasks):
        super(Multitask_Network, self).__init__()

        in_ch = 20
        shared_layer_size = 10
        tower_hidden_size = 5
        output_size = 2
        self.num_tasks = num_tasks

        self.sharedlayer = nn.Sequential(
            nn.Linear(in_ch, shared_layer_size),
            nn.ReLU(),
            nn.Dropout()
        )

        self.conv_towers = []
        for i in range(num_tasks):

            tower = nn.Sequential(
                nn.Linear(shared_layer_size, tower_hidden_size),
                nn.ReLU(),
                nn.Dropout(),
                nn.Linear(tower_hidden_size, output_size)
            )

            self.conv_towers.append(
                    tower
                )
        self.conv_towers = nn.ModuleList(self.conv_towers)

    def forward(self, x):
        x = self.sharedlayer(x)

        output = []
        for i in range(self.num_tasks):
            output.append(self.conv_towers[i](x))

        return output

model = Multitask_Network(num_tasks=2)
x = torch.ones((4, 20))
y = model(x)
y


texts: High Level Understanding of Neural Machine Translation

texts:
I am currently studying Deep Learning based Machine Translation systems but I'm not sure in my understanding the logic of the process. I understand that the source and destination language translation sentence pairs must be represented as word2vec vectors, but why is it necessary to apply two (encoder-decoder) Recurrent Neural Networks? My first idea would be applying only one RNN, where the input is the source language examples (in the form of word2vec vectors) and the output is simply a word2vec sequence of the destination language. Why is it necessary to use another RNN?

My additinal question is if this system is flexible enough to cope with synonimes, word order variations and other disambiguities? Is it capable of approximate the correct meaning of a new to-be-translated source language sentence?

And last but not least: how could one evaluate such a model where many translations can be correct at the same time?


texts: Mathematical algorithms to generate labels for training data

texts:
What are the possible approaches when we need to train a model, but the training dataset is really small? (Assuming we have a lot of data, just not many data are labeled)

I know a library from Stanford: https://hazyresearch.github.io/snorkel/ that can generate training labels based on a some pre-determined experts rules. (Side question, anyone happen to know what's the underlining mathematics/statistics of this library?)

However, I am wondering in the scenarios when the snorkel packages can not be used, what are the approaches to label more data for training? Could Maximum Likelihood Estimator be used here? How would one implement such algorithm for labelling the data for training?

BTW, I am looking for a mathematical approach, not a brute-force one like using Amazon Mechanical Turk.

Thank you!


texts: I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with some real examples.


texts: How to use binary relevance for multi-label text classification?

texts:
I'm trying to use binary relevance for multi-label text classification.
Here is the data I have:

a training set with 6000 short texts (around 500-800 words each) and
some labels attached to them (around 4-6 for each text). There are
almost 500 different labels in the entire set.
a test set with 6000 shorter texts (around 100-200 words each).

The difference of size between my two sets exists because of the source is different.

So, I want to use binary relevance to find the labels of the texts in the test set. To do it, I created a dictionary with all the different words in the entire training set and removed stop words, words who appear only once and words who appear in more that 10% of the texts. I got 14714 different words in my dictionary.
My idea was to create a matrix where each row represents a document and each column a words and each value was the number of occurrence of a word in a document. But with 14714 words and 6000 documents, I will get a matrix of 88 millions of integers! I tried, just to see, to create it and my laptop didn't support it. :)
I even didn't have the time to create my Y matrix and generate a model (I wanted to use a logistic regression) for only one label...

So, my questions are:

Was it a good way to make multi-label classification or is there a better method?
Is it a problem to have a training from one source and to use it to make a model to predict data from another source? Is the different size of the documents a problem?
Did you use logistic regression for this kind of problem?

Thank you!

Edit: I also want to add the most frequent words in my dictionary (after the cleaning part) are common words and totally useless in the field of my research (biology): used, much, two, use, possible, example, ... How can I pass through it?


texts: RNNs for time series prediction - what configurations would make sense

texts:
My question here is mostly about general-intuition logic: when using a RNN (LSTM) for predicting a time series, and you have the goal of, for example, predicting at 100 steps ahead a series of one single feature, what kinds of configurations would make sense for the layer sizes of a simple input | hidden | output RNN, and of the window size (assuming you want to look at more than one point at one so you pick a "window" / "interval"?

More precisely:

use only 1 input neuron and look at one point at a time, or pick a window/interval? - obviously that by looking only at 1 point, you lean entirely on the memory aspect of the network for detecting useful patterns... would this be a good thing?
when using a window, does it make sense to use a set of overlapping-windows, eg. to slide a window of 10 points one point at a time, or non-overlapping windows?
should there number of units in the hidden layer roughly equal the number of steps you aim to predict ahead? (also, similar thing with the relationship between window size and steps to predict ahead)

Or, if answering this is too hard  or time consuming, where could one get the intuitions for answering such questions?

(Because obviously the space of possible configurations is huge, and unless you have a ton of time or resource, you need some intuitions so you can start form a configuration that "roughly works"...)


texts:
Below are the results of two different linear regressions.  The first only has an N of 10 while the second has an N of 43.  The first has a very high association between the dependent variable and the independent variables overall (Adj R2 = 0.948, P = 0.003).  The second one only has one variable in the model with a significant association with the dependent variable and a lower overall association (Adj R2 = 0.436, P < 0.001).

I am trying to understand some of the reasons for these two sets of results.  The first seems spurious due to the small N.  I am still having trouble understanding how the second could have an adjusted R2 as high as it does considering only one variable has a significant association with the dependent.  What kinds of things might be going on here?


texts: TF-IDF: How to handle terms not part of the corpus

texts:
I'm working on a ML.Net based feature to extract keywords from a document corpus using TD-IDF.

Given this test corpus (one document per line):

This is an example to compute n-grams.
N-gram is a sequence of 'N' consecutive words/tokens.
ML.NET's ProduceNgrams API produces vector of n-grams.
Each position in the vector corresponds to a particular n-gram.
The value at each position corresponds to,
the number of times n-gram occured in the data (Tf), or
the inverse of the number of documents that contain the n-gram (Idf),
or compute both and multiply together (Tf-Idf).

And this test document:

foobar This is an example to compute n-grams

I'm getting the following result:

example             = 2,0794
example|compute     = 2,0794
compute             = 1,3863
compute|n           = 0,0000
n                   = 0,0000
n|grams             = 0,0000
grams               = 0,0000
n|gram              = 0,0000
gram                = 0,0000
gram|sequence       = 0,0000

My question is about the term foobar which is not part of the training corpus, and therefore not part of the result set. Is there a recommended way of dealing with out-of-vocabulary terms when using TF-IDF?


texts: How to know for sure if we can learn from a given data or not?

texts: I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?


texts: Combine multiple classifiers to build a multi-modal classifier

texts:
Suppose I am interested in classifying a set of instances composed by different content types, e.g.:

a piece of text
an image

as relevant or non-relevant for a specific class C.

In my classification process I perform the following steps:

Given a sample, I subdivide it in text and image
A first SVM binary classifier (SVM-text), trained only on text, classifies the text as relevant/non-relevant for the class C
A second SVM binary classifier (SVM-image), trained only on images, classifies the image as relevant/non-relevant for the class C

Both SVM-text and SVM-image produce an estimate of the probability of the analyzed content (text or image) of being relevant for the class C. Given this, I am able to state whether the text is relevant for C and the image is relevant for C.

However, these estimates are valid for segments of the original sample (either the text or the image), while it is not clear how to obtain a general opinion on the whole original sample (text+image). How can I combine conveniently the opinions of the two classifiers, so as to obtain a classification for the whole original sample?


texts:
I am doing a study on unsupervised data with various categorical variables. So I have found the FactoMineR package to be really handy for this, particularly with the FAMD functions. I can get to a base FAMD object just fine (and the corresponding charts) but I can't seem to get the predict.FAMD side working. Are there any working examples out there? Here's what I've done:

I have a mix of continuous and categorical data
I have split the data into a test and training set
I have created a FAMD object with my training set
I try to run predict.FAMD against the test set data but I keep on getting an an error of Error in [.data.frame(newdata, , i) : undefined columns selected

Generally, my code looks like:

library(FactoMineR)
res = FAMD(myTrainingData)
res.predict = predict(object=res,newdata=myTestData)

I've also tried variations on explicit columns and even no rows, like:

res.predict = predict(object=res,newdata=myTestData[,1:7])
res.predict = predict(object=res,newdata=myTestData[,c(1,2,3,4,5,6,7)])
res.predict = predict(object=res,newdata=myTestData[0,1:7])

All of these result in the same error. Help?


texts:
I use pandas to do feature extraction for machine learning.
I hope to achieve the following: Consider I have five data processing steps done sequentially, and I execute it once, the results will be saved automatically. Next time, if I change forth step, the library will automatically start from third step. Would this cache function be supported in Pandas or sklearn.pipeline.Pipeline or other data processing libraries naturally without our need to save them explicitly?


texts: std::bad_alloc with densenet and hyperas

texts:
I am filing this issue after being stagnated here for couple of weeks. I am using hyperas to find the hyperparameters for my network, Densenet. My issue here is that my evaluation always fails with bad_alloc issue after few evaluations, After much googling this seems to be a memory related issue. I have tried the following things

decrease batchsize to 2 from 64 still fails
use of K.clear_session() at the start of my create_model or at the end, still bad_alloc,
I noticed the code fails after one evaluation is done and time to evaluate the result. So I decreased the evaluation batchsize to 4 from default 32, still fails

So anyone here have any idea what is the issue here and how can it be fixed?

my code as follows(using hyperas on Densenet):

def create_model(X_train,y_train,X_val,y_val,X_test,y_test):
    K.clear_session()
    epochs = {{choice([5,7,10])}}
    es_patience = 3
    lr_patience = 2
    dropout = {{uniform(0.1,0.5)}}
    depth = {{choice([28,31,34,25])}}
    nb_dense_block = {{choice([3,4])}}
    nb_filter = 16
    growth_rate = {{choice([12,18,24,30])}}
    bn = True
    reduction_ = 0.5
    bs = 32
    lr = {{choice([2E-4,1E-4,5E-4])}}
    weight_file = 'keras_dn_wt_16Oct2200.h5'
    nb_classes = 1
    img_dim = (2,96,96)
    n_channels = 2
    print("------------------------ current config for the test -------------------------")
    print("Depth: ",depth," Growth_rate: ",growth_rate," lr: ",lr," nb_filter: ",nb_filter," dropout: ",dropout)
    print("Epochs ",epochs," batch_size: ",bs," es_patience: ",es_patience," lr_patience: ",lr_patience)
    print("dense_block ",nb_dense_block," reduction_: ",reduction_," bottleneck: ",bn)
    print("------------------------   end of configs        -------------------------")

    model  = DenseNet(depth=depth, nb_dense_block=nb_dense_block,
                 growth_rate=growth_rate, nb_filter=nb_filter,
                 dropout_rate=dropout,activation='sigmoid',
                 input_shape=img_dim,include_top=True,
                 bottleneck=bn,reduction=reduction_,
                 classes=nb_classes,pooling='avg',
                 weights=None)](url)
    model.summary()
    opt = Adam(lr=lr)
    model.compile(loss=binary_crossentropy, optimizer=opt, metrics=['accuracy'])

    es = EarlyStopping(monitor='val_loss', patience=es_patience,verbose=1)
    checkpointer = ModelCheckpoint(filepath=weight_file,verbose=1, save_best_only=True)

    lr_reducer = ReduceLROnPlateau(monitor='val_loss', factor=np.sqrt(0.1), cooldown=0, patience=lr_patience, min_lr=0.5e-6,verbose=1)

    model.fit(X_train,y_train,
          batch_size=bs,
          epochs=epochs,
          callbacks=[lr_reducer,es,checkpointer],
          validation_data=(X_val,y_val),
          verbose=2)

    score, acc = model.evaluate(X_test,y_test)
    print('current test accuracy:', acc)
    pred = model.predict(X_test)
    auc_score = roc_auc_score(y_test,pred)
    print("current test auc_score ------------------> ",auc_score)


texts:
The data below represent the fraction of active chlorine in a chemical product as a function of time after manufacturing

Consider the model $y=\theta_1 - \theta_2 e^{-\theta_3 x} + \varepsilon$, called the Mitcherlich model. Fit the Mitcherlich law to these data and discuss how you obtained the starting values.
Can anybody help me figure out how to do this? I am using SAS and for now this is what i wrote
proc import datafile='/folders/myfolders/xlsx datasets/chlorine.xlsx'
        dbms=xlsx
        out=chlorine
        replace;
        getnames=yes;

*scatter plot;
proc sgscatter data= chlorine;
    plot y*x / reg;
run;

*Fit the Mitcherlich law;
data chlorine;
    set chlorine;
    y2= log(0.4-y);
run;

proc reg data=chlorine;
    model y2=x;
run;
quit;

proc nlin data=chlorine method=Marquardt;
    parameters alpha=0.4 kappa=-0.06601 x0=45;
    Mitcherlich = alpha*(1-exp(-kappa*(x-x0)));
    model y = Mitcherlich;
run;


texts: How to identify topic transition in consecutive sentences using Python?

texts:
I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without transitional words) as follows:

As I really like Mickey Mouse, I was hopping to go to Florida. But my
dad took me to Nevada. Obviously, Mickey Mouse was not there. But, I
attended a camp with other children. And, I really enjoyed and learnt a lot from my
camp.

Here, I want to automatically split this into following sub-paraphs:

As I really like Mickey Mouse, I was hopping to go to Florida. But my
dad took me to Nevada. Obviously, Mickey Mouse was not there.

But, I attended a camp with other children. And, I really enjoyed and learnt a lot from my
camp.

As far as I know, this is not the sentence similarity measurement. What technique should be used here? Any example using python or tensorflow models would be greatly appreciated.


texts:
I'm studying Logistic Regression, there is a thing that I'm not able to understand.

I know what the Logistic Function is, I studied it. But I don't understand why the logistic function is equal to the dot product between $w$ and $x$.

$$ \log \left(\frac{p}{1-p}\right)=\bar{\omega }\cdot \bar{x} $$

For me these are two completely different concepts, I don't understand why are related in this manner. Can you help me?


texts: If a neural network is a universal function approximator, is the irreducible error 0?

texts:
How to marry the fact that (most) neural networks with a single hidden layer are universal function approximators with the fact that in the bias-variance decomposition we consider there to be an irreducible error?

Can the irreducible error be 0? (Then it would be inappropriate named.)


texts:
I am using sklearn multioutput RF regressor to learn statistics in my data. So my target contains several probabilities for the different features, and the sum of all these probabilities is one as they are fractions of how often the feature occurs.
The RF actually learns this property even though I have not enforced anywhere that the outputs should sum to one. I have also added a constant to my targets and the RF then learn that the outputs should sum to one plus that constant, so it is not some normalization.
I'm pretty sure I know how an RF regressor works but I cant explain how it can learn such metafeatures of my data. I would have expected the sum of my output to be somewhere around 1, not always exactly one.
Any ideas?


texts: Why do most of the research papers simply adopt mainstream CNN architectures designed for ImageNet dataset (AlexNet, VGG, ResNet, Inception etc.)?

texts:
Simply stacking multiple layers is not feasible. We need to optimize, and efficiently design the network, which has good learning capabilities (depending on the dataset domain).

So what are the factors to be considered while designing new CNN architectures for datasets from other domains?


texts: RandomForest and tree feature importance in scikit-learn

texts:
What is the difference between model.feature_importances_ and tree.feature_importances_ in the following code:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

# Boston Housing dataset
from sklearn.datasets import load_boston
boston = load_boston()
# Convert 'skleran.bunch' to Pandas dataframe
data = pd.DataFrame(boston.data, columns=boston.feature_names)

# Create train and test sets for cross-validation
X,y = data.iloc[:,:-1], data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 123)

model = RandomForestRegressor()
model.fit(X_train, y_train)

As I understand the following is importance of features:

importance = model.feature_importances_
importance_df = pd.DataFrame(importance, index=X_train.columns,
                      columns=["Importance"])

        Importance
-----------------
CRIM    0.025993
ZN      0.002781
INDUS   0.004832
CHAS    0.000315
NOX     0.028655
RM      0.406285
AGE     0.017987
DIS     0.040696
RAD     0.003615
TAX     0.009281
PTRATIO 0.009103
B       0.012354
LSTAT   0.438106

And what is tree.feature_importance?:

[tree.feature_importances_ for tree in model.estimators_]

How do they differ, calculated and which is more important 0.2 or 0.9? Can't find in the docs.


texts: Predict the accuracy of Linear Regression

texts:
How do I test if the predicted values in Linear Regression model are matching with the actuals?

I tried using - Confusion matrix, but I get this error -

#==============================================================================
# Create confusion matrix to evaluate performance of data
#==============================================================================
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix (dv_test, y_pred)

print(confusionMatrix)

ValueError: Can't handle mix of multiclass and continuous

When I execute the below code -

##Performing Linear Regression
from sklearn.linear_model import LinearRegression
from sklearn import model_selection
regressor=LinearRegression()
##Fit train
regressor.fit(iv_train,dv_train)
y_pred=regressor.predict(iv_test)
print('Accuracy of LR',mean_squared_error(y_pred,dv_test))

It results - Accuracy of LR 7837176694.18

Which is incorrect.

Below is my sample data set -

longitude   latitude    housing_median_age  total_rooms total_bedrooms  population  households  median_income   ocean_proximity median_house_value
-122.23 37.88   41  880 129 322 126 8.3252  NEAR BAY    452600
-122.22 37.86   21  7099    1106    2401    1138    8.3014  NEAR BAY    358500
-122.24 37.85   52  1467    190 496 177 7.2574  NEAR BAY    352100
-122.25 37.85   52  1274    235 558 219 5.6431  NEAR BAY    341300
-122.25 37.85   52  1627    280 565 259 3.8462  NEAR BAY    342200
-122.25 37.85   52  919 213 413 193 4.0368  NEAR BAY    269700
-122.25 37.84   52  2535    489 1094    514 3.6591  NEAR BAY    299200


texts: Calculate future GDP % using machine learning

texts:
I need to estimate the GDP % of a country three years into the future, based on historic data.

I have 30+ years of the following monthly data that includes features such as inflation and unemployment rate:

Year  Month  Inflation  Unemployment %  Other Features  GDP %
1990    1       1.1           6.2           ....         2.3
1990    2       1.3           6.1           ....         2.4

...
...
...

2019    6       0.8           4.8           ....         3.1
2019    7       0.9           4.9           ....         3.3

Using Random Forests (Python's scikit learn library), I can use the data to calculate the GDP of next month, 2019/8:

Once calculated (let's say GDP % = 3.2) I use this value and rerun the entire Random Forest process, including in the history data the value I obtained for 2019/8.

I continue iterating until I calculate 2022/8, and that is my result.

I have two issues:

It's a tedious process to recalculate everything on each iteration, if the historic data is large then this would take too long.
The 2022/8 result is obtained with just one path, I would need to rerun the overall process N times and take the average to have a more precise number.

I need this to work with machine learning (not necessarily Random Forests), any ideas how to improve/change this process?


texts:
There are two types of transfer learning model. One is
feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the pre-trained model can be changed.

According to those categorizations, static word vector like word2vec is a feature extraction model, where each vector encodes the meaning of the word.

The meaning of the word changes context. For example, "Bank of the river"  vs "Bank as a financial institute". These word2vec vectors do not differentiate between these meaning.

Current models like Bert consider the context. Bert is a language representation model. That means, it internally can represent word by contextual word vectors.

By default, Bert is a fine-tuning model. This is where my imagination about fine-tuning started to fall apart.
Let's say, on top of Bert model we created some task-specific layer. Now, if we fine tune, by definition, the weights of the lower level (language representation layer) will change at least a bit that means, the vector of the word will also change (if we compare before and after fine tune). That means the meaning of the word change a bit because of the new task.
If my above interpretation is correct, I can not comprehend this phenomenon for example, the word vectors of task sentiment analysis are different that the word vectors (of the same word) of task question answering. Can anybody help me?

Please correct me if anything above is wrong. Thanks


texts: Calculating correlation for categorical variables

texts:
I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical features. I want to find out features with most highest influence on the target variable. My objectives are:

Correlation between categorical and categorical variables. e.g. For a binary target (like Titanic dataset), I want to find out what is the influence of a category on the target (like, influence of gender on survival (0/1))
Capture some non linear dependencies. e.g. For supermarket sales data, the sales is usually higher during weekends as people might visit such store more during holidays. So we expect to see spikes at an interval of roughly 7 days. Is there any way to capture this non-linearity/seasonality by a correlation coefficient?


texts: How can I have an "undefined" category in multi-class classification

texts:
I'm trying to classify several websites by category (finance, health-care, IT, etc...). I have at my disposal the content of the pages of the websites, and I use the words to classify. For now, I have manually classified some website to train a naive bayes model.

As I'm mostly interested in a high precision (e.g a classified website must be in the right category), I would like to add an "undefined" category in which a website would end up if it is not close enough to the other categories. To be clear, it's not a problem for me if a website is not classified, it's a problem if it is misclassified.

Are there algorithms that would allow this, or a way to train an "undefined" category ?

My best guess for now is to train something like a random forest and define a "minimum score" below which a website is "undefined".


texts: Find most important inputs of LSTM-RNN for multivariate time series modeling

texts:
The question is similar to the one asked in SO, but I think it is more appropriate to be here. It remains anyway unanswered.

  Assuming multivariate time series, how to evaluate the importance of
  all inputs of an already trained LSTM-RNN?

Thus, which are the inputs that affect the output the most?

I am aware of Pearson correlation (for linear relations) and Information Gain based on Entropy reduction to explore input/output relations in general, but this is not what I need. How can you infer from an already trained LSTM network, which inputs affect the output the most? Can you tell by the weights?

Any suggestion or literature would be welcome.


texts: How to apply Machine learning model on time series to predict next time step

texts:
I have done feature engineering on a single variable time series data (spare parts usage), then I turn the time series data into supervised machine learning problem. I have trained and test on the transformed new dataset. My question is how to apply the model to predict one time step, multi time steps into the future? All features are calculated over prior data points, such as lag features, moving window statistics, such as max, min, mean, median over for example, last 5, 10, 15 days.

If you have working examples in Python or any white papers with example, that will be great. Look forward to your help
Shaun


texts: Converting dates to appropriate form to train machine learning model

texts: I am trying apply linear regression on a dataset where the independent variable is a date formatted like '2-jan-08'. How should I convert the date so it can be used for model fitting?


texts: Relationship between gradient norm and layer depth

texts:
I was wondering, if there was some relationship between the gradient norms of a layer in a neural network and its respective depth.
I would suspect the: The smaller the depth of the layer, the bigger their gradient. I can imagine that this is true in general, because the upper layers of a network contribute basically to anything below of them (including the error in the end). Therefore, going by their contribution I would assume that their gradients become small last. This would then also mean, that the values haven't yet converged and profit from continuous training.
But is this true? I wasn't sure, thus tried looking it up and found this: source
If one looks at the image shown at this site, one can clearly see that the exact opposite seems to be the case! The mean gradient norm increases with the depth! How is that possible?

I can imagine this is the case, if the number of neurons per layers also decreases. Thus, while the contribution of each neuron in a lower level is then split up to neurons of the upper layer. As the upper layers have more neurons the gradient thus become smaller. However, all layers in the source appear to be initialized by the same number of neurons (?)
I find this confusing and my understanding of neural nets is definitely not yet sufficient enough. Therefore I would be very happy, if someone could help me to understand this matter better.


texts:
I am looking for an alternative way of analysing my data which will be used for ML. I am using Matlab to implement my code. Using my code, I obtain arrays which are used to describe signals. Each array represents a new and different signal.

I have used the corrcoeff method in Matlab and it delivered what I needed. But, I need another alternative way to verify the correlation between all the signals. Is there a better way to do this which is easy to interpret?

Dumb question: is it better to compare the images of the signal generated by Matlab or to compare the arrays?

Forgive me if this sounds badly formulated but I hope it is clear.

Thank you in advanced!


texts: Sklearn Pipelines - How to carry over PCA?

texts:
I'm developing a pipeline to fit parameters for a gradient boosting classifier while also fitting the optimum number of features in a PCA model. This is the current setup:

pipe = Pipeline([
    ('reduce_dim', PCA()),
    ('classify', GradientBoostingClassifier())
])
score = {'f1': 'f1', 'accuracy': 'accuracy'}
N_FEATURES_OPTIONS = [12,13,14,15]
max_dep = [3,4,5,6]
n_est = [50,80,100, 120, 150]
min_samp = [4,5,6,10]
param_grid = [
    {
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__n_estimators': n_est,
        'classify__max_depth': max_dep,
        'classify__min_samples_split':min_samp
    }]
reducer_labels = ['PCA']

grid_adc = GridSearchCV(pipe, cv=5, n_jobs=-1, param_grid=param_grid, scoring=score, refit='accuracy')
grid_adc.fit(X_train, y_train)
grid_adc.best_params_

Which outputs:

{'classify__max_depth': 3,
 'classify__min_samples_split': 4,
 'classify__n_estimators': 50,
 'reduce_dim': PCA(copy=True, iterated_power='auto', n_components=12, random_state=None,
   svd_solver='auto', tol=0.0, whiten=False),
 'reduce_dim__n_components': 12}

Now I'd like to validate and score the model with cross validation. If I run the following:

cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1)

Does the PCA I fit from the pipeline carry over to the cross_val_score function? If so, does the cross_val_score function transform the data with the PCA every time it generates a new train/test split?

Or do I need to create a new PCA after the pipeline to fit into the cross_val_score function?


texts: How can I predict the acceptance of an article by publisher?

texts:
I am working on this project where I have to predict the acceptance probability of a literary article, let's say a research paper by a publisher, which in this case would be a Journal.

I would like to use a machine learning algorithm for this project which would learn from the pool of articles/research papers published by the Publisher/Journal till date. Later I want to use the current article/research paper at hand to predict the probability of acceptance of this article/research paper on the insights that the algorithm has learnt. Here are my queries:-

What algorithm are there to tackle such task?
How should I make these algorithms learn the word vector, grammars, topics etc?
How do I convert the article into a feature set from which the algorithms can learn?


texts: I am trying to understand the concept of  Confidence Intervals. What is the meaning of point estimates and confidence intervals? What I understood is the point estimate in confidence interval is basically the statistics of sampling distribution. Can we say that after using central limit theorem and finding population mean using CLT rather than giving point estimate we will give confidence interval?


texts: Stacking: How to best treat base learner?

texts:
With stacking, several (diverse) base learners are used to predict the dependent variable $\hat{y}_{b,m}=\beta_{b,m} X$ in a hold-out set, where $m$ are base learner models $1,...,n$. These predictions are used in a second step as explanatory variable(s) in a meta learner $y = \beta_1 X + \beta_2 \hat{y}_b + u$.
I wonder how to best treat $\hat{y}_{b,m}$ in practice. There are basically two options:

Use each base learner's prediction $\hat{y}_{b,m}$ as a separate feature (column) in the meta learner model.
Take the rowmean over the different base learner's predictions $ 1/n \sum \hat{y}_{b,m}$ and use it as a single feature (column) in the meta learner.

My intuition is that both approaches might work, dependent on the choice of the meta learner. E.g. when the meta learner uses shrinkage (e.g. Ridge), this may help to "shrink" some of the "not so useful" $\hat{y}_{b,m}$ when all of the base learners predictions are treated as single feature (OK, correlation might be an issue in linear models). A similar logic might apply to meta learners such as boosted trees (correlation not a big issue). Using each prediction as a single feature may also provide more information (variation in the data) which can be exploited by the meta learner.
Nevertheless, averaging seems to be used quite often (if I'm not mistaken) to create a single feature from different base learners's predictions. I can't really pin down what is be the best approach here.
Are there any insights - theory based or from practical experiance - which help to decide what the best approach is?


texts:
For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. "Italian, Japanese," etc.). Lets also assume our data already a cuisine type column that we can use for accuracy checking.

One way of approaching this problem would be a supervised Latent Dirichlet Allocation approach, where the restaurant type is the response. In this way, topics are trained to be then used in a multinomial logistic regression to guess the cuisine type.

Is the above approach far superior to instead running (unsupervised) LDA with plenty of topics, and then using something like XGBOOST to predict cuisine type? In other words, we run unsupervised LDA, and then project all reviews to vector distances to each topic, and then use these feature vectors to predict cuisine type?

I understand that sLDA will try to pick topics that better characterize each category type but is the former really superior to the latter approach? The reason I ask this is because I don't know of any fast sLDA implementations out there.


texts: Error: valueError: input arrays should have the same number of samples as target arrays. Find 1 input samples and 0 target samples

texts:
I'm trying to do task for system calls classification. The code below is inspired from a text classification project. My system calls are represented as sequences of integers between 1 and 340. The error I got is:

  valueError: input arrays should have the same number of samples as
  target arrays. Find 1 input samples and 0 target samples.

I don't know what to do as it's my first time.

   df = pd.read_csv("data.txt")
   df_test = pd.read_csv("validation.txt")
   #split arrays into train and test data (cross validation)
    train_text, test_text, train_y, test_y =
   train_test_split(df,df,test_size = 0.2)
   MAX_NB_WORDS = 5700

   # get the raw text data
   texts_train = train_text.astype(str)
   texts_test = test_text.astype(str)
   # finally, vectorize the text samples into a 2D integer tensor
   tokenizer = Tokenizer(nb_words=MAX_NB_WORDS, char_level=False)
   tokenizer.fit_on_texts(texts_train)
   sequences = tokenizer.texts_to_sequences(texts_train)
   sequences_test = tokenizer.texts_to_sequences(texts_test)

   word_index = tokenizer.word_index
   type(tokenizer.word_index), len(tokenizer.word_index)
   index_to_word = dict((i, w) for w, i in tokenizer.word_index.items())
    " ".join([index_to_word[i] for i in sequences[0]])
    seq_lens = [len(s) for s in sequences]

    MAX_SEQUENCE_LENGTH = 100
    # pad sequences with 0s
    x_train = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)
    x_test = pad_sequences(sequences_test, maxlen=MAX_SEQUENCE_LENGTH)
    #print('Shape of data train:', x_train.shape)  #cela a donnée (1,100)
    #print('Shape of data test tensor:', x_test.shape)
    y_train = train_y
    y_test = test_y
    print('Shape of label tensor:', y_train.shape)
    EMBEDDING_DIM = 32
    N_CLASSES = 2

    y_train = keras.utils.to_categorical( y_train , N_CLASSES )
    sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='float32')

    embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM,
                    input_length=MAX_SEQUENCE_LENGTH,
                    trainable=True)
    embedded_sequences = embedding_layer(sequence_input)

    average = GlobalAveragePooling1D()(embedded_sequences)
    predictions = Dense(N_CLASSES, activation='softmax')(average)

    model = Model(sequence_input, predictions)
    model.compile(loss='categorical_crossentropy',
      optimizer='adam', metrics=['acc'])
    model.fit(x_train, y_train, validation_split=0.1,
    nb_epoch=10, batch_size=1)
    output_test = model.predict(x_test)
    print("test auc:", roc_auc_score(y_test,output_test[:,1]))


texts: Implementation of Gaussian Mixture Model for clustering when dealing with multidimensional hyperspectral data in python

texts:
I have a python numpy array of size (800,800,4) which is my hyperspectral camera data.

When I try standard GMM methods from scikit-learn I get an error saying that the expected dimension of the data should be less than or equal to 2.

Current method:

gmm=GaussianMixture(covariance_type="full")

gmm=gmm.fit(data)

Are there any python packages or functions available to process higher dimensional data using gmm?