texts: Understanding min_samples_split and min_samples_leaf hyperparameters with DecisionTreeClassifier algorithm

texts:
My dataset consists of 775 samples and 117 features. The features represent developer skills (C++, Hadoop, AWS, etc.) and the output variable is a developer's profile (Frontend Developer, Quality Assurance Developer, etc.). I've used the following code to generate the graph linked below:
min_samples_splits = np.linspace(0.1, 1.0, 10, endpoint=True)
train_results = []
test_results = []
for min_samples_split in min_samples_splits:
    dt = DecisionTreeClassifier(min_samples_split=min_samples_split, random_state=0)
    dt.fit(x_train, y_train)
    train_pred = dt.predict(x_train)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_train, train_pred)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    train_results.append(roc_auc)
    y_pred = dt.predict(x_test)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_pred)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    test_results.append(roc_auc)

from matplotlib.legend_handler import HandlerLine2D

(line1,) = plt.plot(min_samples_splits, train_results, "b", label="Train AUC")
(line2,) = plt.plot(min_samples_splits, test_results, "r", label="Test AUC")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("AUC score")
plt.xlabel("min samples split")
plt.show()

The x-axis is reported as a percentage of samples, which would lead me to conclude that algorithm performance is optimized when 80% of samples are required to split an internal node? But that doesn't make sense as that would mean 620 (80% of 775) samples would be required to split an internal node right?
I've also generated a graph to explore the min_samples_leaf hyperparameter for the same dataset:
min_samples_leafs = np.linspace(0.1, 0.5, 5, endpoint=True)
train_results = []
test_results = []

for min_samples_leaf in min_samples_leafs:
    dt = DecisionTreeClassifier(min_samples_leaf=min_samples_leaf, random_state=0)
    dt.fit(x_train, y_train)
    train_pred = dt.predict(x_train)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_train, train_pred)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    train_results.append(roc_auc)
    y_pred = dt.predict(x_test)
    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_pred)
    roc_auc = auc(false_positive_rate, true_positive_rate)
    test_results.append(roc_auc)

from matplotlib.legend_handler import HandlerLine2D

(line1,) = plt.plot(min_samples_leafs, train_results, "b", label="Train AUC")
(line2,) = plt.plot(min_samples_leafs, test_results, "r", label="Test AUC")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("AUC score")
plt.xlabel("min samples leaf")
plt.show()

Is this graph saying that algorithm performance is optimized when 35% of samples (so 35% of 775 = 271) are required at each leaf node?
I ask because I've experimented with passing different parameters to the min_samples_split and min_samples_leaf hyperparameters and my interpretations of these graphs doesn't line up with the algorithm's performance (which seems to do well when 20 is passed to each parameters in question). What am I not understanding here?
Would really appreciate any insight! Thank you in advance!


texts: is SST=SSE+SSR only in the context of linear regression?

texts:
the problem of regression is to minimize the sum of squared errors, i.e. $\sum\limits_{i=1}^n (y_i - \hat{y}_i)^2 = 0$ .
But only in linear regression could you use the expression $\hat{y}_i = \beta_0 + \beta_1.x_i$ , then minimize the sum of squared errors w.r.t. $\beta_0$ and $\beta_1$ to obtain the following constraints:
$ \begin{align*} \sum\limits_{i=1}^n x_i(y_i - \hat{y}_i) &= 0 \\ \sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) &= 0 \end{align*} $
and then these constraints are used to prove that the quantity $\sum\limits_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) =0$, which yields SST=SSE+SSR.
So, my question is, if suppose regression wasn't linear, would:

SST=SSE+SSR still hold? If yes, why? If no, why?
If the answer to the above question is a no, then a follow-on question I would like to pose is: why is $R^2$ the go-to measure for assessing the regression performance of a model or a technique?


texts: What fields offer most data science job opporunities?

texts:
I'm now transitioning to data scientist as bioinformatics PhD.

What fields need lots of data scientists? Or offer more opportunities?
I guess business/finance and internet?


texts: Confusion matrix in multilabel classification of an object in more than one class simultaneously

texts:
Regarding a classification problem where for example given an image which depicts a human and we are trying to predict their stance and their behavior. For example Human 1: 'Sitting' and 'Eating' in the first image whilst Human 2: 'Standing up' and 'laughing' in the second. What is the appropriate way of applying the confusion matrix on the predictions. Do I have to unify the predictions?

E.g we have 5 different stances and 5 different behaviors, as a result the confusion matrix is of size 25x25 because we have 25 different classes. Or is there any other way of dealing with such problems?

Is it possible to do the same with multiple objects on an image and how?


texts:
Is there a metric in tensorflow.keras.metrics that counts how many time the predicted output and the real output have the same signal?

For example, if the y_pred = [-0.1, 0.3, -0.22, -1] and y_val = [-1.3, 1.2, 0.5, -2], then the output should be 0.75.

I am working on a project that takes as input, along with other features, a column of real values around 0, and should predicted if the next elements of this column should be negative or positive (regardless of their absolute value). As it's only logical, I am using tanh as the activation function, but if I use as metrics accuracy or binary_accuracy, the output is always zero.


texts: Can we optimize regression problems that have categorical variables by encoding them if on the other hand we are inserting multicollinearity?

texts: Can we optimize regression problems that have categorical variables by encoding them if, on the other hand, we are inserting multicollinearity?


texts: Should I remove the trend from timeseries when using DeepAR

texts:
I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM)

I figured out from the paper that SageMaker's DeepAR deals internally with seasonality, but does the same thing stands for trend?
Let's say I have multiple timeseries, where some of them have positive, and some have negative trend. Should I remove trend and then use DeepAR prediction, or should I just ignore it and let DeepAR handle it?


texts:
I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the words

  senior, primary, lead, head, vp, director, stuff, principal, chief,

or the words

  analyst, expert, modeler, researcher, scientist, developer, engineer, architect.

How can I get all such possible words with their distance ?


texts: Optimizer for Convolutional neural network

texts:
What is the best optimizer for Convolutional neural network (CNN)?

Can I use RMSProp for CNN or only for RNN?


texts: Which is better: multi-output model or separate models for similar tasks?

texts:
I am working on two problems:

classification of images into high-level classes (e.g. shoe, dress, jacket etc.)
classification of the attributes of the same images on a lower level (e.g. shoe style, color of the dress etc.), assuming that the high level class is known

Currently, I have designed an architecture for the 2nd problem as a multi-class multi-output network with ResNet50 as the backbone. Now I am dealing with the 1st problem and I have two paths to follow:

consider both problems as different tasks and train separate models for them
share the backbone between 1st and 2nd problems and train only one multi-output model

Which option is better? Which one should work better? Are there any good practices for combining similar tasks?


texts: How do I store/model data needed for my recommendation module?

texts:
I'm reading data from a store's product catalog, a 100mb xml file which contains product-wise attributes like prices, categories, etc.

Given a product_id, my job is to find the products most similar to it. Let us say, I use the following attributes: brand, color, price, categoryto find similarity. Here's how I'm going about things:

1) I parse through the catalog, and save the mapping between each product_id and its feature:value pairs in product_lists.json.

[ { 123:
      { 'brand' : 'abc',
        'color' : 'xyz',
        'category' : 'pqr',
        'price' : 123
      }
  },
  ...
]

2) I go through this product_lists.json and create attributes.json which basically lists all possible values of each attribute:

{ brand : [ 'a1', 'a2', ... ],
  price : [ 10, 1000, ... ],
  category: ['c1', 'c2', ... ],
  color : ['q1', 'q2', ... ]
}

3) I go through the attributes.json file and create indexfile.json, which maps each attribute value to a unique id:

[ { 'color1' : 1, 'color2' : 2, ... },
  { 'price_range_1' : 1, 'price_range_2' : 2, ... },
  { 'category1' : 1, 'category2' : 2, ... },
  { 'brand1' : 1, 'brand2' : 2, ... }
]

4) Then I refer to indexfile.json and product_lists.json and create a final training.json file by replacing values of attributes in the former with indices from the latter:

[ ["225845", [0], 6, [0, 1], [0]],
  ["105016", [], 6, [2], [1]],
  ...
]

5) Lastly, I take in a product_id, go through training.json and find similar product matches using knns, simple == checks, Levenshtein distances, proximity et al depending on the features.

The problem is, this creates a lot of .json files which is obviously very ugly. Then there is the problem of redundancy - while it makes sense to use knns for price/size features, I don't really need to bin, index and then match categorical features like brand or model names.

I am primarily looking for suggestions as to how I can save the data for each step in a more optimal way instead of creating so many separate json files, and if there is a better method overall to perform the similarity check given the unpredictable nature of the features (categorical, discrete, continuous) and a possible need to add weights to the features while computing. I have to do this for about 20 stores, so a universal suggestion would be great.


texts: How to interpret the output of XGBoost importance?

texts:
I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.

What is the meaning of Gain, Cover, and Frequency and how do we interpret them?

Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters here

Are there any other parameters that can tell me more about feature importances?

From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.

I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.

importance_matrix

Output:

           Feature         Gain        Cover    Frequence
  1:            xxx 2.276101e-01 0.0618490331 1.913283e-02
  2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01
  3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01
  4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01
  5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02

214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05
215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05
216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05
217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05
218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05


texts: How would I be able to improve my CNN model (Keras)?

texts:
Recently I read a research paper on age detection using facial images. So right now because of that I was trying to  see how far I could get by applying a CNN to a dataset of facial images (with their respective ages) in order to predict their ages which would be in bins (ex. 0-10, 11-20, 21-30...).

For training and testing

training.shape (50000, 28, 28)
testing.shape (2938, 28, 28)

I tried to keep the images small as they would be able to run faster as well as using grayscale.
And for the actual layers themselves I tried to keep it simple, for now,

model = Sequential()
model.add(Conv2D(64, kernel_size=3, activation='relu', input_shape=(28,28,1)))
model.add(Conv2D(32, kernel_size=3, activation='relu'))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))

Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 64)        640
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 24, 24, 32)        18464
_________________________________________________________________
flatten_1 (Flatten)          (None, 18432)             0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                184330
=================================================================
Total params: 203,434
Trainable params: 203,434
Non-trainable params: 0
_________________________________________________________________

Compiled with the following

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

So far the best accuracy after running for 100 epochs has been 37.16. Which isn't great but recently I've gotten access to one of my schools gpu's so I wanted to fix anything that I'm doing wrong and improve my model. Is there anything you could recommend when it comes to improving the model, theres probably a lot this is more my first time trying to do this.


texts: Looking for an algo transforming numerical attributes into categorical attributes -cleverly

texts:
I created an algorithm which works on categorical attributes. The input data comes with categorical attributes, but numerical ones as well.

How can I apply a pre-processing which transforms the numerical attributes into categorical ones, in clever ways:

the number of categories can adapt to the distribution of numerical values

-> for example if there are 100 values, 99 of them grouped in a small range, and one clear outlier, then 2 categories should be created.

the cutoff between categories should do its best to avoid that two close values are in different categories.

-> for example, it is annoying that for 2 numerical values ranging from 0 to 100, a cutoff at 50 leaves values 49 and 51 in 2 different categories.

Is there a field in computer science which deals with these issues? If there is an algorithm, is it pre-implemented in any language?


texts: Building a content-based recommendation system using products' metadata as features?

texts:
I am currently working on an apparel recommendation system, where I have tabulated data containing a list of products with their respective metadata (brand, category, color etc.)
I have an additional column of client ids to denote which client has bought which product.
I want this content-based recommendation system to recommend a client a bunch of products, based on the metadata of the products they have purchased in the past.
I am trying to find a way to learn user profile based on the past purchases. Could any machine learning algorithm be used for this purpose?
A lot of content-based approaches that I have seen so far take cosine similarities of tfidf representation of the textual description of the product to calculate scores. But, I believe the metadata are too rich to concatenate and then take cosine similarities.
I tried training a classifier for each client (there are 129 clients), but that doesn't seem like a deployment-friendly approach. Are there any suggestion how I can achieve this?
Many thanks in advance!


texts:
I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the two classes now. Then I will use a neural network to classify them into positive and negative classes.

The problem with the dataset is that it posses a lot of redundant data (Retweets basically). Manually deleting them is not possible and I have tried to find a programmable solution via the Tweepy API but couldn't find any.

So my question is do I need to get rid of these redundant records or should i leave them as is?


texts: How do I interpret the output of linear regression model in R?

texts:
I have the following linear regression model and its analysis. There are a few errors, but I am not very sure about the errors. I have not succeeded in finding them so far.
First, the 95% confidence interval for the slope should be

So the calculation is wrong.
Second, I'm not sure about the interpretation of the confidence interval. How would you interpret it in the context ?


texts: When should embeddings not be used for categorical data? What are their limitations?

texts:
I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector.

Vectors corresponding to all one-hot encodings are first embedded into the dense space randomly. As the embedding gets trained, the vectors move in the dense space from their initial random positions to positions that add algebraic meaning to the dense space. The data basically organizes itself.

This process of data arranging itself seems too good to be true.

I am posting this question here because in all the articles I have read about embeddings, they are presented as a silver bullet, and I wasn't able to find any texts on the limitations of embeddings.

This would essentially enable all categorical data to be converted into dense, meaningful representations during preprocessing or as part of training of a bigger model).

This is an example of embeddings being used to convert all categorical features into dense representations:

# Input layer for religion
encoder_liv = Sequential()
encoder_liv.add(Embedding(liv_cats,4,input_length=1))
encoder_liv.add(Flatten())

# Input layer for religion
encoder_edu = Sequential()
encoder_edu.add(Embedding(edu_cats,4,input_length=1))
encoder_edu.add(Flatten())

# Input layer for triggers(x_b)
dense_x = Sequential()
dense_x.add(Dense(4, input_dim=x.shape[1]))

model = Sequential()
model.add(Merge([encoder_liv, encoder_edu, dense_x], mode='concat'))
# model.add(Activation('relu'))
model.add(Dense(output_dim=12))
model.add(Activation('relu'))
model.add(Dense(output_dim=3))
model.add(Activation('softmax'))

model.compile(optimizer='adagrad', loss='categorical_crossentropy', metrics=['accuracy'])

An embedding is used in this video to replace the entire encoder network of a convolutional autoencoder generating human faces. here is the accompanying code.

So, what's the catch? Why should I not use always use an embedding for all categorical data? What are the constraints limiting an embedding's use/success?


texts: Visualizing decision tree with feature names

texts:
from scipy.sparse import hstack
X_tr1 = hstack((X_train_cc_ohe, X_train_csc_ohe, X_train_grade_ohe,X_train_price_norm, X_train_tnppp_norm, X_train_essay_bow, X_train_pt_bow)).tocsr()
X_te1 = hstack((X_test_cc_ohe, X_test_csc_ohe, X_test_grade_ohe, X_test_price_norm, X_test_tnppp_norm, X_test_essay_bow, X_test_pt_bow)).tocsr()

X_train_cc_ohe and all are vectorized categorical data, and X_train_pt_bow is bag of words vectorized text data.
Now i applied decision tree classifier on this model, i got this.

i took max_depth as 3 just for visualization purpose.

my question is i want to get feature names in my output instead of index as X2599, X4 etc.
i know i can do it by vect.get_feature_names() as input to export_graphviz, vect is object of CountVectorizer(), since i have already merged this vectorized data using hstack, now how to get feature names in this decision tree.


texts: I read in this post Is the R language suitable for Big Data that big data constitutes 5TB, and while it does a good job of providing information about the feasibility of working with this type of data in R it provides very little information about Python. I was wondering if Python can work with this much data as well.


texts:
In computer vision, if we don't have a large training set, a common method is to start with a pre-trained model for some related task (e.g., ImageNet) and fine-tune that model to solve our problem.

Can something similar be done with natural language processing problems?  I have a boolean classification problem on sentences and don't have a large enough training set to train a RNN from scratch.  In particular, is there a good way to fine-tune a LSTM or 1D CNN or otherwise do transfer learning?  And, if we want to do classification on sentences, is there a reasonable pre-trained model to start from?


texts: How to construct a Decision tree in R where the training data has a frequency associated with each class

texts:
Essentially, this is my data set

X Class    Sex   Age Survived Freq
1   1st   Male Child       No    0
2   2nd   Male Child       No    0
3   3rd   Male Child       No   35
4  Crew   Male Child       No    0
5   1st Female Child       No    0
6   2nd Female Child       No    0
7   3rd Female Child       No   17
8  Crew Female Child       No    0
9   1st   Male Adult       No  118
10  2nd   Male Adult       No  154
11  3rd   Male Adult       No  387
12  Crew   Male Adult       No  670
13  1st Female Adult       No    4
14  2nd Female Adult       No   13
15  3rd Female Adult       No   89
16  Crew Female Adult       No    3
17  1st   Male Child      Yes    5
18  2nd   Male Child      Yes   11
19  3rd   Male Child      Yes   13
20  Crew  Male Child      Yes    0
21  1st Female Child      Yes    1
22  2nd Female Child      Yes   13
23  3rd Female Child      Yes   14
24  Crew Female Child      Yes    0
25  1st   Male Adult      Yes   57
26  2nd   Male Adult      Yes   14
27  3rd   Male Adult      Yes   75
28  Crew  Male Adult      Yes  192
29  1st Female Adult      Yes  140
30  2nd Female Adult      Yes   80
31  3rd Female Adult      Yes   76
32  Crew Female Adult      Yes   20

If there were no frequency but only single valued data, then I know how to invoke rpart to construct a decision tree for me. How to do it considering the frequency of each class?

I am a beginner in R. Thanks


texts: Are annotated audio datasets augmented with mutated versions the way image datasets are?

texts:
Data augmentation is very standard for annotated image datasets for tasks like image labelling.  Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust to noise and not overfit on irrelevant features.

How are for example speech recognition training datasets pre-processed?  Are they augmented with background noise or other mutations like image datasets are?  Has there been any work in this direction?


texts:
I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.

I attached paper and R package that implement SMOTE for regression, can anyone recommend a similar package in Python? Otherwise, what other methods can be use to upsample the numerical target variable?

SMOTE for Regression

smoteRegress: SMOTE algorithm for imbalanced regression problems

Update:

I found the following python library which implements Synthetic Minority Over-Sampling Technique for Regression with Gaussian Noise

smogn


texts: Confusion for considering accuracy or standard deviation in selecting the best parameters

texts:
I have a model with a various parameters to test.

The size of the dataset I have is not really large (~500 documents).

My issue is that when I test the parameters using 10 CV, some of them produce high accuracy value but the Standard deviation value of the folds (accuracy values of the folds) is high.

ex.

Model setup 1: acc: 0.81, STD: 0.23
Model setup 2: acc: 0.76, STD: 0.05

Setup 1 has higher accuracy but the std is high, where setup 2 has lower accuracy but with more stable results.

Thus, how can I pick the best model?


texts: Reducing MAE or RMSE of linear regression

texts:
I am trying to guess a home price and at the end I intend to figure out a formula by using linear regression. As you can see here, I have 1480 samples with 45 features in which price(fiyat) is the target variable.
Do higher values for $RMSE$ and $MAE$ mean that the dataset cannot be trained in a good manner?
Is there a way to reduce these values?
As you can see $R2$ seems well.

How can we say that how much percentage of error occurs for the guesses on average? If we can how much could you say?
There is really substantial difference between the prices and guesses as being seen below:


texts:
Light gbm has the metric with log_loss for binary or multi classification.

is Random Forest also has the loss function with log_loss?


texts: Why am I getting crossvalidation scores of 0 only

texts:
I am trying Catboost package with iris dataset with following code:

from sklearn.datasets import load_iris
iris = load_iris()
from catboost import CatBoostClassifier
model = CatBoostClassifier(iterations=50,
                      learning_rate=0.1, depth=4,
                      loss_function='MultiClass')
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, iris.data, iris.target)
print(scores)

The output is:

[0. 0. 0.]

Why are the scores 0 only? I expected them to be close to 1. I tried adjusting parameters but results are still the same. Are these errors rather than classification accuracies? Thanks for your insight.

Edit: It appears that with CatBoostClassifier, cross_val_score() uses KFold() rather than StratifiedKFold(), since adding cv=StratifiedKFold() in cross_val_score function solves this problem.

With sklearn's classifiers such as LogisticRegression or SVC, cross_val_score uses StratifiedKFold as default (see here).


texts: Getting same result for all predictions in cnn

texts:
This is my first time training a model in cnn and predicting results but I am getting same value for images I have input. Here is my code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.optimizers import Adam
from keras.layers.normalization import BatchNormalization
from keras.utils import np_utils
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D,
GlobalAveragePooling2D
from keras.layers.advanced_activations import LeakyReLU
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
classifier=Sequential()
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train/=255
X_test/=255
number_of_classes = 10
Y_train = np_utils.to_categorical(y_train, number_of_classes)
Y_test = np_utils.to_categorical(y_test, number_of_classes)
classifier.add(Convolution2D(32,3,3,input_shape=
(28,28,1),activation='relu'))
classifier.add(BatchNormalization(axis=-1))
classifier.add(MaxPooling2D(pool_size=(2,2)))

classifier.add(Convolution2D(32,3,3,activation='relu'))
classifier.add(BatchNormalization(axis=-1))
classifier.add(MaxPooling2D(pool_size=(2,2)))

classifier.add(Flatten())

classifier.add(Dense(output_dim=256,activation='relu'))
classifier.add(BatchNormalization())
classifier.add(Dense(output_dim=10,activation='softmax'))

classifier.compile(optimizer='adam',loss='categorical_crossentropy',metrics=
['accuracy'])
gen = ImageDataGenerator(rotation_range=8, width_shift_range=0.08,
shear_range=0.3, height_shift_range=0.08, zoom_range=0.08)

test_gen = ImageDataGenerator()
train_generator = gen.flow(X_train, Y_train, batch_size=64)
test_generator = test_gen.flow(X_test, Y_test, batch_size=64)
classifier.fit_generator(train_generator, steps_per_epoch=60000, epochs=1,
                validation_data=test_generator, validation_steps=10000)
import cv2
image = cv2.imread("pitrain.png")
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) # grayscale
ret,thresh = cv2.threshold(gray,150,255,cv2.THRESH_BINARY_INV)
#threshold
kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(3,3))
dilated = cv2.dilate(thresh,kernel,iterations = 13) # dilate
im2,contours, hierarchy =
cv2.findContours(dilated,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
# get contours
# for each contour found, draw a rectangle around it on original image
for contour in contours:

# get rectangle bounding contour

[x,y,w,h] = cv2.boundingRect(contour)
# discard areas that are too large

if h>300 and w>300:

    continue
# discard areas that are too small

if h<40 or w<40:

    continue
# draw rectangle around contour on original image

cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,255),2)
image.shape
image=image[:,:,:1]
newimg = cv2.resize(image,(28,28))
img.shape
img = np.reshape(newimg,[1,28,28,1])
cv2.imshow("screen",img)
classifier.predict(img)

The output I am getting is array of zeros with 1 at third position.
This is where I copied the contour part from https://www.quora.com/How-do-I-extract-a-particular-object-from-images-using-OpenCV

Epoch is equal to 1 because I only wanted to test my model and still I got accuracy of above 99%


texts:
I set up a forecasting model that predicts call data. The forecast model uses a random forest regression model.

Data:
I have call data about every day in 15 minutes intervals of a year since 2013.

Here is a plot of the accumulated values over months:

It can be clearly seen that call data has almost doubled in 2017 over 2016. This trend should also be observable for the next few years.

features:

First, the format of my data:

 DATE                     CALL
 ....
 2017-10-23 10:15:00.000    259
 2017-10-23 10:30:00.000    292
 2017-10-23 10:45:00.000    309
 ....

From this I extracted the following features:
I have extracted the following features to predict my target variable Y (call data):

-Weekday
-Month
-Holiday (yes / no)
-Interval of the day

So I ask my model:

What is the call volume of a day and interval with the following features?

I have used the years 2015-2016-2017 to train the model. However, the model does not give the desired prognosis.

He even predicts the days for 2017 wrong. Although I gave him the data as training data.

Questions:

- Should I work on my features?

- How do I show my forecasting model that the data will double year by year
  as observable since 2016?


texts: Getting some data from a csv file into a list of tuples using Python

texts:
I have a csv file of almost 2 million rows and about 20 columns. I am interested in about 40000 of these rows, and in 5 of the columns 'squat', 'bench', 'deadlift', 'bodyweight', 'total'. I'd like to get tuples of these 5 values, but only of the rows that have 'female' in column 'sex' and 'ipf' in column 'parent federation'. Also sometimes one of the first 5 columns has no value specified, I'd like to ignore those.

I can do this by going row by row, checking the condition and then adding a tuple to a list, but this is way too slow and takes up too much memory. Is there a way to expand on this how to take CSV file input in list of tuples using only a couple of columns and filtering?


texts: SKLearn Boston dataset gradient descent not working

texts:
I am trying to compare some simple methods for linear regression as an exercise.

I have already used LinearRegression from the SKLearn library in python as well as the formula of linear regression.

As a last test, I wanted to implement gradient descent myself, however, my method takes too long to converge and the values that are found are not correct compared to my other two methods. This is what my code looks like in python:

def mse(X, theta, Y):
    h_theta = X.dot(theta)
    return 1/(2*X.shape[0])*np.sum(np.square(h_theta-Y))

def gradient_descent_step(X, theta, Y, alpha):
    h_theta = X.dot(theta)
    difference = h_theta-Y.reshape(-1, 1)
    update = X.T.dot(difference)
    m_inverse = 1/X.shape[0]
    update = -alpha*m_inverse*update
    return theta + update

X = np.append(np.ones((boston.data.shape[0], 1)), boston.data, axis=1)
theta = np.ones((X.shape[1], 1))
print(theta.shape)
print(X.shape)
alpha = 0.0000001
for i in range(500000):
    #print("Cost in step " + str(i) + ": " + str(mse(X, theta, boston.target)))
    theta = gradient_descent_step(X, theta, boston.target, alpha)

print(theta)
print("Cost: " + str(mse(X, theta, boston.target)))

My learning rate is very small, because otherwise, the cost increases in each iteration. I do not see any mistake in my code, so I do not know what is wrong. I have not normalized the data, but my other two methods work fine without normalizing the data, so I think this one should too?

After running the above piece of code, I get a remaining cost of 31988.011713726573 which is too large.

What am I doing wrong?


texts: Data structure design for supporting arbitrary number of columns in table or database

texts:
I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data.

The task I'm currently struggling is: user needs to enter data column by column, which would consist of a number n of ID's , a number m of attributes and a number k of classes, with the conditions that n, m > 0 and k >= 0. Data is heterogeneous, so indexes can be both numeric or text, and the same goes for attributes and classes. I'm supposing there will be no null in the data for simplicity.

I'm currently thinking on:

1) Creating a table with more than enough columns (all with null values), so that I can work using only the non-null columns (which will be got from user input). However this would limit the size of the datasets people could input.

2) Resorting to create an specialized data structure on a programming language, do all the work there and finally, create a table dynamically  to store the result data there.

3) Using a database specialized for this kind of data (maybe a document-based DB).

4) Create a data structure on the RDBMS itself (I'm using PostgreSQL), let's say a variable size array, so that I can create the table directly from the user input, using only 3 variable arrays (one for indexes, one for attributes and one for classes). However, I keep in mind that attributes and indexes could be of different types, so the array would have to support heterogeneous data type and I don't know if this is possible on a RDBMS or SQL.

I've been looking for information on information but got no result until now. Any guidances to a package, language library, extension or paper, thesis, technical report with relevant information would be appreciated.
Also, personal experiences with doing something similar could be useful.


texts: Share Jupyter Notebook with a non programmer

texts:
I would like to share a Jupyter Notebook with somebody who is not a programmer (via a URL).

I understand that I must publish my work to GitHub and then use http://nbviewer.jupyter.org/ to render the notebook. However, I don't know how to correctly publish my .ipynb file into Github.


texts:
I have a dataset with at least 70% of labels incorrect.
I'd expect that incorrect labels would compensate each other while true labels will be taught properly (given a very large dataset).
For example, if I have 300 samples saying a => -1 and 300 samples saying a => 1, the result for the input "a" eventually will be 0 (for a regression problem).
If I use Adam for the example above, won't it affect the results for the inputs with noisy labels due to its adaptive nature? Won't it be better to use SGD instead and decay the learning rate, or does Adam change its weights only at the end of every epoch?


texts: Spark MLLib - how to re-use TF-IDF model

texts:
I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm.
Here's the process:

Datasets:

Full sample data (labeled)

Training (labeled)

Test (labeled)

Unseen (non-labeled)

This is my current workflow:

Fit IDF model (idf-1) on full Sample data

Apply(Transform) idf-1 on full sample data

Split data set into Training and Test data

Fit ML model on Training data

Apply(Transform) model on Test data

Apply(Transform) idf-1 on Unseen data

Apply(Transform) model on Unseen data

I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data.

Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for |D| and DF|t, D| while use new TF|t, D| ?

Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?


texts: From where does BERT get the tokens it predicts?

texts:
When BERT is used for masked language modeling, it masks a token and then tries to predict it.
What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression problem) and then use that token? Or does it do a softmax over all possible word tokens? For the latter, isn't there just an enormous amount of possible tokens? I have a hard time imaging BERT treats it like a classification problem where # classes = # all possible word tokens.
From where does BERT get the token it predicts?


texts: How to recognize product based on image using neural network?

texts:
Company has many products in their offer (some about 100,000), some of these are very similar to each other. In database there is available only one image per product.

Company want to make possible to recognize product based on video camera and display its specification. Is it possible to train new model with this kind of data or build it using existing model?


texts: Estimating model for transition probabilities of a Markov Chain

texts:
Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\times S$ values of (independent) variables, where $K$ is the number of variables that I can use to explain the transition probabities ($p_ij$ are my dependent variables and the matrices $X_k$ are the independent variables).

Remember that $\sum_j pij=1$ for each $t$.

In the end, I am looking for panel models to explain the transition probabilities, where the parameters are constant over time and (with maybe exception the constant) the parameters are also constant over diferent transition probabilities.

Just to be clear... Consider the following example... Imagine that an animal prefers to stay in places that there are food and water. Let the $T\times S$ matrix (X_F) the matrix that tells the amount of food in each place $s\in S$ and in each time $t\in T$ and (X_W) the matrix that tells the amount of water  in each place $s\in S$ and in each time $t\in T$.

I want to use $X_F$ and $X_S$ to explain the transition probabilities. I do have the values of the transition probabilities over time and I want to use these matrices to explain their values.

I think that I design a kind of fixed effect logit model for each state in $S$. However, I would have to estimate $S$ logit models. I believe that the probabilities $p_{ij}$ and $p_{ji}$ should not be estimated in different models, since they seem to be related.

Any hints? Are there solutions in the literature of such kind of problem?


texts: Selecting time window for very different feature lengths

texts:
I'm doing my first steps with python, keras and a 1D time series classification. I can train a model, save it, apply it to validation or test data.
I'll build my first implementation around this tutorial.
Now, I want to port this knowledge to my actual project:

A sensor is outputting a voltage each 1min*
The feature duration is around 1-5min to half a day or longer**

Please see the figure below to get a better understanding of the data:

I want to classify features that are very different in their duration, as mentioned in **
I can imagine to just train two different models with greatly different time-windows. What do you recommend, is there a best practice?


texts: Using EM (Expectation Maximization) algorithm for Training Logistic Regression

texts: Is it possible to learn the weights for a logistic regression classifier using EM (Expectation Maximization)algorithm? Is there any instance reference?


texts:
I am learning about Markov Chain and Bayesian Nets. However at this point I am a bit confused about what types of problems are modelled with the two different models presented to us. From what I understand (mostly from the examples I have read) Markov Chains are being used to represent the change in a single type of variable over time. So for example a random variable X representing the weather. Let X = {sun, rain}. Then for a markov chain, at time = 0 we are given P(X) and a transition model $P(X_t/X_{t-1})$. So with this knowledge we could calculate $P(X_\infty)$. Like asking a question, given the initial distribution of a random variable X and a transition model, what would be the value of P(X=x) at time t? The solution of such a question can be answered by mini forward algorithm.
Now for bayesian network, from what I understand, we model dependencies among different random variables. So here essentially we have some random variables that may have a causation relationship with other variables. There are some nice properties about such networks that let us define the joint over all variables easily.
Onto my question -> Often the topic of markov chains is introduced before bayesian networks. What is the relationship between the two? because I can't seem to draw parallels between them, to me I see both as quite different approaches at modelling quite different problems.
In what other contexts can markov chains be used? or are they always used to model a single variable varying over time steps? I hope to gain some clarity to distinguish between the two and hopefully this would help me understand the topics better! Any suggestions/readings/links are much appreciated!


texts: How to resample one dataset to conform to the distribution of another dataset?

texts:
I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??

I check the similarity/difference of two datasets by checking individual features from DS_A against DS_B, in shape, and percentiles. Features are mostly numerical, some binary, some normalized.

Background:

Some time ago I trained a model using dataset DS_B as ground truth. Now, I want to retrain the model with more recent data and see if the performance improves. The new ground truth data I collect is DS_A, but due to practical reasons, new data is collected somewhat differently, and hence the feature distribution in the new data set is different from the old data set.


texts: I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand what it means. I know what a feature is but I don't understand the idea of latent features. Could please explain it? Or at least point me to a paper/place where I can read about it?


texts: Low accuracy in multi-class classification despite all data being generated from rules

texts:
I have a well defined data where i have cleaned up my data to final form which has 20 features mapping to a number between 1 to 100. Upto 5 features are enabled(value set to 1) for each row. The data looks something like below

 Result|f1|f2|...f19|f20
 45    |0 | 1|... 1 | 0
 92    |0 | 0|... 1 | 1

I'm trying to build machine learning models that can give me good accuracy and preferably models which can handle warm_start since each iteration generates 1 row that i need to fit into existing build model.

below are 2 classifiers that i tried to set some baseline

randclf = RandomForestClassifier(n_estimators=50)
decclf = DecisionTreeClassifier(criterion = "gini", random_state = 100,max_depth=3, min_samples_leaf=5)

However even with 100,000 records i'm getting very poor result with accuracy around 15-20%. considering how predictable data is(data is generated based on finite set of rules) i was expecting very high accuracy.

I'm i doing something wrong, i want get the high accuracy in classifying data(predicting Result) based on features given, can you suggest some models that might work well this kind of data. what about tensorflow and neural network approach?

data:

https://github.com/sachinhegde6/machinelearningdata

Update:
Data imbalance is something i cant help as they are generated based on rules.


texts: DNN predicting the same value for train+test Data

texts:
I have trained a Deep Neural architecture for regression problem and after the hundred's of epochs, model predicting the same output for both training and testing data.
When I reduced the batch size, atleast I'm not getting the same value for all the samples.
According to me reasons could be, model is not getting trained and gradient died in between. If it's correct, Im not sure why it's happening in my case where I have just "3 CNN + 3 DNN" layers on my architecture.

Using ReLu as an activation function, changing to LeakyRelu will be beneficial?
How batch size contributing on it?
Any other probable reason for this problem.

Regards.


texts: Purpose of converting continuous data to categorical data

texts:
I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data.

For example, they converted both the Age and Fare features into ordinal data bins.

I understand that categorizing data like this is helpful when doing data analytics manually, as fewer categories makes data easier to understand from a human perspective. But intuitively, I would think that doing this would cause our data to lose precision, thus leading to our model losing precision or accuracy.

Can someone explain when is converting numerical data to ordinal data is appropriate, and the underlying statistics of why it is effective?


texts: What's the difference between sequence preprocessing and text preprocessing in Keras?

texts: In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" and "text" are the same. How to understand the differences between these two preprocessing operations?


texts: Trouble understanding the odd behavior of a trained Autoencoder

texts: I have built an Autoencoder model (aim is to do anomaly detection). Due to large difference in scale of different features, I have transformed the data into [0,1] interval using MinMaxScaler() from sklearn. Now that I have trained the model, I noticed a curious, and at present unable to explain, fact that the value of the reconstructed features, for all 60 of them, lie within 0.6 and 0.7. To be specific, for each feature, the range is between 0.6 and 0.75. A histogram of the any random feature and its reconstruction looks like this: or like this  (Orange is the reconstructed feature for the training set). In fact, if I plot the mean for each reconstructed feature vs the mean the original feature, it looks like this . Again the orange plot is the mean of the reconstructed feature. I am still unable to explain it other than the fact the model underfits. But why the range is 0.6-0.7 is still illusive. If anybody can shed light on this, that would be great. If you need more info, please leave a comment. Thanks in advance.


texts: Consolidating Region Dimensions in Tableau

texts: I am currently using the example excel spreadsheet in Tableau. After dragging category and segment dimensions to rows and the region dimension and quantity measure to columns, I realize there are two many regions. Tableau has seemed to automatically group these regions into Africa, Asia Pacific, Europe, LATAM, and USCA. How do I make this more general "super" regions the columns instead of the originals?


texts: Does multiple cross validations after a model selection make sense?

texts:
Let be $M$ a model. Let also be $H_{1 \leq i \leq n}$ some hypothesis of $M$.

I have a dataset $\mathcal{D}$ and I want to run $K$-fold cross validation on $\mathcal{D}$ to pick the best model $H_j = \text{argmin}_{H_i} \ \ \varepsilon(H_i)$ where $\varepsilon$ denotes the RMSE.

Once I get $H_j$, I want to perform multiple $K$-fold cross validations on $\mathcal{D}$ to build a confidence intervals of this estimator.

I want to do this procedure with some models $M_k$ to select the best one based on the confidence intervals.

My question : Is this procedure meaningless ? Please be kind and provide answers with references.

Thanks !! Keep safe and stay home guys :)


texts: Text PreProcessing not improving accuracy

texts: I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs review dump to predict if a review is positive or negative and I wasn't preprocessing initially but recently started to add it in. My accuracy score is 84% using SGDClassifier but whether or not I preprocess the training/testing dataset it seems to be the same?


texts: What method/algorithm for constrained multi-target regression

texts:
I am working with three dimensional measurement data and want to model them using a multivariate linear regression.
I have already implemented a simple gradient descent algorithm to solve the classic regression problem
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_3x_3.$
However my ultimate goal is to fit three target variables with a hard condition:
$y_1 = \beta_{10} + \beta_{11}x_1 + \beta_{12}x_2 + \beta_{13}x_3,\\
y_2 = \beta_{20} + \beta_{12}x_1 + \beta_{22}x_2 + \beta_{23}x_3,\\
y_3 = \beta_{30} + \beta_{31}x_1 + \beta_{32}x_2 + \beta_{33}x_3,\\
\text{subject to }\beta_{11}+\beta_{22}+\beta_{33}=0.$
Or in other words
$\underline{y} = \underline{\underline{B}}\cdot\underline{x} + \underline{\beta_0},\\
\text{subject to } \mathrm{tr}(\underline{\underline{B}})=0.$
I've searched high and low for multi-target or multi-output regression methods but I could not find out how I would possibly go on to implement a hard condition into them. Happy for any kind of recommendations, even just keywords for further research.


texts: In survival analysis, which is the correct way to introduce a variable which changes the survival rate but occurs at different times?

texts:
I am making a survival analysis with a cox regression with proportional hazards, we  want to analyze wheter the introduction of a phenomenon influences the time until the death of an individual.

A similar example would be: We have patients which were given a medicine at different stages of the disease (and at different ages too and some were not given the medicine at all), how the treatment variable should be introduced so we can measure the "strength" of the introduced change.

I have a possible solution:

Create a dummy exogenous variable for ranges of the time in which the medicine was introduced. "First 0-6 months", "6 m - 1 year", "1-2 years", "Never", this measured since the birth of the patient.
Create the same exogenous variable but not dummy.
Change the time variable and measure it not from birth but from the introduction of the medicine (the problem would be the definition of this variable for the patients without medicine).
Duplicate the rows of the individuals who had the medicine. In the first repetition, the time would be between birth and the introduction of the medicine as a "censored death"; the second repetition would include the time between the introduction of the medicine and the actual death.

Nowadays, the model includes the introduction of the medicine as a dummy but doesn't have into account the time in which it was introduced.


texts: How to resolve country and nationality entities?

texts:
I've tried stemming and  lemmatization on this but nothing has quite worked so far.

How can I resolve country name and nationality as a singular entity?

For example:

Canada and Canadian should just be one entity: Canada
Uganda and Ugandan should just be Uganda

It does seem like stemming is one approach here. I just have found it misses a fair number of countries.


texts: H2o autoencoder anomaly detection for multivariate time series data

texts:
I am applying H2o autoencoder for anomaly detection for multivariate time series data.

I have many time series data for different metrics of network elements which are recorded every 15 minutes. I want to find out that:

At which timestamp that the anomaly happens
What is/are the main metric(s) causing the anomaly

I form the data matrix with columns as metrics and rows as the values of the metrics at a specific timestamp. For example, [[9:15, v1, v2, v3...], [9:30, v4, v5, v6, ...], ...].

I then train the data with autoencoder and use h2o.anomaly to check which data points have reconstruction MSE greater than the 99% percentile as the suspected anomalies. And these give me the timestamp when the possible anomalies happen.

By using this method, can I get the information of the main metrics causing the anomaly?

Thanks in advance.


texts: Calculating optimal number of topics for topic modeling (LDA)

texts:
am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think it is okay? or it is better to use other algorithms rather than LDA. It is worth mentioning that when I run my commands to visualize the topics-keywords for 10 topics, the plot shows 2 main topics and the others had almost a strong overlap. Is there any valid range for coherence?
Many thanks to share your comments as I am a beginner in topic modeling.


texts: How is Bayes theorem being applied in expanding the formula of Binary Independence Model?

texts:
The BIM formula makes use of Bayes theorem. Can anyone please explain:

How to read the probability of the form P(R=1|x,q)? Is it P((R=1|x),q) or P(R=1|(x,q))? Does , stand for AND here?
How exactly Bayes theorem being applied to expand it?


texts:
I have set of technical sentences extracted from few research papers.

e.g., Via analyzing the peer feedback content, it was found that the feedback provided by the mixed mode group was of better quality than that provided by the "peer comments" group; that is, the former provided more detailed feedback to individuals than the latter.

I want to extract the entities in the sentences and the relationships that take part with the identified. (e.g., SOV relationships)

What are the tools that I can use to efficiently extract them?


texts: General approahces for grouping a continuous variable based on text data?

texts:
I have a general methodological question. I have two columns of data, with one a column a numeric variable for age and another column a short character variable for text responses to a question.

My goal is to group the age variable (that is, create cut points for the age variable), based on the text responses. I'm unfamiliar with any general approaches for doing this sort of analysis. What general approaches would you recommend? Ideally I'd like to categorize the age variable based on linguistic similarity of the text responses.


texts: What would be the best way to structure and mine this set of data?

texts:
http://pastebin.com/K0eq8cyZ

I went through each season of "It's Always Sunny in Philadelphia" and determined the character groupings (D=Dennis, F=Frank, C=Charlie, M=Mac, B=Sweet Dee) for each episode. I also starred "winners" for some episodes. How best could I organize this data, in what type of database, and what data science tools would extract the most information out of it?

I was thinking of making an SQL table like so:

             (1)     (2)      (3)     (4)     (5)
Episode# | Dennis | Frank | Charlie | Mac | Sweet Dee
008      |    5   |  3,4  |  2,4    | 2,3 |    1
010      |    5   |  3,4,6|  2,4,6  |2,3,6|    1

...where all the values are arrays of ints. 6 represents that the character won the episode and each number represents one of the 5 characters. Thoughts?


texts: Choose points to maximize volume of convex hull

texts:
Suppose I have N points (labeled 1, 2, ..., k, ..., N) in D dimensions. I'd like to choose the order of points such that, after each point, the volume of the convex hull is maximized.

In other words, the volume of the convex hull is a function of the number of points V(k) (V(1) = V(2) = 0), I'd like to choose the sequence of points {k_1, k_2, ..., k_N} such that V(k) is maximized for each value of k.

The two algorithms I've thought of don't seem very efficient:

Brute force is O(N!)
Divide the space into boxes, bounded by the k points. Pick a point, then look in each of the little boxes, ordered by how far away they are from the box that contains the starting point. If there are multiple points in the box you choose, either randomly pick, or choose the point that's the furthest away. Now compute the centroid of the points you have. Then repeat until all points are consumed. This seems like O(N^p).

Is there a cute, out of the box solution to this problem? Or a paper where someone has solved it already?


texts: Collection Of Variable Length Sequences and Descriptions: A Search Problem

texts:
I have a tough problem and need some advice:

Suppose I have a collection of variable length sequences, many of which are unique -- imagine the moves to a chess game, eg

  d4 Nf6
  c4 g6
  Nc3 Bg7
  e4 d6
  Nf3 O-O
  Be2 c5
  O-O Bg4

...

and for each item in this collection, I have another collection of descriptions generated by humans (think comments - comment_1: "cool game", comment_2: "awesome sacrifice")

The goal is to mine the associations between the comments and the sequences to tag the sequences with human readable labels for search purposes.

I've thought about topic modeling for label generation + clustering/grouping the sequences, but I cannot figure out how to do something like cluster the games. I have millions of examples of the sequences if that helps. Any idea how I can measure the distance/similarity between sequences like this? Some kind of embedding? I've thought about trying a word2vec / doc2vec approach, but haven't tested yet.

Ideally I'd be able to input an unseen sequence and suggest labels/human-readable description for this sequence as well.


texts: Which machine learning/AI method is this problem related to?

texts: I am new to machine learning and AI, and I am working on a problem in which I need to clean a table (database) of text/words. For example I should delete words like and, the, etc. , or replace words like COMP with company or substitute other acronyms with the suitable words. I wanted to find some resources on the techniques proposed for this purpose and to find what else I can do to better clean the table. Thanks for your help.


texts: Regression model for continuous dependent variable and count independent variables

texts:
I am currently learning R and I am relatively inexperienced in the field.  Hope I can get some advice from you guys!

I am working on a project where I have to estimate the average processing time of different work items (tasks).

I have the following panel data:

My sample size is n=2000 individual workers, and T=10 (each time interval is a four week period)

Independent variables: 51 different work items.  I have count data for each work item (# of times they are performed by each worker over a four week period)
Dependent variable: Total Working Hour of the worker (over a 4 week period)

The goal of my analysis is to find the regression coefficents (which are estimâtes of the average completion time of each work item).  I may also include other regressors (other than #of work items) such as experience, age... into my model.

y= Bo + B1*X1 +...+Bk*Xk + e

y: total working hours
X: # of each work items type

Issues:

Right now, I finished cleaning and processing the data and I performed some exploratory data analysis.

Some work items have a lot of zeros (the work item is only performed once or twice by several workers in the time period).
From VIF, I can see that there are imperfect multicollinearity in the independent variables.  Some independent variables have VIF of 5 to 6.

Questions:

Any advice on how I should specify my model?

I look at boxplots and eliminate outliers of each regressor,  I see that some regressors are highly skewed (due to lots of zéros).

I also plot each regressors against the total complétion time to see if there is any linear relation. So do, other looks more like a quadratic relation.

Any way to deal with the multicollinearity aside from eliminating the regressors that have high VIF?  This is because I need to estimate the coefficent of each of the work item.
Should I set the intercept to 0?  I know for sure that when ALL the regressors are 0 (# of work items are all 0, I should have zero total working hours).

I would also welcome any other advice for this problem. Thanks!


texts: Cross Entropy vs Entropy (Decision Tree)

texts:
Several papers/books I have read say that cross-entropy is used when looking for the best split in a classification tree, e.g. The Elements of Statistical Learning (Hastie, Tibshirani, Friedman) without even mentioning entropy in the context of classification trees.

Yet, other sources mention entropy and not cross-entropy as a measure of finding the best splits. Are both measures usable? Is only cross-entropy used? Since the two concepts significantly differ from each other as far as my understanding goes.


texts:
When we train a neural network model for a classification problem, we usually have a dense output layer of size equal to the number of labels we have.

If the layer size was greater, the model can still be trained and have output

How can we interpret such output? are there any applications for this?


texts: How can I arrange data in different columns (for different constituents) from one?

texts:
I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want Aluminium, Berrylium and other variables in that column to be in different columns. The data file is attached.

SITE_NAME   SAMP_DATE_TIME      SAMPLE      Value   Units   METHOD_CATEGORY
SITE-7-1    10/04/1988 00:00:00 Aluminum    150     ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Aluminum    150     ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Beryllium   5       ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Beryllium   5       ug/L    INORGANIC

Data : https://drive.google.com/open?id=1SUhYd21U5iQmilABNwoaIXLfsc7Mo6dW

The output I am trying to get is:

SITE_NAME   SAMP_DATE_TIME  Aluminum    Beryllium   Bromide Bromomethane    Chloride    Hexachlorobenzene   Isosafrole  Zinc
SITE-7-1    10/4/1988 0:00   150        5           10      7250            10          10                  75
SITE-7-1    12/29/1988 0:00  150        5           10      8100            10          10                  16
SITE-7-1    5/10/1989 0:00   150        5           1000    10              8100        10                  10          16


texts: What should be the MINIMUM value of 'k' in LSH (Locality Sensitive Hashing) for 20M + data points?

texts:
For 20M + images, I'm thinking about using LSH for similarity of  Vectors or data points or more precisely image Embeddings extracted from ResNet. My data points are supposed to be growing but for the sake of simplicity, let us suppose we have 20M images.
What should be the minimum value of k so that I avoid Hash collision? I think  k = 25 as 2^25 = 33554432 is the minimum value to be used. Also, is there any method to find the optimum value of k.
I know there is a tradeoff between speed and accuracy. More the k, more the hashes, more the buckets, better the accuracy but lower the speed. Also, I can't find the optimum k by running a loop and checking the graph for 20M practically.


texts: How to train ML algorithm with multiple values in target data?

texts:
I am new to data science and machine learning and looking for some help.
I am trying to train a machine with following data set:

Here, L3 is the target variable.
As it can be seen that the target variable contains a set of possible values, I am wondering how I can train the machine with such a data.
In my test data, I might want to predict any of the one or more values given one value.
Example test case :

20    c     aa

'aa' is given value and I want to predict other values based on the values 20, c and 'aa'.
Is it possible to do what I am trying to achieve?

Thank you.


texts: Modeling uncertainty from Logistic Regression

texts:
Logistic regression is a part in a simulation pipeline that I use for some scenario analysis. The dataset that this is based on is not small but relatively noisy, and only one explanatory variable/feature. Of course I can say something about this uncertainty using frequentist or Bayesian methods but I would like to use this in the sequential simulation step as well, to get a fairer final estimate. What I'm planning on doing should work but is somewhat computationally expensive and I would like to know if someone has a better idea.

Approximate distribution of the coefficients of logistic regression with MCMC

Now simulate this n times:

Sample one of the two coefficients from this distribution
Fix this coefficient and use MCMC again to get approximated conditional distribution over second coefficient (could I get this in a better way? This seems highly correlated with the first coefficient)
Do the rest of the simulation run using this sampled model

I feel like there should be a better way but I'm uncertain.


texts: Hierarchical Clustering on transaction data

texts:
Problem Statement: Let's say I have buyer transactional data for every product, features are categorical and numeric. I want to cluster purchases that have similar attributes in terms of who's purchasing the product, their demographics etc .

I tried using hierarchical clustering and it returns certain clusters which are super similar and also clusters that do not look very similar, not sure why its even clustering those "not similar looking" orders

I am currently doing one-hot encoding for categorical variables, and minmax normalization for numeric variables.

I am using hierarchical clustering from scipy, using ward as the linkage method and using scipy.cluster.hierarchy.fcluster(Z, t, criterion='distance')
to form flat clusters with t set to 1.

Questions:
Is there a better way to transform categorical and numerical variables for this clustering?
How can I get better clusters - currently getting orders that look dissimilar when I look at the attributes being clustered together?
Should I be using a different distance function? Gower distance?
Any other linkage function?
Any metric I can use to measure how good my clustering is?
How can I determine parameter "t" in scipy.cluster.hierarchy.fcluster
Also is there a way to add weights to certain feature values to form clusters? For example, if there's a categorical feature which has a value with very high frequency in general population, I don't really want it to be given too much weightage while clustering


texts:
It is mentioned on this page that Label Propagation of scikit-learn can be used for regression also. However, nothing is mentioned on its API page.

How can I do semi-supervised learning for regression with Label Propagation or any other method?


texts:
This question is regarding a use case related to predictive maintenance. The final model built (based on the steps mentioned below) is used to predict the failure of a particular component for a particular type of device. In the Machine Learning context, it's regression use case for which deep learning model(s) are being built.

The model building process consists of two stages:

Initially based on the data (temporal in nature) from multiple devices, one single model has been built. The model has been cross validated with the data from the same set of devices (expected).
Now based on the model type and hyperparameters decided at #1, models are being built for individual devices (using data from respective devices. The temporal frequency of data at this stage is different from the frequency of the data used at #1). These models (for individual devices) will be deployed in production (i.e. one model per device). The model built at #1 will NOT be used in production.

Is this two stage process a valid approach of model building?

As per my understanding, instead of the two stage process and large number of individual models, a single model can be built with data from all devices (with matching frequency) in one shot. There is a possibility that devices differ from each other based on different characteristics. But those characteristics could be used as a feature while building the single model.

Based on domain knowledge, I am more or less convinced that one single model can server multiple devices. But, my question is more about the validity of two stage approach:

Decide model type and hyperparameters in the first stage.
Build individual models with data from individual devices based on the model type and hyperparameters decided in the first stage.


texts:
I need to find a good clustering for this data using sci-kit.

KNN is not appropriate as it creates blobs although these data are linearly separated.

import hypertools as hyp
hyp.plot(tocluster,'o', n_clusters=10)


texts: In transfer learning, we always use new data to retrain the pre-trained model. But, what is the specific and official definition of retraining? Or what papers mentioned this definition, in transfer learning field or reinforcement learning field?


texts: How does back propagation works through layers like maxpooling and padding?

texts: I know back propagation takes derivatives (changing one quantity wrt other). But how this is applied when there is maxpooling layer in between two Conv2D layers? How it gains its original shape when there is padding added?


texts:
I have large 3 dimensional dataset that I need to classify in a particular way. I'm hoping for something like a k-means clustering method, but instead of inputting the number of clusters I would like to input the maximum radius of a cluster.

Said another way, given a sphere of a defined size, I would like to find the minimum number of non-empty spheres that will cover all the data and classify the points accordingly.

I would like to output a 2D array of the same height and width, with labels that represent the sphere each point in the 3D array was classified in. No two points with the same label should have a euclidian distance greater than the maximum radius, no sphere should be empty, no spheres should overlap and the algorithm should use the minimum number of spheres possible to accomplish this.

Though I have little knowledge of the field, I have been looking into clustering algorithms for a little while and have not found one that accomplishes this.


texts: Auto.arima with xreg in R, restriction on forecast periods

texts:
I am using the forecast package and implement auto.arima with xreg. Here I want to forecast only for 1 year ahead but I am unable to use h parameter in the forecast function. Below is the reason for that:

  Definition is given in manual(F1 check):

  h = "Number of period of forecast but if xreg is used 'h' is ignored and the forecast period will be number of rows"

Please suggest me an alternate way to use h for the specific period forecast.


texts:
I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers which should stay and also similar cases can be seen in future observations. When a trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation)

Question: the main question is what should I do with one (or a few) bad score(s) out of many? How should I report them and how useful would that be?
Using 10 splits and 10 repeats for a data of size ~3000 observations, I will get 100 r-squared scores which are all in a good range (0.97 to 0.99). There is only one guy ruining the game and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01  9.73912516e-01  ... -1.15353090e+04 ...  9.72986827e-01]

What shall I do in this case? how to report it and/or how to cure it?


texts:
I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers.

Actually I don't wish to drop the data as it is an outlier because am not sure whether it is really an outlier or a meaningful value (like how income could be skewed).

So what I did is just standardize the columns using standard scaler.

Is it okay to do this? Am assuming that by doing this standard scaling, I have retained the data as well as got rid of outliers.

I did refer couple of posts (post1, post2) but couldn't get an idea

Is this what standardization does? Can anyone suggest me is there anyother way to handle outliers without dropping the records?

Can you help?


texts: Model with two targets(classification, regression)?

texts:
I have a dataset with features like Cylinders, Transmission, ExteriorColor, OdometerReading, StockType(New/Old), MakeModel, ModelYear, SoldYearMonth, CountSoldByMonth, Price, TopSelling.

Target column is 1 if a particular MakeModel is the top 5 selling vehicle in a month. So, given 60 months data I have to predict the next months top selling vehicles. I also have to predict the price go the vehicle. So I guess, I need to do both Classification and Regression .

MSRP   Cylinders   Transmission   ExteriorColor   OdometerReading   StockType     MakeModel            SoldYearMonth   CountSoldByMonth   Price   TopSelling
42098      8            MYC            Black           70348           Used    Chevrolet Tahoe            5/1/2019           10           39998     0
26390      4            MNK            Black            10             New      Buick Encore              5/1/2019           31           26390     1

and so on..
MSRP and Price are not always same.

How do I approach this problem?


texts: How to create group IDs for people in longitudinal data

texts:
I have a large data set which contains individuals and the address where they live. I want to create a group ID based off shared addresses (the working idea: people who share the same address can be considered as part of the same family/household). And from that household ID, my PI wants to investigate households/families migration overtime due to cost of living increases/decreases.

However, the difficulty is the dataset/analysis is longitudinal. So we have this data set spanning multiple consecutive time periods. We want to attach a household ID to each person, which they can be associated with at any point in the data. This has a couple issues.

People move in/out of households.
People start their own households with other people
The dataset doesn't keep track of people under 18 so when they come of age they pop up in the period of data where they turn 18
etc

The PI is flexible on their definition of households and we have so far come up with a couple of ideas.

Anchor households: create household IDs with the linked addresses at the beginning of the study, and having those individuals associated with this starting ID. Issue: individuals breaking/split off from their households resulting in
Captain/HeadofHouse: following one individual in the household at the start of the data, and grouping people who come into their household based on their assigned Captain ID. Issue: hard to make a distinction who gets assigned captain.
Multiple IDs: Assigning IDs at each period of data and then creating a graph for association. Best idea so far, but might make analysis a little more difficult.
Webbing: using component connection to attach each individual through the time periods. Weak connections eliminated (1-2 associations or less). E.g. I would be associated with each roommate my roommate has had. Issue: Super Messy (although might be fun to try and implement)

So I am looking for resources or suggestions on how to deal with a longitudinal grouping problem. So far I have looked into connected components, associative groups,and graph theory. Please, if you have any suggestions I would be very grateful. I am using Python so any library suggestions would be appreciated as well.

Please let me know if I need to explain anything further, or if there is some other information which would be helpful.


texts: Transfer learning from saved frozen graph

texts:
Using tensorflow graph (core tensor api's), I have trained a model and saved as a frozen graph.

As a part of transfer learning, I want to use these trained layers and extend my ML network with some new deep un-trained layers.

How it can be done in tensorflow using core graph api's?
Is it possible to integrate these frozen graph layers with Keras for further layers training?

What I did so far:

For 1:

Read the weights of trained layer.
Create a new architecture, which have layers similar to the trained layer and other new layers.
Set the weights of similar layer with the weights read in step 1 and set these layers trainable=False.
Training of the new architecture.

Not sure if it's a right way or can I have some easiest way of doing this?

For 2:
 Not sure


texts:
Building my machine learning model, I have a target variable that consists of dollar values ranging from $0.01$ to $1000000$+. I want to transform these dollar values in to a scale of 1 to 10 and then try and predict that level of 1 through 10.

What is the best approach to normalize my dollar values in to a scale of 1 to 10?


texts: with ML/DL model Is possible predict numbers of items required?

texts:
I have a dataset is regarding ambulance call data.
Data sample:
     v_type      district   gender  complaint      age  Month
0   Advanced    District 1  Male    Chest Pain      28  jan
1   Advanced    District 2  Male    Heart Problem   50  dec
2   General     District 3  Male    Cardiac Arrest  76  jun
3   Advanced    District 4  Male    Heart Problem   45  oct
4   General     District 5  Female  Cardiac Arrest  52  nov
5   Advanced    District 1  Male    Chest Pain      34  feb
6   Advanced    District 2  Male    Cardiac Arrest  44  jun
7   General     District 3  Female  Heart Problem   55  july
8   Advanced    District 4  Female  Heart Problem   86  may
9   General     District 5  Male    Heart Problem   65  aug
10  General     District 1  Male    Heart Problem   60  nov
11  Advanced    District 2  Male    Chest Pain      36  mar

In the data v_type (Vehicle type) we have Advanced (Advanced featured emergency vehicle) and General (Basic featured emergency vehicle).
How to predict how many advanced or general ambulances should arrange in a district for a particular month?
Example: If in a month (jan) and in district3 has huge complaint then predict and need to show 5 or 6.. Advanced vehicle type is required


texts:
This is an interview question

  How are neural nets related to Fourier transforms?

I could find papers that talk about methods to process the Discrete Fourier Transform (DFT) by a single-layer neural network with a linear transfer function. Is there some other correlation that I'm missing?


texts: what is the meaning of independent and identically distribution of samples in a dataset for neuralnetworks

texts:
random variables like heads or tail that generated by flipping a coin is independent because each time we tossing the result isn't depend on previous toss( in other words the function that generates random variable doesn't have memory). the example also is identically distributed because each toss result follows same distribution (binomial distribution).
but what is the meaning of IID in the case of images classification( like hand written digits dataset). each sample is independent from previous samples but how we ensure they generated with the same probability distribution(or what is the meaning of that in this specific case)?
or in the case of recurrent neural network after that we create over sample (according to the constrains of the problem) from the series. we can do shuffling to ensure the samples in dataset are independent but how about identically distribution of samples?


texts: How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

texts:
I was looking into the multi-label classification on the output layer of a Neural Network.
I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons.
So for example an Output would look like :

Output

1

0

-1

0

1

I used to take the tanh- activation function and partition the neuron into 3 ( y<-0.5, -0.5<y<0.5, y>0.5) to decide the class in each of those neurons after the prediction. See Figure below.

Question:
Are there any better alternatives on how to approach this ternary Output activation?
I stumbled upon this blog post regarding n-ary activation functions which I found very interesting.

I think the newly suggested would make the partitioning mentioned above much more meaningful!
Or do you think I should one-hot-encode my whole system such that a neuron can only have a value of 0 or 1 and then just use sigmoids activation with a threshold of 0.5?


texts: Amount of data needed and hypothesis for SVD

texts:
I was looking into the definition of SVD and trying to understand which are the conditions needed to be met in order to be able to use it.

Is there any hypothesis concerning the distribution of the data that I want to apply SVD on ?
Is the condition (number of data vectors > the dimension of the vectors space) strict ? Or is there always a way to decompose a dataset using SVD ? In the definition of SVD, all websites that I could find start by the same condition without fearther explanitions :

  given an $m\times n$ real matrix A with $n>m$, A can be written as ...


texts: Display a subset of classes in axes in ggplot

texts:
I have the following plot, is there any way in ggplot to display just the numbers 1 to 10 instead of all of them?
Numbers from 10 and after are not so important, but I need to display the ones before.
Thank you


texts: How can I include these 4 variables into my code in R

texts:
Here's a glimpse of the dataset I'm using.
SpeedDating dataset:

The data set I'm using is SpeedDating. My goal is to look at whether or people prefer to date each other of the same race. I was working with the variables DecisionM, DecisionF, RaceM and RaceF, wherein the Decision variables, 1 = yes and 0 = no to a second date, with my end goal being able to visualize this in some ggplot chart. My code so far is as shows:
test <- SpeedDating %>%
  filter(DecisionM, DecisionF) %>%
   group_by(RaceM == "Caucasian") %>%
  group_by(RaceF == "Caucasian") %>%
  summarize(count = n()) %>%
  filter(count == max(count))
test

I know I'm doing it wrong here but I don't know where to start, I thought about combining the two Race variables but I'm sure how to that right now either. Any help is appreciated. If you have any questions for me please ask.


texts: How to make forecast in LSTM with specific/defined feature values

texts:
I am analyzing the avocado dataset to predict the future prices of avocado depending on the region and type (organic/conventional).

I've trained my model  which seems to be working. The test results look good.

Here is what my data looks like after processing: the date (which I turned into the index column), total volume, type (label encoded to 1s and 0s), and 54 columns for the 54 regions that I one-hot encoded.

However, I'm not sure how to generate data for future prediction given specific X feature values. Basically I want the end user to provide a (future) date, a type, a region, and my model would predict the price of the avocado. For example, I want to predict the price of organic avocado for Albany in 3 months. I know that my model could forecast for a certain date, based on the prices of the past date. But how would it work when I assign the region and type feature to a specific value? Is it even feasible? Should I be using another model? I know that I could train a LSTM model for each of the 54 regions, but I'm sure there is a better way...

Sorry if this is a trivial question, I'm a beginner and I've been stuck on this for days and would really appreciate any help/guidance!


texts: How do I train Xgboost classifier for ECG Signal data?

texts:
I am testing https://www.physionet.org/challenge/2017/sources/ submission.

I like one of the submission code, which use Xgboost to train the classifier. Training data is in .mat file which can be converted to csv format.

In below code, I have pretrained model xgb.bin, using which I can test any input signal. But I Want to train the model using different data and create my own training model.

Here is the code which predicts class name for given input ecg file

def predict(data):

    #data = io.loadmat(path)['val'][0]

    from numpy import genfromtxt
    data = genfromtxt('testdata/val.csv', delimiter=',')

    features_noise = np.zeros((5, ))

    snr, rr_num, var, fr, fr2 = find_noise_features(data)
    features_noise[0] = snr
    features_noise[1] = rr_num
    features_noise[2] = var
    features_noise[3] = fr
    features_noise[4] = fr2
    features = extract_basic_features(data, 30000)
    features = np.hstack((features, features_noise.reshape(1, -1)))

    mean_ = np.array([15.96300284066109753667, 0.00412371298595770857, 38811.34497233365254942328,
                      0.48050717744965593115, 0.14397582347542958736])
    scale_ = np.array([4.22917401559752281770, 0.00093664880988427878, 62350.76443798459513345733,
                       0.15396567666240373873, 0.07085474966801086349])
    features_noise -= mean_
    features_noise /= scale_

    prediction = 0
    if features_noise[0] < -2.9:
        prediction = 3
    if features_noise[2] > 6.0:
        prediction = 3
    if features_noise[3] > 3.0:
        prediction = 3
    if features_noise[4] < -2.0:
        prediction = 3

    bst = xgb.Booster({'nthread': 4})
    bst.load_model("xgb.bin")

    dfeatures = xgb.DMatrix(features)
    prediction_prob = bst.predict(dfeatures)
    prediction = np.argmax(prediction_prob)

    return prediction

def run(data):
    prediction = predict(data)
    print(prediction)

I want to train this classifer using my own dataset. Above model is trained on https://www.physionet.org/challenge/2017/training2017.zip dataset.

Looking at above code, do you have any clue how can I train the model using Xgboost.

Here is my thought steps

Calculate features for all training signal in zip file
Calculate feature_noise as it is in above code for all signals
Calculate dfeatures features for each signals (Should I calculate it for each signals or for all signal together)
Train dfeatures for all signals using Xgboost (How?)
Store the xgboost model

Is this correct steps to do?


texts: Train a point rather than a bounding box for object detection

texts: In the graph below, object detection is done through a point that points to the right place rather than a bounding box (e.g., using faster RCNN). What method enables to  train such a point?


texts: I am using R. I have a data frame with "year , price, mileage" columns. I want to group the df by year first and then sort each group by mileage. How can I do this?


texts:
Is there a way to test out simple filters before committing to coding them?

Like if I want to estimate the feasibility of recognizing some features from images. Or to estimate the effort/sophistication of required methods.

Then can I try out something in Photoshop or something in order to discover "where to look for"? Prior to coding?


texts: How to classify and cluster this time series data

texts:
I have post already the question few months ago about my project that I'm starting to work on. This post can be see here:
Human activity recognition using smartphone data set problem

Now, I know this is based around multivariate time series analysis and tasks are to classify and cluster the data. I have gathered some materials (e-books, tutorials etc.) on this but still can't see a more detailed picture of how even I should start. Here's the tutorial that looks like it might be helpful but the thing is my data looks differently and I'm not really sure if this can be applied to my work.

http://little-book-of-r-for-multivariate-analysis.readthedocs.org/en/latest/src/multivariateanalysis.html#scatterplots-of-the-principal-components

So basically, my questions are:

How I can start on some very basic analysis? How to read data so it any meaning for me.
Any tips and advises will be much appreciated!
Note: I'm just the beginner in data science.


texts: What features are extracted from pre-trained model of CNN Keras?

texts:
I would like to use the CNN pre-trained model in feature extraction but I don't know what features are extracted from that.
Please let me know!


texts: What kind of research I can do with my data set?

texts:
I have a data containing 200K ads (sale and rent property in the United Arab Emirates, or UAE). Here is a sample:

190459 obs. of  29 variables:
 $ id                  : chr  "tnydu.biz/DafdVO" "tnydu.biz/DafloP" "tnydu.biz/Dafmvy" "tnydu.biz/Dafuy5" ...
 $ AdType              : chr  "sale" "sale" "sale" "sale" ...
 $ AgencyFees          : num  NA NA NA NA NA NA NA NA NA NA ...
 $ Amenities           : chr  "CentralA/C&Heating,SharedPool,SharedGym,Security,BuiltinWardrobes,ViewofWater,ViewofLandmark" "Study,CentralA/C&Heating,Balcony,Security,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchenAppliances,Vi"| __truncated__ "CentralA/C&Heating,SharedSpa,Security,ConciergeService,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchen"| __truncated__ "CentralA/C&Heating,Balcony,Security,CoveredParking,BuiltinWardrobes,BuiltinKitchenAppliances,ViewofLandmark" ...
 $ AnnualCommunityFee  : chr  "9000" NA NA NA ...
 $ AreaDescription     : chr  "\r\n            \r\n                \r\n                    1.9 km from Najmat Reem Marina\r\n                \r\n            \"| __truncated__ "\r\n            \r\n                \r\n                    0.3 km from The Dubai Mall\r\n                \r\n            \r\n "| __truncated__ "\r\n            \r\n                \r\n            \r\n        " "\r\n            \r\n                \r\n                    0.7 km from The Dubai Mall\r\n                \r\n            \r\n "| __truncated__ ...
 $ Bathrooms           : num  1 NA NA NA 5 3 NA NA 3 3 ...
 $ Bedrooms            : num  0 3 0 1 3 2 2 2 3 3 ...
 $ Building            : chr  "HydraAvenueTowers" "BurjVista1" "TheAddressDubaiMall" "TheAddressDowntown" ...
 $ City                : chr  "AbuDhabi" "Dubai" "Dubai" "Dubai" ...
 $ PublishDate         : POSIXct, format: "2015-10-30" "2015-11-11" "2015-11-13" "2015-11-09" ...
 $ DealerCode          : chr  "599942" "604296" "604296" "604296" ...
 $ DealerName          : chr  "STARWOOD PROPERTIES BROKER" "BLUE PALACE REAL ESTATE BROKERS" "BLUE PALACE REAL ESTATE BROKERS" "BLUE PALACE REAL ESTATE BROKERS" ...
 $ Developer           : chr  "HydraProperties" "EMAAR" "EMAAR" NA ...
 $ Furnished           : chr  NA NA NA NA ...
 $ ListedBy            : chr  "Agent" "Agent" "Agent" "Agent" ...
 $ Location            : chr  "City of Lights, Tamouh Marina Square" "Downtown Dubai, Dubai" "Downtown, Cairo" "Downtown Dubai, Dubai" ...
 $ LocationGPSLatitude : num  24.5 25.2 30 25.2 25.1 ...
 $ LocationGPSLongitude: num  54.4 55.3 31.3 55.3 55.1 ...
 $ Price               : num  900000 3822888 2150000 3200000 5500000 ...
 $ PriceSqFt           : num  1129 2185 3909 3422 2431 ...
 $ PropertyReference   : chr  NA "BP9801" "DT-K20" " BP8958" ...
 $ PropertyType        : chr  "apartment" "apartment" "apartment" "apartment" ...
 $ ReadyBy             : POSIXct, format: "2015-04-30" NA NA NA ...
 $ RentIsPaid          : chr  NA NA NA NA ...
 $ ShortLink           : chr  "tnydu.biz/DafdVO" "tnydu.biz/DafloP" "tnydu.biz/Dafmvy" "tnydu.biz/Dafuy5" ...
 $ Size                : num  797 1749 550 935 2262 ...
 $ TotalClosingFee     : chr  NA NA NA NA ...
 $ VirtualView         : chr  NA NA NA NA ...

I'm looking for help figuring out what kind of questions I can answer using these data.
For example,
1. What the mean price per year for a 1-bedroom apartment to rent in different areas of Dubai?
2. How are supplies for apartments and villas distributed on the city map?
3. Does the number of bathrooms influence on sale/rent price?
4. Where is the best place for investing to get more profit from apartment cost and rent revenue?

I have already done some of this (plotted the map with mean price for studio in Dubai in thousands AED/year, 1USD = 3.66AED):

From your experience, what other questions can I ask? What I have to be aware in this data set? I can share my data set if anyone want to examine it (it is a 12mb zip).


texts: Which book is the best for introduction to analysis social network using python3

texts:
I am a beginner studying social network analysis.
I installed python 3 just 2 weeks ago.
There are a lot of books for python and social network analysis.

I couldn't choose one of them.
I found one book named "Mining the Social Web (Analyzing Data from Facebook, Twitter, Linkedln, and Other Social Media Sites)" written by Matthhew A. Russell.
This book looks very interesting and fits in my purpose, but it is based on python 2.

Is there any good books with python 3? I usually use Twitter, Facebook, or Blog data.

In addition, could you recommend any good book for nodeXL and UCINET?


texts:
..., to create proper consecutive batches, where the nth input sequence in a batch starts off exactly where the nth input sequence ended in the previous batch.
  Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (Kindle Locations 12018-12020). O'Reilly Media. Kindle Edition.

So data for a stateful RNN looks like:

Why not like the below?

Why does stateful use batches like the bottom one? As far as i know, it seems like the bottom one's hidden state has longer memory. (e.g. seq 4, seq 1-2-3, but the top one, seq 4, just seq 3)

Maybe some reasons is there?


texts:
Can anybody explain me what are the effects on the model if we have sparse data in our dataset. And also how to deal these sparse matrices ?

Thank you.


texts: In CycleGAN are there two different generators and two different discriminators?

texts: I am trying to assimilate the contents of this paper. I have a confusion about how many different networks are there in the architecture of CycleGAN. To my understanding, the concept of cycle means that there is only one generator and one discriminator which changes roles (Discriminator as generator; and Generator as Discriminator) depending on what loss we want to calculate (whether Y -> X or X -> Y). However, my friend I have a confusion and we have been debating about this without a consensus. He says that there are totally four different networks. Two generators G1 and G2; and two discriminators D1 and D2. What is the truth ?


texts:
I have a vector and want to detect outliers in it.

I need an outlier detection method (a non-parametric method) which can detect red points as outliers.

Edit:
I have a lot of vectors like this. The pattern is similar to this but the values are different. I mean we can see the height since the values are not constant but overall the patterns of outliers against the normal data are like this. This means in one place they beak the continuity of the blues points and go in different directions. Here is another figure.

Thanks in advance.


texts: Sensitivity analysis of a machine learning model

texts:
Let’s say I have a set of input variables (A, B, C and D) and I predict a target (y) using a machine learning model (XGBRegressor in my case) with a reasonable performance (5% relative error on test set).

from sklearn.datasets import make_regression
import pandas as pd
from xgboost import XGBRegressor

X, y = make_regression(n_samples=500, n_features=4, n_informative=2, noise=0.3)
X = pd.DataFrame(X, columns=['A', 'B', 'C', 'D'])

model = XGBRegressor()
model.fit(X, y)

Now, I want to do some kind of sensitivity analysis on this model by answering two questions:

What is the impact of a 5% independent increase in variables A, B and C (not D) on the target variable?
From variables A, B, C and D; which combination of values of A, B and C (without touching D) increases the target y value by 10, minimizing the sum of A, B and C.

I have already answered question one (see this gist).
However, how can question 2 be coded? I imagine that this implies an optimization problem.