texts: Is there a machine learning model suited well for longitudinal data?

texts:
I have a fairly large (>100K rows) dataset with multiple (daily) measurements per individual, for a few thousand individuals. The number of measurements per individual vary, and there are many null values (that is, one row may have missing values for certain variables/measurements, but not for all). I also have a daily outcome (extrapolated, but let's assume it's fair to do so, so there is a binary outcome for each day when measurements are taken).

My question goal is to model the outcome, such that I can predict daily outcomes for new individuals.

My background is in research, and I am familiar with some statistics and ML, and overall still fairly new to data science. I am wondering if there are any particular known ML algorithms that can be used to model such data. I am cautious about using logistic regression from something like python's scikit learn because the observations are not independent (they are highly correlated on an individual level). From my knowledge, these kind of data are well-suited for a mixed effects logistic regression or longitudinal logistic regression. However, I haven't been able to find any widely used ML algorithms for it, and I would like to pursue an ML approach rather than fitting a statistical model using something like lme4 package in R.

Could someone recommend an available ML algorithm to model such data?

PS: I did some research and found a few research articles on the topic but nothing widely used or clearly implemented. The structure of the data I am working with strikes me as very common, so I thought I'd ask.


texts:
I am reading the book "Data Science for Business" by Foster Provost & Tom Fawcett. Only a fourth of the way through.

I am unclear about the concept of Objective Function. I will nevertheless take a stab and would like to seek feedback.

Say that we decided to use standard linear regression for a given problem (assume we have training data). Is the goal of objective function to determine the best parameters (weights) for this training data?

Thanks in advance!


texts: An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked like they need to be checked in ARIMA model.Do we have to check for it in LSTM??


texts:
I am attempting to determine if a given phrase (or a few words) is present in a relatively large text. For example:

Text:

  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce sed
  tristique purus, id lobortis justo. Vestibulum ante ipsum primis in
  faucibus orci luctus et ultrices posuere cubilia Curae; Cras vitae
  neque non nibh elementum malesuada convallis et nunc. Nam vel tellus
  nec nunc dictum dignissim eu ut felis. In Tony Starkeget efficitur
  nunc. Cras ultrices turpis est, ac eleifend leo congue at. Donec lorem
  diam, mattis sed sollicitudin ac, tincidunt eu sem. Curabitur vel
  euismod lectus, sit amet tempor massa. Vivamus ut dictum nisl. Aliquam
  et urna sit amet urna hendrerit tincidunt in a mauris. Class aptent
  taciti sociosqu ad litora torquent per conubia nostra, per inceptos
  himenaeos. Maecenas vel justo metus. Sed gravida egestas velit,
  porttitor pulvinar justo hendrerit et.

Phrase/Words to match in the text above:

tony.stark
t.stark
stark_tony
starktony

The intention here is to infer if the person(Tony Stark) is being mentioned in a block of text.

I have read up on some fuzzy word match algorithms like Levenshtein and Soundex and also tested them in the above application but they appear to be useful for one word to one word match, not in the above application where various permutations of Tony stark is possible in both the pattern and text.

Would anyone be able to advice which fuzzy word matching algorithms would be ideal for this application, and perhaps share resources and sample code for its implementation.

Thanks.


texts:
If I parse following sentence:

  He is playing cricket in ground with grass.

with stanford parser, the result is:

(ROOT
  (S
    (NP (PRP He))
    (VP (VBZ is)
      (VP (VBG playing)
        (NP
          (NP (NN cricket))
          (PP (IN in)
            (NP (NN ground))))
        (PP (IN with)
          (NP (NN grass)))))
    (. .)))

Is there any parser who can correct the result on the basis of probability as the probability of appearing grass with ground is higher than cricket?

Or parser which do chunking before parsing like this:

  He is playing cricket |in ground| |with grass|.

and calculate probability with all combinations before generating the parse tree.

He is playing cricket |in ground| |with grass|.
He is playing cricket |with grass| |in ground|.


texts: Feature engineering while using neural networks

texts:
When doing Data Science, we concentrate on feature engineering first - check on correlations, imputations, tranformations etc.

Do we have to follow the same steps while feeding the inputs to a neural net? Or what should be the steps taken?


texts: alternatives to pyspark to query a csv file locally (no distributed computation needed)

texts:
I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as matplotlib to plot data. I know there are some workarounds to do it with pyspark but I found it very complicated.

My question is: is there any alternative to pyspark, possibly python-native, that allows to query a csv file allowing to use python libraries?


texts: Dense Layer vs convolutional layer - when to use them and how

texts: I know what's the difference between the two, however I am a little confused on to use them. I have also seen some models that have a mix of both. what's the logic behind it? or is it only random things?


texts:
I am kind of new to the data mining subject but i need help to choose a learning algorithm for my application:
The problem: identifying that a certain curve or data set belongs to a certain fault in a Component.

My training data should be like this:

Motor Current Values:
[0.5,0.6,...,0.4] -> Fault in Komponent 1

[0.2,0.3,...,0.4] -> Fault in Komponent 3

[1,0.7,...,0.4] -> Fault in Komponent 2

.
.
.
[0.8,0.7,...,0.3] -> Fault in Komponent 3

And i was wondering if i should using a cluster analysis (k-means and save centroid centers then compute the distance of each new entry then give a fuzzy estimation to where which cluster my data might belong to).

Decision Tree algorithm(entropy of the values and stuff), Or should i calculate a distance between the Nominal data (Healthy(No fault) Curve) and the faulty data and play on a more simple basic decision tree with thresholds ?

Proceed with a peak analysis and count the number of peaks within my data and add it to my learning algorithm.

And when should i do a data pre-processing like normalization ?

This is an example of my entry parameters : [Speed(mm/s), Motor Current Values(Ampere), DistanceToNominalState(Ampere)]

Here is what my data looks like.

Any suggestions ?


texts: What to infer from a given scatter plot?

texts:
I am quite new to data science and I have been given the task to find relationship between two entities for a given data.This is the scatter plot graph I made

It is a time series data.Here x axis is the mean and y axis is the mse (mean square error).

What I can infer from this graph is the mean and mse don't really go along with each other.In other words when mean and mse doesn't share any strong relationship?Is my understanding of the graph correct?Is there anything else that can be found?


texts: Different results with same input in request/response if input by one row or several rows in Azure Machine Learning Studio

texts:
I created a classifier by trained SVM and PCA as well as normalization transformation. And use Request/Response Excel to test.
And if I only input for one row, all the PCA would be "0"

but if I put more than one row together then the result would change and both behavior doesn't make sense.

I read some other questions below and I wonder if something wrong in "Normalization" calculating that it didn't count by referring "Applying transformation" but simply normalizing by data input.
https://social.msdn.microsoft.com/Forums/azure/en-US/bb800d7e-2fbf-4874-a525-aae9ace5df4f/azure-ml-webservice-same-sample-file-but-different-predictions?forum=MachineLearning
https://social.msdn.microsoft.com/Forums/en-US/21938a5a-ae9c-4559-82f0-95889b359206/normalization-and-predictive-web-service?forum=MachineLearning


texts: Evaluate Topic Modelling on synthetic data

texts:
I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics distributions. I believe it is a good way to test unsupervised methods. The issue is that I completely fail to find the correct number of topics.
I am using NMF and LDA from gensim with c_v and u_mass coherence scores. It should be easy game to find the optimal number of topics so I do not use hyperparameters for tuning. I believe the issue is deeper than that.
The code is available here.
It is well documented. The script to run is 'myscript.py'. It uses functions in 'mymodule.py'. You just need to install gensim and pandas (see requirements.txt if needed).
Any thoughts?


texts: Each time the result of XGBoost is different

texts:
I'm running xgboost and I fix the seed number
but the result is different for each time when I rerun the xgboost with same dataset

xgb = XGBClassifier(
     bagging_fraction= 0.8,
     boosting= 'gbdt',
    colsample_bytree= 0.7,
    feature_fraction= 0.9,
    learning_rate= 0.05,
    max_bin= 32,
    max_depth= 10,
    min_child_weight= 11,
    missing= -999,
    n_estimators= 400,
    nthread= 4,
    num_leaves= 100,
    predictor= 'gpu_predictor',
    seed= 1000,
    silent= 1,
    subsample= 0.8,
    tree_method= 'gpu_hist',
    verbose= True
)

This is my parameter and

# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split

# This is explicitly used for undersampling.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
kf = StratifiedKFold(n_splits=5, shuffle=True)
# partially based on https://www.kaggle.com/c0conuts/xgb-k-folds-fastai-pca
predicts = []
for train_index, test_index in kf.split(X_train, y_train):
    print("###")
    X_train, X_val = X.iloc[train_index], X.iloc[test_index]
    y_train, y_val = y.iloc[train_index], y.iloc[test_index]
    xgb.fit(X_train, y_train, eval_set=[(X_val, y_val)],
            early_stopping_rounds=30)
    predicts.append(xgb.predict(X_test))


texts: How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance

texts: I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to my mounted EBS volume and why do I not see the contents of my EBS volume in the r-studio files area (bottom right side? ) . Also, I had a file in an s3 bucket. I passed this command to bring it from s3 to my ebs volume: s3cmd get s3://data-analysis/input-data/filename.csv . I assume this command downloads the file from s3 into the ebs volume. But I can't find it in RStudio Server! I have scoured the internet looking for help on this but not able to solve my problem.


texts: How is vector A converted to single value scalar in Andrew Ng's course?

texts:
In Andrew Ng's deep learning course on Coursera, how is a single scalar value obtained from a flattened image (feature vector)?
First there is $w.T$ of shape $(1, n_X)$ which is multiplied by $X$ of shape $(n_X, 400)$, so by the laws of linear algebra, the remaining vector is of shape $(1, 400)$. This is then passed through Sigmoid to form vector A, and the shape remains at $(1, 400)$. How is this vector then converted to a binary scalar value ($0$ or $1$) for prediction $y$-hat?


texts: Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

texts:
I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated.

Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model:

Choose loss based on your problem at hand. I use default one - deviance
Pick n_estimators as large as (computationally) possible (e.g. 600).
Tune max_depth, learning_rate, min_samples_leaf, and max_features via
grid search.
Increase n_estimators even more and tune learning_rate
again holding the other parameters fixed.

Scikit-learn provides a convenient API for hyperparameter tuning and grid search.

Let's look at python code the code:

train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(new_features, target, random_state=42,train_size=0.1 )
gb_grid_params = {'learning_rate': [0.1, 0.05, 0.02, 0.01],
              'max_depth': [4, 6, 8],
              'min_samples_leaf': [20, 50,100,150],
              #'max_features': [1.0, 0.3, 0.1]
              }
print(gb_grid_params)

gb_gs = GradientBoostingClassifier(n_estimators = 600)

clf = grid_search.GridSearchCV(gb_gs,
                               gb_grid_params,
                               cv=2,
                               scoring='roc_auc',
                               verbose = 3,
                               n_jobs=10);
clf.fit(train_gs_X, train_gs_Y);

When I obtain the parameters values I cross validate modes to check overfitting.

scores = cross_validation.cross_val_score(gb,
                                          all_data, target,
                                          scoring="roc_auc",
                                          n_jobs=6,
                                          cv=3);
"Accuracy: %0.5f (+/- %0.5f)"%(scores.mean(), scores.std())

Is my approach sufficient?
Is it a correct way to tune Boosted Decision Trees hyperparameters?
Do you have any idea how can I improve my tuning procedure?
I know there exist method like Gaussian Process, which is faster, I mean can find the optimal hyperparameters configuration in less steps but it is not issue. I want to increase the performance measured as a ROC auc.

The second issue is how to deal with unbalanced trees?

I have two ideas:

Use same number of signal and background (or whatever you call it) events. The problem with this approach is to skipp huge number of possibly useful events.
Use DecisionTree parameters class_weight

See my code below:

signal_event_no = counts = data[target == 1].count()[0]
background_event_no = counts = data[target == 0].count()[0]
ratio_background_to_signal = float(background_event_no)/signal_event_no
ratio_background_to_signal = numpy.round(ratio_background_to_signal, 3)
train_X, test_X, train_Y, test_Y = train_test_split(new_features, target, random_state=42,train_size=0.5 )
gb6 = GradientBoostingClassifier( n_estimators=400, learning_rate=0.2,
   class_weight=ratio_background_to_signal, max_depth=6)

Any other ideas?

The last but not the least. How can I change the hyperparameters tuning procedure in respect to xgboost? What hyperparrameters should I take care? Is it the same set as for Gradient Boosted classifier?


texts: Factor Analysis of Mixed Data : Correlation of categorical variables togeather

texts:
I have a dataset containing mixed-type data (categorical & numerical) and want to explore the dataset further. Following this post (https://nextjournal.com/pc-methods/calculate-pc-mixed-data) I looked at package called prince which does Factor Analysis for Mixed Data in python. I am able to find the correlation between the features and its principal components using the package. The correlations for the numerical components are fine. However the correlations for the categorical variables given independently.

What I mean by this is,
If we consider a numerical feature called E2 Fruity Red with values (2,4,5,7,3,3). I would end up with a correlation for component 1 as 0.874. But if I had a categorical feature called E1 Fruitty with values ('A','A','B','B','B','B') then I end up with two correlation values called E1_Fruitty_A : -0.59, E1_Fruitty_B : 0.59. for component 1

I am wondering if there is a way to get just one correlation value for the categorical variables? (one correlation value for all the categories in the feature).

Also note that in this case even though I end up with same number (-0.59 & 0.59) one positive and one negative, when performing FAMD in other datasets with has four categories in a categorical feature, I ended up with different numbers.

Here is an example from the prince website,

https://github.com/MaxHalford/prince

from prince import FAMD

df = pd.DataFrame(
data = [
    ['A','A','A',2,5,7,6,3,6,7],
    ['A','A','A',4,4,4,2,4,4,3],
    ['B','A','B',5,2,1,1,7,1,1],
    ['B','A','B',7,2,1,2,2,2,2],
    ['B','B','B',3,5,6,5,2,6,6],
    ['B','B','A',3,5,4,5,1,7,5]
    ],
    columns = ['E1 fruity','E1 woody','E1 cofee','E2 red fruit','E2 roasted','E2 vanillin','E2 woody','E3 fruity','E3 butter','E3 woody'],
    index = ['Wine {}'.format(i+1) for i in range(6)]
)

df['oak type'] = [1,2,2,2,1,1]

#instatitate famd(factor analysis with mixed data) object

famd = FAMD(n_components = 2,random_state = 42)
famd.fit(df.drop('oak type',axis = 'columns'))

# Compute correlations
print('correlations : {}'.format(famd.column_correlations(df)))

DataFrame :

output of correlation :

Note how the categorical columns E1_fruity, E1_woody, E1_cofee are sperated into categories in the correlation E1_fruity_A, E1_fruity_B and so on...in correlations


texts: Performance of model in production varying greatly from train-test data

texts:
I was wondering if anyone has any advice on where to start digging for this problem. I have a model which has gone through development and all train/cv/test data sets now perform above 95% both for accuracy and F-Score. The total development data set is around 60k samples, with a 2/3 split for positive and negative samples.  These samples are based on extracts for the months of Jan to Nov of last year. Final test results were:

Precision: 0.9751 Recall: 0.9320 Accuracy 0.9693 F score 0.9531

However, the first runs in production showed a very high precision:95%+ but a very low recall:~50%. Accuracy = 48%, FScore = 68%.

Any thoughts from the group on this, where to look, potential causes.  We will run this over the next couple of months, as we may have exceptional variations due to the Xmas period, but we were surprised. Any help appreciated. Thanks


texts: Designing a network for multiclass regression

texts:
I'd like to model a continuous conditional probability distribution for two classes on a given data set. eg the height of men and women from a set of inputs. I can train a regression model (DNN, CNN, XGB etc.) to get the height of all my data with ok accuracy. I can train a classifier model (DNN, CNN, XGB) to separate the two populations. But doing both is proving to an issue.
I tried the following:

Classify, partition on the predicted class, then regress the two groups.
Add height and sex as the two targets of the model.
add male height and female height as two targets of the model, and set 0 otherwise in training.

All give pretty poor performance:

high training bias (and not great performance)
regression and classification typically have different objectives in order to perform well. One always fails in my experience.
The third one distorts the regression distribution at low values (due to the 0's)

Any tips for approaches to improve? I'm typically using Keras to build simple models but open to try new things if there are examples I can follow.


texts: TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'(Stateful LSTM Keras)

texts:
So I have a trained LSTM model with which I am trying to predict future values. The model is stateful as seen below

model = Sequential()
model.add(GRU(units=50,batch_size=1, input_shape=(None, 1),stateful=True, return_sequences=True))
model.add(Activation('tanh'))
model.add(GRU(units=50,stateful=True,return_sequences=True))
model.add(Dropout(.1))
model.add(GRU(1, return_sequences=True,stateful=True))
model.add(Lambda(lambda x: x * 1.0012))
model.load_weights("weights.h5")
model.compile(loss='mse', optimizer='adam')

When I try to predict on the entire data and then pass that through in a loop to predict future vals I get the title error. This ONLY occurs if steps is set.

If I set steps with a batch_size it tells me to set batch_size to None and if I do that I get the error above. Very strange. What's going on here? It also will work if I do not set steps and set batch to any int divisible by the # of samples.

Below is the prediction code:

model.reset_states()
predictions = model.predict(entire_data,batch_size=1)
futureElement = predictions[:,-1:,:]
futureElements = []
forecast1 = []
futureElements.append(futureElement)
for i in range(time):
    futureElement = model.predict(futureElement,batch_size=None,steps=60)
    forecast1.append(futureElement)


texts: What undergraduate degree should I take to get into data science?

texts:
I'm going to do an undergraduate degree next year. Data Science major is there but I was wondering if a statistics major with a minor in data science would be more valuable? Which degree would be more in demand?
I am also planning on doing as many internships as possible to gain practical skills and experience. I am also contemplating the value of a masters degree, and whether it is worth the time and money, or if an undergrad+internships will suffice. I already know Python and I will learn R and SQL as well. Whether this is in university or otherwise (or both).
I will want to specialise in data analysis using statistical analysis paired with artificial intelligence.
EDIT: I’m in Australia and looking to go to the University of Melbourne (I’m pretty sure its #1 in Australia for computer science and math/statistics)


texts: How to combine sparse text features with user smile for sentiment classification?

texts:
I am trying to perform sentiment classification task where I have some text and some information about whether the user smiled or not.
Now when I use count-vectorizer to convert my text to feature set (Bag of Words) approach it results in more than 5000 features. When I add smile to these 5000 features, total number of features become 5001. However, I do not see any improvement in the classification accuracy. I think the 5000 sparse features dominate the one and only smile feature. Hence no effect on the classification. Can anyone recommend, what is the good way to merge textual feature with smile, prosody or other features?


texts: How to integrate google cloud with dropbox and jupyter notebook using tensorflow

texts:
So I opened up a google cloud account and have access to global and local (us east 1) resources (Compute Engine API , NVIDIA K80 GPUs) and connected it to my dropbox.

Next, I followed this youtube video to try to connect it to my jupyter notebook.

The code to be entered into the google cloud platform is as follows:

sudo apt-get update
sudo apt-get - -assume-yes upgrade
sudo apt-get - -assume-yes install software-properties-common
sudo apt-get install python-setuptools python-dev build-essentials
sudo easy_install pip
sudo pip install jupyter
jupyter notebook - - generate-config
sudo nano ~/.jupyter/jupyter_notebook_config.py

Insert these following lines under #configuration file for jupyter notebook
c=get_config()

c.NotebookApp.ip = ‘*’
c.NotebookApp.open_browser = False (or True)
c.NotebookApp.port=1000

ctrl+alt? enter ctrl+x?
from 3.5.0.tar.gz you write tmux0.tar.gz
ctrl+b shift+double_course (to get two diff windows) or exit to go out of the double

jupter notebook password
jupyter notebook –no-browser –port=1000

And then you go to your browser and write external ip:port (ex. 35.197.7.72:1000)
At the top window, upload your environment

BUT, before running the following line:

sudo nano ~/.jupyter/jupter_notebook_config.py

It asks me if I want to overwrite .../jupyter/jupyte_notebook_config.py with default config, and with both scenarios (when entering yes and when entering no) after I run the line it says

  sudo: nano: command not found

I even tried to pip install nano and it worked (the installation) but still the same response - anyone know what I should do?


texts:
Following this paper https://www.aclweb.org/anthology/D17-1079, I'm stuck with Paragraph 2 "Baseline Segmentation Model". In this section they basically explain how they feed the chinese word embeddings into the bi-LSTM model. In particular they use this formula for $e^f_i$ (forward word representation):

$$ e^f_i = concat_1(e_{w_i}, e_{w_{i−1}w_i}),
    = tanh(W_1[e_{w_i}; e_{w_{i−1}w_i}]) $$

where $w_i$ is the $i$-th character in a sentence $\{w_1,w_2,\dots,w_N\}$, where $N$ is the sentence length. We have a corresponding embedding $e_{w_i}$ and $e_{w_{i−1}w_i}$ for each character unigram $w_i$ and character bigram $w_{i−1}w_i$.

I don't really get what they are doing here and how can I replicate it with Keras?


texts: How would you teach a machine how to count?

texts:
var digits = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"];
var numbers = [digits[0], digits[1]...digits[9], digits[1] + digits[0], digits[1] + digits[1]...];

What would you use to build a model that can predictNextNumber(19)=20 by understanding how counting works?


texts: In RNNs why do networks always use the last output vs the last input?

texts:
All the descriptions of RNNs introduce some equation like:

$\ h_t = f_W(h_{t-1},x_t)$

and I'm wondering why we don't just go straight to the "source", ie the last input like :

$\ h_t = f_W(x_{t-1},x_t)$

The only thing I can maybe think of is maybe the RNN can store some sort of "state" in ht-1

But then, along the same vein, what would happen in an LSTM architecture if we used previous inputs vs previous outputs? It seems to me that the presence of the memory functionality should be able to avoid this potential problem (if it even is one)


texts: Forecasting using Boosting methods on Non-stationary Time Series data

texts:
Theoretical Noob question -
Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set and then add seasonality/trend components while forecasting?
Thanks in advance.


texts: TF-IDF not a strong measure in this senario?

texts:
I am dealing with a data where I have only two documents and there are some words which are present in both. Now Term Frequencies (tf) of these words are very high for respective single document than the other. For e.g.

Word1 is present in Documents D1 and D2, and
tf(Word1,D1) = 1000
tf(Word1,D2) = 3
But since Word1 is present in both the documents
IDF(Word1) = 0
TF-IDF(Word1,d) = 0 for all d belonging to {D1,D2}

So inspite having a very powerful presence in a single document, TF-IDF score will always be 0. One solution I could think is to take word1 as absent if tf(Word1) < threshold. However I still don't feel this is good enough as the Word present in only one single document is given IDF score of only 0.5. I feel like TF-IDF is not a good measurement when number of documents are very low. Any suggestions here?


texts: Interpreting Loss in neural network: Neural network train loss gradually tappers and validation loss never reaches a minima

texts:
Unable to improve the network validation loss. Is it overfitting/underfitting. How can I get a better validation loss?.The code is below

def create_model(lr=0.05):
    #tf.random.set_seed(1)

    tf.keras.backend.clear_session()
    gc.collect()

    # Dense input
    dense_input = Input(shape=(len(dense_cols), ), name='dense1')

    # Embedding input
    #Turns positive integers (indexes) into dense vectors of fixed size.
    wday_input = Input(shape=(1,), name='wday')
    month_input = Input(shape=(1,), name='month')

    event_type_1_input = Input(shape=(1,), name='event_type_1')
    item_id_input = Input(shape=(1,), name='item_id')
    dept_id_input = Input(shape=(1,), name='dept_id')
    store_id_input = Input(shape=(1,), name='store_id')
    cat_id_input = Input(shape=(1,), name='cat_id')
    state_id_input = Input(shape=(1,), name='state_id')

    wday_emb = Flatten()(Embedding(7, 3)(wday_input))
    month_emb = Flatten()(Embedding(12, 2)(month_input))

    event_type_1_emb = Flatten()(Embedding(5, 1)(event_type_1_input))

    item_id_emb = Flatten()(Embedding(3049, 3)(item_id_input))
    dept_id_emb = Flatten()(Embedding(7, 1)(dept_id_input))
    store_id_emb = Flatten()(Embedding(10, 1)(store_id_input))
    cat_id_emb = Flatten()(Embedding(3, 1)(cat_id_input))
    state_id_emb = Flatten()(Embedding(3, 1)(state_id_input))

    # Combine dense and embedding parts and add dense layers. Exit on linear scale.
    x1 = concatenate([dense_input,
                     event_type_1_emb,
                     wday_emb ,
                     month_emb,
                     item_id_emb, dept_id_emb, store_id_emb,
                     cat_id_emb, state_id_emb])

    x = BatchNormalization()(x1)

    x = Dense(7142, activation=None,kernel_initializer='lecun_normal',kernel_regularizer= regularizers.l1_l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = AlphaDropout(0.30)(x)

    x = Dense(714, activation=None,kernel_initializer='lecun_normal',kernel_regularizer = regularizers.l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = AlphaDropout(0.3)(x)
    x = Dense(34, activation = None,kernel_initializer='lecun_normal',kernel_regularizer = regularizers.l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = Add()([x,x1])
    outputs = Dense(1, activation="softplus", name='output',kernel_regularizer = regularizers.l2(0.001))(x)

    inputs = {"dense1": dense_input, "wday": wday_input, "month": month_input,# "year": year_input,

             "event_type_1": event_type_1_input,

              "item_id": item_id_input, "dept_id": dept_id_input, "store_id": store_id_input,
              "cat_id": cat_id_input, "state_id": state_id_input}

    # Connect input and output
    model = Model(inputs, outputs)

    model.compile(loss=keras.losses.mean_squared_error,
                  metrics=["mse","mape","mae"],
                  #optimizer=keras.optimizers.SGD(learning_rate=lr_schedule))
                  optimizer=keras.optimizers.RMSprop(learning_rate=lr))
    return model
```


texts: Why does a simple query succeed in mysql but fail in druid?

texts:
I have a mysql database table with only one column called __time which stores a time stamp.  The following query works perfectly and will return the __time of a record and the __time of the record just before it:
SELECT a1.*,
(SELECT MAX(a2.__time) FROM report AS a2 WHERE a2.__time < a1.__time) AS last_time
FROM report AS a1

I imported my mysql data into a druid table with the exact same table name, columns and data.  I then ran the same query, but druid gives me the error:
Error: Unknown exception

Cannot build plan for query: SELECT a1.*, (SELECT MAX(a2.__time) FROM report AS a2 WHERE a2.__time < a1.__time) AS last_time FROM report AS a1

org.apache.druid.java.util.common.ISE

My questions are:

where do I go to get more information on what exception actually happened?
what did I do wrong with my query?


texts: use .apply() function to change values to a column of the dataframe

texts:
I have a dataframe which is the following:

and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string inside each list. What I am doing is the following:

dataFrame['opcodes'] = dataFrame['instructions'].apply(instruction.split()[0] for instruction in dataFrame['instructions'])

but I get the following message:

TypeError: 'generator' object is not callable

so, my objective is to change the values only of the column instructions.

Can somebody please help me? Thank's in advance.


texts:
Once the Model is built we want to check its performance, i did the following

Predicted it on training set.
Compute confusion matrix and ROC curve on training set.
Predicted on test set
Computed Confusion Matrix and ROC Curve on test set.

I want to know is it a right way and what more precise can be done?

I am using Decision Tress  Random Forest SVM Both Linear Radial Naive Bayeian

Logistic regression and Neural Network.  Do all of them have different

Performance Measure


texts: Where to find pre-trained models for transfer learning

texts: I am new to the machine learning field, but I wanted to try and implement a simple texture classification algorithm with Keras (to give a clear idea about what I mean as 'texture', [this][1] is the most basic texture database that can be found). However, I have a very small set of data, so I thought to try to apply transfer learning to the problem; however, I couldn't find anything on this online, so I wanted to understand which are the best places to look for a pre-trained neural network. Do you have any suggestion in this regard? Which website is best for getting an idea on how to start a machine learning project?


texts: Skip-thought models applied to phrases instead of sentences

texts:
My goal is to build a statistical model with domain specific phrase embeddings.

To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence embeddings, I am more interested in complete phrases as embeddings. The goal is to be able to create this model and then use some kind of transfer learning to create phrase embeddings from the unsupervised task of skip-thought modeling.

One of the biggest issues we have is that while our corpus is fairly large, it is only 23,000 free text notes and thus, approaching maybe a 11 million words at best. Thus, a standard word embedding model would have issues wrt data sparsity. Also, our entities of interest are basically multi-token phrases. This is why we are looking at building a model using skip-thought vectors, where instead of sentence embeddings, the unit of interest is a phrase, and then using transfer learning to create phrase embeddings.

I am wondering if anyone is familiar with any relevant research on this, and how I would go about training a corpus of relevant documents to create a statistical model of phrase embeddings using skip-thoughts. Would I first create a word2vec model and then use that for input to further train the model? Is this similar to creating a word2vec skip-gram model wwithout worrying about sentence boundaries? Any recommended packages out there? I have not had any luck pinning down specifics on how to do this.

NB: One problem with our text, is that not all sentences in the text are clearly delineated as a sentence. So, I'm thinking this might be beneficial, in that I could have a boat load of skip-grams as phrases that cross sentences boundaries.


texts:
I am using Keras to train different models on the COCO keypoints dataset. All of the models I am working with are used for image segmentation, so they output heatmaps corresponding to the labels.

All of the segmentation models use a binary cross-entropy loss and an accuracy metric. However, I have realized now that these are not fit for keypoint detection because in a 256x256 image, there is only a single '1' label for the keypoint. This drives the loss to 0 and the accuracy to 100% VERY quickly.

What is commonly done for loss/metrics in keypoint detection?


texts:
In one of the recent blog post by Deepmind, they have used game theory in Alpha Star algorithm.
Deep Mind Alpha-Star:

Mastering this problem requires breakthroughs in several AI research challenges including:

Game theory: StarCraft is a game where, just like rock-paper-scissors, there is no single best strategy. As such, an AI training process needs to continually explore and expand the frontiers of strategic knowledge.

Where the game theory is applied when it comes to reinforcement learning?


texts: Using TSNE to Visualize Clusters in Python

texts:
I'm using TSNE to visualize my clusters but the output seems a bit strange. There are supposed to be 3 clusters but instead, there are 4 lines. Is there something wrong with how I'm visualizing them or is it the kmeans method itself?

import pandas as pd
import numpy as np
import ast
from sklearn import metrics
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
from sklearn.decomposition import TruncatedSVD

colNames = ['unixTime', 'sampleAmount','Time','samplingRate', 'Data']

data = pd.read_csv("project_fan.csv",  sep = ';', error_bad_lines = False, names = colNames)

# changing data into list
data['Data'] = data.Data.transform(ast.literal_eval)

# Selecting the average value from the list and replacing the list with it
data['Data'] = data.Data.apply(np.mean)

kmeanModel = KMeans(n_clusters = 3)
kmeanModel.fit(data)

y = kmeanModel.labels_

X_train, X_test, y_train, y_test = train_test_split(data, y, test_size = 0.2, random_state = 1)

k = 3
tfs_reduced = TruncatedSVD(n_components=k, random_state=0).fit_transform(data)
tfs_embedded = TSNE(n_components=2, perplexity=40, verbose=2).fit_transform(tfs_reduced)
fig = plt.figure(figsize = (10, 10))
ax = plt.axes()
plt.scatter(tfs_embedded[:, 0], tfs_embedded[:, 1], marker = "x", c = km.labels_)
plt.show()

Sample Dataset:

       unixTime  sampleAmount  Time  samplingRate         Data
0  1.556891e+09         16384   340  48188.235294  1620.242170
1  1.556891e+09         16384   341  48046.920821  1620.237716
2  1.556891e+09         16384   340  48188.235294  1620.236340
3  1.556891e+09         16384   340  48188.235294  1620.229289
4  1.556891e+09         16384   340  48188.235294  1620.227541

Output:


texts: Unable to figure out nInputPlane in SpatialConvolution in torch?

texts:
Documentaion for Spatial Convolution define it as

module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH])

nInputPlane: The number of expected input planes in the image given into forward().

nOutputPlane: The number of output planes the convolution layer will produce.

I don't have any experience with torch but i guess i have used a similar function in keras

Convolution2D(64, 3, 3, border_mode='same', input_shape=(3, 256, 256))

which takes as input the shape of the image that is 256*256 in rgb.

I have read usage of Spatial Convolution in torch as below but unable to figure out what does the nInputPlane and nOutputPlane paramter correspond to?

local convLayer = nn.SpatialConvolutionMM(384, 384, 1, 1, 1, 1, 0, 0)

In the code above what does these 384,384 represent ?


texts:
I was doing some digging to get some deeper understanding on the Capacity Planning done for setting up a Hadoop Cluster.
I have a daily ~100 GB of data generated and would like to find how a Capacity planning needs to be done for it

Daily Input : 80 ~ 100 GB
Project Duration : 1 year
Block Size : 128 MB
Replication : 3
Compression : 30 %

It would be really helpful if you can guide me through and give me a good insight on how and what factors should be taken into consideration for the planning.


texts: Building a Tic Tac Toe game which learns by itself

texts:
Let's assume we have a huge database gathered which contains winning sequences, which each player played and won. (Both the players are programmed, to obtain random numbers and attempt playing the game).

  What would be the best way to use these winning sequences each player
  has gathered for future plays? Are we going to program one occurrence
  to play without using historical data and few other occurrences using
  historical data?

  What would be the best way to save the winning sequences, the entire
  path which both the player One and Player Two has taken or only the
  sequence used by the winning player?

This is my first attempt to machine learning paradigm and hope some support could be taken!


texts: How do we decide between XGBoost, RandomForest and Decision tree?

texts: What do we take into consideration while deciding which technique should be used when dealing with a particular dataset? I understand that there isn't any hard and fast rule to this. Do we use XGBoost only when there are a lot of features in the dataset and RandomForest for otherwise cases? Or are we suppose to hit and trial and find whichever gets us better results everytime?


texts: Python TimeSeries Error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value

texts:
I'm creating a time series model in Python. I converted a int field that counted the days (1,2,3,4) to a datetime field using the following code:
from datetime import datetime, timedelta

start = datetime.strptime('2016-01-01', '%Y-%m-%d')

df['Day'] = [start + timedelta(x) for x in range(730)]

I checked and it worked. The first date starts with 01/01/2016. I checked the dtypes of my two columns. The Day column is a datetime64[ns], while the revenue columns is float64.
When I try to plot the following code, I get an error that says I'm receiving this error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units. I'm a little confused as to what's causing this since my datatype on the day columns is a datetime.
rolling_mean = df.rolling(window = 12).mean()
rolling_std = df.rolling(window = 12).std()
plt.plot(df, color = 'blue', label = 'Original')
plt.plot(rolling_mean, color = 'red', label = 'Rolling Mean')
plt.plot(rolling_std, color = 'black', label = 'Rolling Std')
plt.legend(loc = 'best')
plt.title('Rolling Mean & Rolling Standard Deviation')
plt.show()
plt.close()


texts: Get feature importance for each observation with XGBoost

texts:
I have trained an XGBoost binary classifier and I would like to extract features importance for each observation I give to the model (I already have global features importance).

More specifically, I am looking for a way to determine, for each instance given to the model, which features have the most impact and make the input belong to one class or another. I would like to know something like the top 5 features which make the observation belong to some class and indications on how I should modify these 5 features so that the probability of belonging to this class decreases or increases.

For example, let’s say my model predicts whether a house costs more than 100,000 dollars (this is the positive class) based on its location, surface and number of bedrooms. I give it the following input: London, 400 square foots, 4 bedrooms and my model predicts a probability of 56% for the house to be in the positive class. I am looking for a Python module or a function that would show the most influential features for each observation.


texts: Export pandas dataframe to a nested dictionary from multiple columns

texts:
Its a similar question to

Export pandas to dictionary by combining multiple row values

But in this case I want something different.

from pandas import DataFrame

df = DataFrame([
           ['A', 123, 1],
           ['B', 345, 5],
           ['C', 712, 4],
           ['B', 768, 2],
           ['B', 768, 3],
           ['A', 123, 9],
           ['C', 178, 6],
           ['C', 178, 5],
           ['A', 321, 3]],
           columns=['maingroup', 'subgroup', 'selectedCol'])

I'd want the output to be:

{
 'A': {'123':[1, 9], '321':[3]},
 'B': {'345':[5], '768':[2, 3]},
 'C': {'712':[4], '178':[6, 5]}
}


texts: Get cosine similarity with constraint on the size of strings

texts:
I want to get the similarity between strings (in python)
However, I also want to add a weight based on the size of the strings. If the cosine similarity is 1, but the string is of 2 words, I want the final similarity to be lower than a cosine similarity of 1 with 10 words per string.

Until now, this is what I'm doing:

def weight_cosine_similarity(num1, num2):
    """
    num1: nb of words of string 1
    num2: nb of words of string 2
    """

    distance = (abs(num1 - num2) + 1)  # + 1 to avoid dividing by zero
    total_words = num1 + num2
    sigm_total_words = 1 / (1 + math.exp(-total_words))

    return sigm_total_words / distance

Two things impact the weight:
1. if number of words in string 1 is different than string 2, the weight will be smaller
2. if the total number of words is high, the weight will be higher (using sigmoid to get a number between 0 and 1)

The cosine similarity do the rest of the job.

My question: Do you know any better ways to better answer my problem, which is, to find a good weight when comparing two strings.

Thanks for your help.


texts: Intutitively what advantages are there to use more data than absolutely necessary while fitting a linear regression?

texts:
If I have p features in a vector, I can get a unique solution for the weights by using just the p independent rows(data points) in the input, why should then I be using more data points than absolutely necessary?
What advantage does it offer?


texts: interpolation - graphical quality evaluation

texts:
I try to compare different interpolation models quality and I'm looking for a graphical tool to do that.
Application case: I'm not familiar with intepolation using neural networks. I decide to test it on a dataset having 5 inputs for 1 output: https://archive.ics.uci.edu/ml/datasets/Airfoil+Self-Noise# And to prospect it quickly, I used Orange canvas that integrate the sklearn Multi layer perceptron. I'm surprised to see that "bigger the network is, better is the result". And I would like to investigate it.
Basic metrics are MSE, RMSE, MAE, R2. But it is a too global approach. Maybe the interpolation is good on a specific range and not somewhere else, and I would like to see that. In sklearn there is also the "Explained variance score" metric that is close to what I'm looking for. But I was imagining a simple "quality curve" between estimated values vs original values. I'm certainaly missing something. That's why I need your help.
Can you share some practice, tools, or keywords to help me in visual regression quality understanding ?


texts: Using scikit Learn - Neural network to produce ROC Curves

texts:
I want to verify that the logic of the way I am producing ROC curves is correct. (irrelevant of the technical understanding of the actual code). I have a data set which I want to classify. I am using a neural network specifically MLPClassifier function form python's scikit Learn module.

I am passing a training data set to the fit function and then using the predict function with the testing data set. I am then outputting a confusion matrix with a false positive value and a true positive value.

what I would like to do is calculate a ROC curve where I need a set of true positive and false positive values. would it make sense to run the neural network (MLPClassifier) multiple times with different targets each time and record the different true positive and false positive values?

It seems simple enough for me. Is it the right way to produce a ROC curve with a neural network?


texts: How to save/load a Model (Pickle) with a specific path/directory

texts:
I seems like a very basic think but I couldnt find an answer to it.

I want to save my model to a specific directory using pickle.
The two algorithms below work fine for saving it in the same directory as the code itself but I want to save all my models in a dedicated folder.
I tried to just change the "filename" to "filepath" and well, make it a path but the world isnt that easy it seems.

Example Path: C:\Learning\Python\Data Science\02_TrainedModels

.

# save the model to disk
filename = 'Father_Son_Height_Model.pckl'
pickle.dump(lm, open(filename, 'wb'))

filename = 'Father_Son_Height_Model.pckl'
loaded_model =    pickle.load(open(filename, 'rb'))

With This Code:

# save the model to disk
filepath = r'H:\99_Lernen\Python\Data Science\02_Trained Models\Father_Son_Height_Model.pckl'
pickle.dump(lm, open(filepath, 'wb'))

I get this Error:
FileNotFoundError: [Errno 2] No such file or directory: 'H:\99_Lernen\Python\Data Science\02_Trained Models\Father_Son_Height_Model.pckl'

In this line of code:

pickle.dump(lm, open(filepath, 'wb'))


texts: Is it true that Machine Learning is basically the study of Pattern recognition along with Optimization?

texts:
I study Pattern Recognition (Classification, Clustering, NeuralNet, Decision tree, and so on. But, excluding textual analysis), and Optimization (study of computational complexity, and trade off of computation and complexity of the target hypotheses, genetic algorithm, and so on) in my university.

If somewhere else somebody studies "Machine Learning", can I assume that they are actually studying Pattern Recognition + Optimization?

That is, is machine learning the study of pattern recognition along with optimization?


texts: Why Davies-Bould chose a number ob cluster higher than Silhouette or Calinsky Harabasz?

texts:
I am doing use several metrics in order to know what number of clusters is correct in order to do this I selected 3 clustering algorithms and 3 internal evaluation metrics, Silhouette, Calinsky Harbasz and Davies Bouldin.
The results of this was the following:
      S    CH    DB
Kmean 3    3     9
Agglo 2    2     9
Gauss 3    3     10

The original dataset has 3 groups, and in general S and CH works well, the questions is what DB always returns a high value for the number of clusters?
Thanks


texts: Feature Engineering - correlation with binary outcome - Titanic Dataset - Ticket feature

texts:
I am currently building my first machine learning model using the titanic dataset.
After the data exploration, I decided to focus my attention on the 'Ticket' feature.
One thing I have noticed about this feature is that it is not unique per each passenger; this had led me to believe that other features can be extracted from this variable:

Is_Group -> to indicate if the ticket represents a group booking
Group_Size -> the number of passengers in each group.

# save the tickets that appear more than once (i.e. group tickets)
_ = (data.Ticket.value_counts()>1).to_dict()
ls = []
for key in _:
    if _[key]==True:
        ls.append(key)
#extract the feature
data['Is_Group'] = data['Ticket'].apply(lambda x: 1 if x in(ls) else 0)

# create another dict containing the number of counts per each ticket
group_size = (data.Ticket.value_counts()).to_dict()
# extract the feature from the mapping
data['Group_Size'] = data['Ticket'].map(group_size).fillna(0)

The reason I am doing this is because I wanted to explore the nature of the relationship between the extracted features from Ticket and the target Survived (and later decide how to deal with outliers in the SibSp and Parch feature):

data.groupby(['Is_Group','Group_Size'])['Survived'].describe()

From the above table I can see that group size 2/3 have almost a 60% and 70% chance of survival. Now, this led me to think that there is a correlation (or at least some sort of relationship between group size and Survived). Therefore, I decided to create a correlation matrix to make sure that is the case.

As I expected there is a correlation between Is_Group and Group_Size (as they have been extracted from the same feature) but there is no correlation between these extracted features and Survived. Hence my, confusion.
I thought given the high mean values of Survived for Group_Size (2,3) there was a relationship but clearly I am getting something wrong here.

Can anyone help clear this doubt of mine?


texts: Guidance to create a machine learning alg to find multiple overlapping date over multiple period

texts:
i have a dataset of work period from a lot of people during multiple years and i would like to find the overlapping date between these periods, the data would look something like this:

Name;Start;End;Valid
John Doe;2015-01-01;2017-01-01;Yes
Jonh Doe;2016-02-03;2016-06-04;No
Jane Doe;1980-01-01;2018-01-01;Yes

As you can see i can have multiple lines for the same person and are mapped as valid or not, there are a few more rules like if the person is a professor he can have overlapping work periods but the core is to find these wrong periods created by human error.

Since i am a beginner in machine learning i would to try to implement this as a learning project although i intend to use and expand in the future if all goes well, at least to my eyes seems like a somewhat easy task to do.

I would like your opinion in what kind of algorithm i should use, or what framework i should or not use, or even if this is not very cost effective to implement on ML.

I thought of somekind of supervised learning using tensorflow, maybe is too much?

Also any tips, tricks or tutorials would be much appreciate it.

EDIT: More Information
At first i am providing the answer, which would be the "Yes" or "No" on Valid, but when i provide something like:

Row;ID;Name;Start;End;
1;1;John Doe;2015-01-01;2017-01-01;
2;1;Jonh Doe;2016-02-03;2016-06-04;
3;1;Jonh Doe;2017-06-02;2018-04-01;
4;1;Jonh Doe;1990-01-01;2017-07-01;

So i would like it to point out, hey, John Doe has 2 invalid periods, row 2 and 4, why should it give me that? because 2016-02-03 to 2016-06-04 is already covered by row 1 and row 4 overlap with row 1,2,3

At first i thought on making this with regular programming to check for everything, but there are some cases where human error make this quite a difficult task to do, thats why ml got in to my mind, the algorithm does not need to have really high accuracy, this is mostly a learning project.


texts: How to club the orders in such a way that maximum number of items are common amongst them?

texts:
Consider the following data set:
The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst them.
For example:

SO1 SO3 SO5  can be clubbed for 8 items ( Item 1,2,3,5,6,8,9,11) and SO2 and SO4 can clubbed together because 9 items are common ( Item 1,3,4,5,6,8,9,10,11).

The approach I followed was:
 I found out the number of combinations possible as in (SO1 SO2), (SO1 SO3), (SO1 SO4)……(SO1 SO2 SO3), (SO1 SO2 SO4),….(SO1 SO2 SO3 SO4),…(SO1 SO2 SO3 SO4 SO5). For n number of SOs I would be getting around 2^n -n -1 combinations.
Later I compared the data in each combination for equality.

Based on the number of matches, I thought I would be able to select the combinations. But this process would become cumbersome for 300 shop orders with around 2000 items. And it is taking a lot of time to compute as well.


texts:
I am solving a problem that address this question "What are the Actions that lead to high or low score?"

I have the following Data that consist of text and score , I want to derive the words or Actions from text that lead to high/low score

I have huge data , and text length can be a paragraph, sample data like below

+----------------------+-------+
|         Text         | Score |
+----------------------+-------+
| Support Team Goal    |    90 |
| Generate Lead        |    80 |
| Contact 30 customers |    30 |
| Support Team Goal    |    30 |
+----------------------+-------+

Approach followed:
I followed Naive Bayes way.

First I classified my Data into High (score(75 and above)) and Low (score below 75)
I converted the High to Term Document Matrix and the Low as well
I found words appearing only in low and judge they lead to low score (include word if frequency greater than 7 lets say)
I found words appearing only in high and judge they lead to high score (include word if frequency greater than 7 lets say in high)
I found probability of common words, For words appearing in both high or low, I calculated the probability of the word occurrence to happen in low/high (include word if probability greater than 75%)

Note: I worked with bigrams. Not sure if this is the right approach to analyze my problem, or there are better techniques. Please advise


texts:
Suppose I've this dataset:

Employee_ID  Store_ID  Company_ID  Stock_ID  Product_Value
    1            1          1          2           3.7
    4            1          4          2            8
                           ...

Where:

Employee_ID: Is the unique number for a employee that sold the product
Store_ID: Unique number for Store Chain
Company_ID: Unique number for product supplier
Stock_ID: Unique number for the product purchased
Product_Value: Product Value

And I want to make a segmentation over my dataset using K-Means... Its make sense use all the variables for my dataset?


texts: A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected?

texts:
There is no other description about the data, if it is univariate, bivariate, etc. neither the type of distribution is given.

I recently came across this question, I would like to know how skewness affects unaffected data percentage


texts: Does Python have R's tidytext equivalent?

texts: I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.


texts: LIterature on query generation from a labelled document term matrix

texts:
I have a labelled dataset of relevant and non-relevant documents for which I built a boolean document term matrix.
I am trying to develop an algorithm which given this input would create a text-based boolean search rule which identifies a subset of the data favouring first of all sensitivity and then specificity.
I'd like to know published literature on the topic. I made some initial search but couldn't find anything related. I'd be glad if you can point me to relevant papers/authors.


texts:
Assume I have a multi-class classification task. The labels are:

Class 1
Class 2
Class 3

After LabelEncoder(), the labels are transformed into 0-1-2.
My questions are:

Do the labels have to start from 0?
Do the labels have to be sequential?
What happens if I replace all label 0s with 3 so that my labels are 1-2-3 instead of 0-1-2 (This is done before training)
If the labels were numeric such as 10-100-1000, will I still have to use LabelEncoder() to encode them into 0-1-2?


texts: Choosing an optimizer to perfectly fit a neural networks to training data

texts:
In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae < 1e-04) on the training data?

Details: In my particular case, the input of the function is 60-dimensional and the output is 1-dimensional (the training data set is prepared by solving a forward model). The input and output are normalized and the sample mean is converted to 0. My neuron's activation function is tanh (this gave me a better result than ReLu and Sigmoid). I have so far used Keras's "adam" and "sgd" as optimizers and I have tried with various learning rates.

In addition, I have tried with increasing the number of neurons in each hidden layer and increasing the number of hidden layers as well. At some point, the total number of my trainable parameters was more than 100 million.

However, even for trying different batch sizes and 10,000 epochs, my best mean absolute error (mae) was never below 0.07. The only part I haven't yet experimented much is to customize/use advance optimizers. I am completely clueless why the neural network can't find a set of trainable parameters where, at least, it can over-fit the training data?

Any suggestions from the experts? Thanks in advance for your time and patience. I really appreciate your support.

An example code:

input_size = ND
nodes = 10000
inp = Input(shape=(input_size,),name='Input')
l0 = Dense(nodes, activation='tanh',name='Level0')(inp)
l1 = Dense(nodes, activation='tanh',name='Level1')(l0)

lo = Dense(1, activation='tanh',name='Level_out')(l1)

merged = Model(inputs=inp,outputs=l10)
opt = optimizers.adam(lr = 1e-5)
merged.compile(optimizer=opt,
              loss='mean_squared_error',
              metrics=['mae'])


texts:
Linear Classification

Consider a labeled training set shown in figure below:

Label   -1  -1  -1  -1  -1  +1  +1  +1  +1  +1

Coordinates (0,0)   (2,0)   (3,0)   (0,2)   (2,2)   (5,1)   (5,2)   (2,4)   (4,4)   (5,5)

Perceptron mistakes 1   9   10  5   9   11  0   3   1   1

Suppose that we now find the linear separator that maximizes the margin instead of running the perceptron algorithm.

What are the parameters  0  and    corresponding to the maximum margin separator?


texts:
I'm currently learning about Collaborative Learning and Content-based Recommendation.

One of the main things that is discussed in both methods is about calculating similarity between two users or two items. Commonly, similarity between two entity (each represented by a vector) can be calculated using Cosine similarity or other similarity coefficients.

One question that I haven't found the answer is about calculating similarity between two entities in which each has a feature that explains each entity's location.

Let's say, the entities in my data is about restaurants and each restaurant has several attributes such as

food price average
user rating/review score
location in $(\text{lat}, \text{long})$
etc.

For food price average or user rating, maybe it's still relevant to include them in cosine similarity function, but how about location–which is described in a tuple of real numbers? What's the best way to include distance between entities as a component in the similarity function?

Thanks!


texts: Knowledge Graph as an input to a neural network

texts:
I want to create a neural network that takes as an input a knolwedge subgraph(different types of nodes and different types of edges) to predict some properties. For instance an input in the graph can be :
:Mike :likes :chocolate;
      :studies :Biology.

Assuming we have n nodes and r relation types. How can I provide this as an input in a neural network and how will the architecture of the neural network look like.


texts:
I am having Sales data of 2018 and 19. I need to convert to time series. The data is not having daily sales

View(df)

Sales          Date

75606         11/01/18

95620         16/01/18

55666         21/01/18

56270         29/01/18

45600         11/02/18

65620         18/02/18

50660         26/02/18

76200         10/03/18

75606         20/03/18

95620         27/03/18

55666         28/03/18

56270         29/03/18

45600         17/04/18

65620         24/04/18

50660         02/05/18

76200         16/05/18

And so on till 2019 Dec

Here the problem is if I convert to time series by using following code

tsn <- ts(df[,1], start =c(2018, 1), frequency=12)

It's considering only 1st 12 rows
The output will be having 1st 12 values

So how to convert the whole data into time series. Should I aggregate monthly wise and convert that into time series or is there any way to do.


texts:
I am looking for a way to quantify the performance of multi-class model labelers, and thus compare them.
I want to account for the fact that some classes are ‘closer’ than others (for example a car is ‘closer’ to a ‘truck’ than a ‘flower’ is.   So, if a labeler classifies a car as a truck that is better than classifying the car as a flower.
I am considering using a Jaccard similarity score.  Will this do what I want?


texts:
Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$

I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e.

$result=[(0, 2), (0, 3), (1, 0)]$

I tried taking cartesian but it is giving other possibilities.
Also I tried with reverse of the above and then joining but it is also giving many use cases.
Please let me know the correct approach to solve the above.


texts: implementing an algorithm that mixes data clustering and linear regression

texts:
i have the following dataframe available in the link as a csv, it conveys information about stars.

more specifically -
column ID represents arbitrary ID of sample.
column z represents my target variable (response).
the other columns represent the attributes available for each sample (predictors) and their corresponding measurement errors.

i used the following code to reduce the 11D data to 3 principal components and plotted the scatter of the data in principal space (with color indicating of the target variable Z)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

#first we remove the target z and ID from the dataset and standredize it (mean=0 and std=1)
pca = PCA(n_components=3)
data_for_pca=data_clean.iloc[:,2:13]
data_for_pca=StandardScaler().fit_transform(data_for_pca)
#now we perform the pca and get the amount of variance, or relative information that each new component holds.
principal_c=pca.fit_transform(data_for_pca)
pd.DataFrame(pca.explained_variance_ratio_).transpose()

import matplotlib.cm as cmx
from mpl_toolkits.mplot3d import Axes3D
def scatter3d(x,y,z, cs, colorsMap='jet'):
    cm = plt.get_cmap(colorsMap)
    cNorm = matplotlib.colors.Normalize(vmin=min(cs), vmax=max(cs))
    scalarMap = cmx.ScalarMappable(norm=cNorm, cmap=cm)
    fig6 = plt.figure()
    ax6 = Axes3D(fig6)
    ax6.scatter(x, y, z, c=scalarMap.to_rgba(cs))
    ax6.set_xlabel('pc1',fontweight='bold')
    ax6.set_ylabel('pc2',fontweight='bold')
    ax6.set_zlabel('pc3',fontweight='bold')

    scalarMap.set_array(cs)
    fig6.colorbar(scalarMap)

    plt.show()
scatter3d(principal_c[:,0],principal_c[:,1], principal_c[:,2],np.array(data.iloc[:,1]))

i attached the dataframe and code so anyone can reproduce and observe the 3d plot from all directions, my main intention with this question is to check if my intuitive analysis of the results is good and how to implement my idea on predicting z from the predictor data.

i see that the dots have somewhat of a smooth spherical gradient centered - roughly speaking - at (-2.5,-1,-0.25).
perhaps i should implement some kind of gaussian kernel? if that's a good idea, how can i implement it?

another observation is that the data is slightly clustered in "plates" (as can be seen  .

maybe i should perform seperate linear regression for each cluster/plate. and let the algorithm classify each point to a cluster/plate, then for each cluster/plate i can infer the target with the more sensitive linear regression coefficients.

if you think that might work, how should i implement it?

maybe there's a more rigorous approach to the further analysis of the PCA? (i mean that i'm kinda using my eyes to decide what is best to do but i'm sure there's a computational approach to the task).

would love to hear opinion and advice, this is a solo project that is part of an attempt to get a better understanding of data science after a BSc in physics.

thanks in advance!


texts: Given a list of thresholds in descending order will the corresponding FPRs and TPRs lists always be in ascending order?

texts:
I've made a series of predictions with a machine learning model. I have a list y of labels and a list p of predicted probabilities such that p[i] is the predicted probability of the entry associated with y[i].

With Python's sklearn library I ran

fprs, tprs, thresholds = metrics.roc_curve(y, p)

thresholds is a list in descending order of decision thresholds, while fprs and tprs is the list of associated false positive rates and true positive rates where fprs[i] and tprs[i] are the respective rates given the decision threshold at thresholds[i].

My question is, given that thresholds is in descending order, will fprs and tprs be guaranteed to be in ascending order? I believe that as the decision threshold decreases, more and more probabilities will be marked positive, thus both rates will increase monotonically.

But I've been wrong before about things that seem obvious to me, and I'm writing a tool that must assume these lists will be sorted in ascending order, so I'd like to be sure


texts: Where can I find a software library for pairwise matching (ideally, Python, R, Java)?

texts: I am looking for a library that implements a pairwise ranking algorithm. For example, if I have 200 writing samples from 100 people (two samples from each individual) and I want to identify which samples belong together (i.e., were written by the same person), what library could I use?


texts: How to plot train test error for classification models like Support Vector Classification(SVC). I am using SVC from sklearn module, not able to get train and test errors to plot


texts: Why does exploration in DQN not lead to instability?

texts:
Why does action exploration in DQN not lead to instability?

I see in DQN algorithms, that it selects random actions even after some iterations.

My question is how does this approach not lead to instability? Even the final value of epsilon (the probability of a random action) is non-zero!

If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?


texts: Q : Which metrics to use for fitting problems?

texts:
I have a basic fitting problem : 20 different features, a set of 5000 examples, 1 output.
I try different architectures to find the best one, and here the problem is : how do I make my decision at the end ?
I am saving r² and MSE at the moment, but is there anything else I could use ?
I know other fields in NN use fscores, accuracy and other fancy metrics, but for a basic fitting problem, how can I choose between two architecture when the metrics are very similar ? And in fact, if I plot the network at the end, better metrics (R) can lead to very weird behaviors like small clusters everywhere (on a usual X=Y plot) so I suspect the R scores to be a bad metric for my problem.


texts: Hidden Markov Model with Autoregressive emission model?

texts:
So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to a standard multivariate normal distribution.
In other words, conditional on the hidden state, a particular GMM model produces the observations.
Is it possible to replace this GMM with an autoregressive model, for example, a Vector Autoregression (VAR) in the multivariate case? And if yes, how would the parameters of this model be updated within the Baum-Welch forward/backward parameter estimation framework?


texts: Does MNIST generalise to european handwriting?

texts: I want to use the MNIST dataset to teach my neural network to recognise numbers. My problem is: The data I am working with is "european". What i mean by that is: the seven always has a dash (it often has a dash in MNIST too so that is more unlikely to cause a problem) and more importantly: the 1 is always written with this hook in Europe, whereas Americans tend to use a straight line (almost 100% of MNIST 1s are a straight line). So it seems to me that training on MNIST can cause problems. Have you ever ran into that problem? And if so, how did you solve it? I haven't found good solutions online.


texts: Close set and open set classification at the same time

texts: Is it possible to use a neural network(or another approach) to classify image based on trained data and at the same time if new image classes are introduced in the test set it should classify those unseen images(open set data) to new classes(kind of telling me which new class this new unseen data belongs to?) on which training is not done.


texts: python linear search code, what’s a better example if a linear search Algorithm?

texts:
Having issues with this python code

def linerar_search(list, target):
  """ Returns the index position of the target if found, else returns None"""

for i in range(0, len(list)):
    if list[i] == target:
      return i
    return none

    def verify(index):
        if index is not None:
            print("Target found at index: ", index)
        else:
                print("Target not found in list")

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

result = linear_search(numbers, 12)
verify(result)

showing error in line 6


texts: Use output of K-Mean for Logistics regression

texts:
I've created a binary classifier using K Mean, which predicts fraud and legitimate accounts, 0 and 1. This uses two features, let's say, A and B.

Now, I want to use other features like C and D, to predict fraud and legitimate accounts. Can I use output class predicted in Kmeans, and use it as a label for logistics regression using features C and D?

Sorry, if this sounds like dummy questions, I'm just starting into Data Science, so please let me know if I missed something fundamental.


texts:
I'm trying to retrain Inception model final layer for a binary classification.

My training image set contain 2000 images in class 1 and more than 6000 images in class 2.

Will this huge difference in number of images of each class in training set affect my classification?


texts: What is the difference between GPT blocks and BERT blocks

texts:
Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses Encoder only does this mean that the only difference between the two is basically in the masking part?
The cross attention layer in the Decoder is omitted since there is no Encoder within GPT right?


texts:
What's the notation for the worst element/solution?
The best "something" can be denoted with *, i.e.:
$R^* = R_{best}$
$R^? = R_{worst}$


texts: Should I oversample my validation data to get better F1 score and PRC?

texts:
I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use precision/recall/f1 score to decide the performance of model.
I used SMOTE to oversample my training data (after splitting the validation set out). Now my model is trained with oversampled training data, and I am going to test it with validation set. If I just validate it on original validation data, I get a F1 score around 0.05, and the classification report is followed:
          precision    recall  f1-score   support

 Class 0       1.00      0.86      0.93      7606
 Class 1       0.03      0.75      0.05        36

If I oversample my validation data, I get a F1 score around 0.85:
          precision    recall  f1-score   support

 Class 0       0.84      0.86      0.85      7606
 Class 1       0.86      0.83      0.85      7606

My question is:

Should I use an oversampled validation set? (because the result is much prettier but I think the model is the same anyway)

Why do I have such bad metrics on the original validation data? Is it because the data size is not big enough?


texts: Detecting redundancy with Pearson correlation in continuous features

texts:
I have a set of variables that I want to use for a regression or a classification problem.
Having computed the correlation matrix of these variables, I discovered that some of them has an inter-variables Pearson correlation
values as high as 1.

Does this mean that these variables hold redundant information for the learner?
Is it safe to remove one of them without risking information loss? if yes, how to chose the one to remove?


texts:
I have a huge string in R where I want to remove single characters from. So for example, the string might be

blabla a test this that b á

And the result should be

blabla test this that

How do I do this in R?


texts: Applying word embedding function on a large pandas dataset

texts:
We have a pandas dataframe with one column ('message') and 3.9 million rows and need to convert these messages to their word embeddings using Google's Universal Sentence encoder.
import pandas as pd
import tensorflow_hub as hub

url = 'https://tfhub.dev/google/universal-sentence-encoder/4'
embed = hub.load(url)

def __return_embedding(entry):
  return embed([entry])[0].numpy().tolist()

csv_files = [csvfile for csvfile in os.listdir(os.getcwd()) if csvfile.endswith("." + 'csv')]
csvs = [pd.read_csv(f'{os.getcwd()}/{csv}', low_memory=False) for csv in csv_files]
dataframe = pd.concat(csvs, axis=0, ignore_index=True)

embeddings = __return_embedding(dataframe.message) # this says InvalidArgumentError:  input must be a vector, got shape: [1,3900000]

dataframe['embedding'] = np.vectorize(__return_embedding)(dataframe['message']) # This one blows up the RAM on google colab and it crashes.

We tried to use chunking as well but it also resulted in the memory to blow up on Google Colab, and we used chunking on a file with 950K rows.
Can someone help ?


texts: How to learn word embedding from a context on the fly?

texts:
Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a basic notion of a its meaning.

Question: How the tahiliuk's vector could be learned from the context?

Assumption:
a fixed input vocabulary, mapped to a set of word vectors.

Remarks:

Tahiliuk is a completely new word. There is now such word within
training data.
Using approaches like "fastText" will not work here because of
tahiliuk is a random word.
One possible way is to initialize the embedding  as the sum of their context words and then rapidly refine the embedding with a high learning rate. This approach is used in the following paper, but this approach seems to be not perfect

Herbelot, A., & Baroni, M. (2017). High-risk learning: acquiring new word vectors from tiny data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.


texts: Interpreting Categorical Crossentropy Loss

texts: I would like to ask for clarification about the loss values outputted during training using Categorical Crossentropy as the loss function. If I have 11 categories, and my loss is (for the sake of the argument) 2, does this mean that my model is on average 2 categories off the correct category, or is the loss used purely for comparative purposes and cannot be interpreted like I am suggesting ?


texts: Neural Network data type conversion - float from int?

texts: In case of image classification with 0-255 pixel values of integer type, is it necessary/recommended to convert the values into float with Neural Networks? This conversion seems me unnecessary (moreover, float slows down matrix operations) but several implementations follow this practice. Why?


texts: How is the Gaussian noise given to this BLSTM based GAN?

texts:
In a conditional GAN, we give a random noise along with a label to the generator as input. In this paper, I don't understand why in one section of the paper, they say they are giving the random noise as input and the in another section of the paper they are saying it is concatenated to the output.
page 2

page 2 footnote

page 3 model setup section

little overview of the paper:  Code switching is a phenomenon in spoken language where we switch between two different languages. Mixed language models improve the accuracy of automatic speech recognition to higher degree but the problem is less availability of mixed language written sentences. Thus, as a data augmentation technique, a conditional GAN is developed to synthesize English, Mandarin mixed sentences from a pure Mandarin sentence. The trained generator acts as an agent telling which words in the Mandarin sentence have to be translated. It outputs a binary array (of length equal to input Mandarin sentence length). Both generator and discriminator are BLSTM networks.
#####EDIT: The author accepted that it is a typo, noise should be concatenated after the embedding layer not to the output of BLSTM
Author's reply:
It is a typo in page 3.
The noise is concatenated with the output of the embedding layer.
Thanks for your correction.
#####


texts: Keras - Autoencoder different from Encoder + Decoder

texts:
I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The central layer of my Autoencoder is a Dense layer, because I would like to learn it afterwards.

My problem is that if I compile and fit the whole Autoencoder, written as Decoder()Encoder()(x) where x is the input, I get a different prediction when I do

autoencoder.predict(training_set)

w.r.t. if I first encode the training set in a set of central features, and then let the decoder decode them. These two approaches should give identical answers, once the Autoencoder has been trained.

from tensorflow.keras.layers import Input, Dense, BatchNormalization, Flatten, Lambda, Activation, Conv1D, MaxPooling1D, UpSampling1D, Reshape
from tensorflow.keras.models import Model
from tensorflow.keras import optimizers
from tensorflow.keras.layers import GaussianNoise
import keras.backend as K
from tensorflow.keras.layers import Add

import tensorflow as tf

import scipy.io
import sys
import matplotlib.pyplot as plt
import numpy as np
import copy

training = # some training set, 1500 samples of 501 point each
testing = # some testing set, 500 samples of 501 point each

# reshaping for CNN
training = np.reshape(training, [1500, 501, 1])
testing = np.reshape(testing, [500, 501, 1])

# normalize input
X_mean = training.mean()
oscillations -= X_mean
X_std = training.std()
training /= X_std

copy_of_test = copy.copy(testing)
testing -= X_mean
testing /= X_std

### MODEL ###

def Encoder():
    encoder_input = Input(batch_shape=(None, 501, 1))
    e1 = Conv1D(256,3, activation='tanh', padding='valid')(encoder_input)
    e2 = MaxPooling1D(2)(e1)
    e3 = Conv1D(32,3, activation='tanh', padding='valid')(e2)
    e4 = MaxPooling1D(2)(e3)
    e5 = Flatten()(e4)
    encoded = Dense(32,activation = 'tanh')(e5)
    return Model(encoder_input, encoded)

def Decoder():
    encoded_input = Input(shape=(32,))
    encoded_reshaped = Reshape((32,1))(encoded_input)
    d1 = Conv1D(32, 3, activation='tanh', padding='valid', name='decod_conv1d_1')(encoded_reshaped)
    d2 = UpSampling1D(2, name='decod_upsampling1d_1')(d1)
    d3 = Conv1D(256, 3, activation='tanh', padding='valid', name='decod_conv1d_2')(d2)
    d4 = UpSampling1D(2, name='decod_upsampling1d_2')(d3)
    d5 = Flatten(name='decod_flatten')(d4)
    d6 = Dense(501, name='decod_dense1')(d5)
    decoded = Reshape((501,1), name='decod_reshape')(d6)
    return Model(encoded_input, decoded)

# define input to the model:
x = Input(batch_shape=(None, 501, 1))
y = Input(shape=(32,))

# make the model:
autoencoder = Model(x, Decoder()(Encoder()(x)))

# compile the model:
autoencoder.compile(optimizer='adam', loss='mse')
for layer in autoencoder.layers: print(K.int_shape(layer.output))

epochs = 100
batch_size = 100
validation_split = 0.2
# train the model
history = autoencoder.fit(x = training, y = training,
                    epochs=epochs,
                    batch_size=batch_size,
                    validation_split=validation_split)

# Encoder
encoder = Model(inputs=x, outputs=Encoder()(x), name='encoder')
print('enc:')
for layer in encoder.layers: print(K.int_shape(layer.output))
features = encoder.predict(training) # features

# Decoder
decoder = Model(inputs=y, outputs=Decoder()(y), name='decoder')
print('dec:')
for layer in decoder.layers: print(K.int_shape(layer.output))
score = decoder.predict(features) #
score = np.squeeze(score)

predictions = autoencoder.predict(training)
predictions = np.squeeze(predictions)

# plotting one random case
# score should be equal to predictions!
# because score is obtained from the trained decoder acting on the encoded features, while predictions are obtained form the Autoencoder acting on the training set
plt.plot(score[100], label='eD')
plt.plot(predictions[100], label='AE')
plt.legend()
plt.show()
plt.close()


texts: Data preparation and machine learning algorithm for click prediction

texts:
I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have been advised to try logit since logit seems to be what MS and Google are using. I have some questions regarding logistic regression:

Click and non click is a very very unbalanced class and the simple GLM predictions do not look good. How can I make the data work better with the GLM?

All the variables I have are categorical and things like device and city can be numerous. Also the frequency of occurrence of some devices or some cities can be very very low. How can I deal with this distribution of categorical variables?

One of the variables that we get is device ID. This is a very unique feature that can be translated to a user's identity. How can I make use of it in logit, or should it be used in a completely different model based on user identity?


texts: Checking regression coefficients' stability?

texts:
I have a model with 13 independent variables (2 of them are categorical variables) and 678 observations. All of the variables are significant.I'm planning to check the coefficients stability. My approach is to start from a model with one variable and find that coefficients. Then I’ll add another variable; however, I'm keeping the previous variable also. I’ll continue until that all variables are in the final model. I'm using the following R code:

add.APV<-lm(HPV~log(APV),data=Mydata)
add1(add.APV,scope =~ AWD + Variety.bodyandchassis + Flexibility + Model.types + Year + log(CAC)+ Outsourcing + Platformstrategy + Hourly.total + Newownership + Vehicle.Launch + Segment.MD., test="F", k=2, trace=TRUE)

And I'm getting the following error:

Error in factor.scope(attr(terms1, "factors"), list(add = attr(terms2,  :
  upper scope has term ‘NA’ not included in model

I was wondering what the problem is. Any feedback would be appreciated.


texts: Get Logistic regression scores in CNN using Keras

texts:
from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K
from keras.regularizers import L1L2

batch_size = 128
num_classes = 10
epochs = 2

# input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
             activation='relu',
             input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(10, activation='softmax',
kernel_regularizer=L1L2(l1=0.0, l2=0.1)))
model.compile(optimizer='sgd',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

The Above code uses logistic regression With the CNN in Keras. How do I get the score assigned to each image by the logistic regression function.


texts: How can I use Data Science to profoundly contribute to Humanity?

texts:
Undergraduate researcher here. I worked at many traditional scientific research labs, ranging from cancer biology, to radiation medicine, and to supercapacitors. I'm thinking of switching to Statistics and Computer Science from Computational Biology to join the exciting field of Data Science.

What ways can I contribute to Humanity without a technical background in basic science? I don't know how I will feel about making better clickbaits. I've thought about continuing in scientific research but I will lack the science background. I've thought about working for the United Nations. What other ways can I contribute? Inspire me!


texts: How to extract the reason behind a prediction using TensorFlow?

texts:
I created a CNN using TensorFlow2 and trained it as a binary classifier. Is there a way to extract the influence of each pixel upon the prediction?

I am trying to obtain a mask similar to the following:


texts: Build a Generalised Regression Model Containing Linear and Non Linear Predictor Variables with a Target Variable in Python

texts:
Imagine a dataset having five predictor variables and a target variable, through scatter plot I observed three predictor variables having a linear relationship with the target variable and the other two having a nonlinear relationship.

How can I build a Generalized Linear Regression model in such a way that the non-linearity of the two variables is explained along with linear relationship of the other three variables?


texts: Can absolute or relative contributions from X be calculated for a multiplicative model? $\log{ y}$ ~ $\log {x_1} + \log{x_2}$

texts:
(How) can absolute or relative contributions be calculated for a multiplicative (log-log) model?
Relative contributions from a linear (additive) model
E.g., there are 3 contributors to $y$ (given by the three additive terms):
$$y = \beta_1 x_{1} + \beta_2 x_{2} + \alpha$$
In this case, I would interpret the absolute contribution of $x_1$ to $y$ to be $\beta_1 x_{1}$, and the relative contribution of $x_1$ to $y$ to be:
$$\frac{\beta_1 x_{1}}{y}$$
(assuming everything is positive)
Relative contributions from a log-log (multiplicative) model
In log-space, a model could take the following form:
$$\log{y} = \beta_1 \log{x_{1}} + \beta_2 \log{x_{2}} + \alpha$$
which in the 'real-world', assumes the following form:
$$y =  e^\alpha x_{1}^{\beta_1}x_{2}^{\beta_2}$$
But (how) can absolute or relative contributions be calculated from such a multiplicative (log-log) model? i.e., (how) can we calculate how much $x_1$ contributes to $y$? For example if $e^\alpha=10$, and $x_1^{\beta_1} = 100$ and $x_2^{\beta_2} = 1000$, then $y = 10^6$, but what portion of that 1,000,000 was contributed to by $x_1$?


texts: I need to force my random forest model to learn one crucial relationship between features. How can I achieve that?

texts:
Say, given 10 independent features as input to my RF model, when feature 1 and feature 3 are 100 (or less), my model output must be 5 despite the values of the other features. How can I teach that relationship to my RF model - so it will always obey that?

I have tried adding repeated artificial data points to the training set where features 1 and 3 are less or equal to 100 and the output is always zero, but the RF model does not seem to understand/learn that relation just based on that.

I also tried to play with (1) number of trees (2) number of nodes and (3) max nodes without any success.

I am using the mlr library in R to create my RF model. Thanks.


texts: How to Normalise features for small datasets?

texts:
I am working with a small dataset ( N  = 50 ).
I would like to normalise  my input features.
I am facing the following issues:

Because of the small size of the dataset the range of training input features would differ from the range of testing input features.
The input features do not have a theoretical upper bound.

Can you suggest normalisation techniques suitable for this task? Any paper suggestions would also be appreciated.


texts: Ram usage of model keep increasing when training to previously trained model

texts:
Update
It keep increasing when i load the saved model like this.

return tf.keras.models.load_model("models/min__1576172002.model")

and i am saving model using

   agent.model.save(f'models/min__{int(time.time())}.model')

I declare my model as

model = Sequential()
model.add(Dense(6,activation='relu',input_shape=(4,)))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(2,activation='softmax'))

model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

Then its working ok.
But after some time as it was not showing results, I use

model = Sequential()
model.add(Dense(6,activation='relu',input_shape=(4,)))
model.add(Dropout(0.2))

# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(2,activation='softmax'))

model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

I added only one more layer. But now.
I see RAM usage of computer. It is keep increasing and not stopping. If i pause training then RAM will also pause at that point and when i continue training  then RAM will increase and it took 32GB of RAM (and still increasing )and previous model with one less layer took only 6GB of RAM


texts: How does reduction affect training in Keras?

texts:
From my understanding, a loss function first computes the loss for each sample, then it either reduces the list of losses and outputs a scalar or outputs the list itself.
For example, the Keras doc mentions this difference for the Mean Squared Error:

By default, loss functions return one scalar loss value per input sample, e.g.

>>> tf.keras.losses.mean_squared_error(tf.ones((2, 2,)), tf.zeros((2, 2)))
<tf.Tensor: shape=(2,), dtype=float32, numpy=array([1., 1.], dtype=float32)>

However, loss class instances feature a reduction constructor
argument, which defaults to "sum_over_batch_size" (i.e. average).
Allowable values are "sum_over_batch_size", "sum", and "none":

>>> loss_fn = tf.keras.losses.MeanSquaredError(reduction='sum_over_batch_size')
>>> loss_fn(tf.ones((2, 2,)), tf.zeros((2, 2)))
<tf.Tensor: shape=(), dtype=float32, numpy=1.0>

When using fit(), this difference is irrelevant since reduction is
handled by the framework.

Although when training my model I get different results after 1 epoch.
If I use either loss=tf.keras.losses.MeanSquaredError(reduction='none') or loss=tf.keras.losses.mean_squared_error I get loss=0.1402.
Instead if I use loss=tf.keras.losses.MeanSquaredError() I get loss=0.1267.
In the first case reduction should be done regardless what I pass as a parameter for reduction=. Why do you think that happens?


texts: Are there any methods of supervised learning that return a bitmap instead of a set of parameters?

texts:
For example, the SVM or ANN methods perform search of a surface which would separate the data points in a best way. This surface is returned in the vector or parametric form. Are there methods returning a spatial bitmap each voxel of which contains a numeric value defining a class for all points lying within a given voxel?
I would like to share some of the results of my attempts in this direction. Since I'm relatively new in machine learning I can't be quite sure that this was not done before. Perhaps there are some reasons that are obscure to me which make this approach meaningless for a real data.
The below images were obtained by representing each data point by RBF and calculating influence of all such RBFs for each voxel. The adaptive grid was used to optimize calculations. Taking the first step in this direction, I received encouraging performance results with the dummy data and with the one real dataset. Obviously, more tests are required to draw conclusions about the effectiveness of this method. Perhaps, this approach will be useful for noise reduction and datasets with complicated clusters, also it is capable of processing data points in batch mode having no significant impact to a final result.

Dummy dataset.

Zip Code dataset:


texts: How to find the most similar image between two datasets?

texts:
How do you measure the image similarity using (Mean squared error), find the most similar one, and save them as a pair of images in a folder with their similarity.
I have two images datasets:
Dataset 1= (image1.jpg, image2.jpg, image3.jpg, ............).
Dataset 2= (image1.jpg, image2.jpg, image3.jpg, ............).

How do you measure the image similarity and save the most similar one in another folder?


texts: When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?

texts:
I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.

I've looked through ResNet paper and "Delving Deep into Rectifiers" paper (He initialization paper), but I haven't found any mention on normal init vs uniform init.

Also:

  Batch Normalization allows us to use much higher learning rates and be less careful about initialization.

In Batch Normalization paper's abstract, it is said that Batch Normalization allows us to be less careful about initialization.

ResNet itself is still care on when to use normal init vs uniform init (rather than just go with the uniform init).

So:

When to use (He or Glorot) normal-distributed initialization over uniform initialization?
What are normal-distributed initialization effects with Batch Normalization?

Notes aside:

It rhymes to use normal init with Batch Normalization, but I haven't found any paper to back this fact.
I knew that ResNet uses He init over Glorot init because He init does better on a deep network.
I've understood about Glorot init vs He init.
My question is about Normal vs Uniform init.


texts: How to choose feedforward architecture for few number of features but very large instance?

texts:
Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable.
The size of X will  be 1,000,000 X 100 and the y will be 1,000,000 X 1. The value of y is binary.
Under this case, how to pick feedforward architecture?
Such as 100-50-1 or 5000-2500-50-1, any suggestion.
Thanks.