I just want to make sure i am on the right lines so please correct me if wrong. I am testing which hyperparmets are best for logisitic regession on my data X, y where X is featrues and y is target. X, y are made from my training set. I also hhave a test set.  from sklearn.linear_model import LogisticRegression  # split train into target and features      y = Train['target']     X = Train.drop(['target'], axis = 1)     X = pd.get_dummies(X) #split test data into target and features   y_test = Test['target'] X_test = Test.drop(['target'], axis = 1) X_test = pd.get_dummies(X_test)   logistic = LogisticRegression()  # initialize the model # Create regularization penalty space      param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }  clf=GridSearchCV(logistic,param_grid=param_grid,cv=5)    best_model = clf.fit(X, y)# View best hyperparameters print('Best Penalty:', best_model.best_estimator_.get_params()['penalty']) print('Best C:', best_model.best_estimator_.get_params()['C']) #   I will now use these hyper parameters and 'train' it on my training data. Just so i'm sure when we say train do i then take best_model and train on the whole X data e.g.:   bestLog=best_model.best_estimator_ trained_model=bestLog.fit(X,y) predicted=trained_model.predict(X_test)   then use this output above as my final model to test?
18820
In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and then we train the model using the best hyperparameters with the training set and apply the trained model to the test to get the performance.  If we have a small dataset, we cannot create training and test set (not enough samples). Therefore, we will do cross validation (k-fold, leave-one-out, etc) to evaluate the model performance.  I have seen nested cross validation (whether repeated or stratified) has been used in the setting of small dataset, i.e., to generate generalized model performance while optimizing parameters selection. My question is, how can I obtain the best hyperparameters in nested cross validation (repeated/no repeated)?  I am interested in doing this in scikit-learn, if possible. I am a bit confused about how to do it.  I have read several resources but none gave me the definitive answer for this question:
98301
Why does logistic regression not converge for a linearly separable data set?   For linear separable data sets the model parameters go to infinity when mimizing the error function (according to Bishop2006, Pattern recognition and machine learning, section 4.3.2). I don't understand why. It must be connected to setting the score function, i.e. the derivative of the error function, to zero.  The error function is given by $E(\mathbf{w})$ as shown in the figure below. Where $\mathbf{t}$ is the target variable vector of the training set and $\mathbf{w}$ the parameter / coefficient vector. The posterior probabiltiy for class 1 is given by  $$y(n) = p(C_1|\mathbf{x}_n) = \sigma(\mathbf{w}'\mathbf{x}_n).$$ Where sigma is the logistic sigmoid function and $\mathbf{x}_n$ is the feature vector for instance $n$ including the predictors.     My second question to logistic regression is if the to posterior probabilities $p(C_1|\mathbf{x}_n)$ and $p(C_2|\mathbf{x}_n)$ sum to one?
103046
I'm unclear on the exact process of using the validation data.  Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.  Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.
60685
So I am using LASSO in order to reduce the amount of independent variables (~150) I have for my logistic model (n=1200). However, when doing so, the end result (i.e. number of predictors) it chooses varies depending on the seed I provide. 2-3 predictors are consistent, but then sometimes others are added. I guess this is due to the cross-validation for the best lambda parameter often used for LASSO. However, it just seem so inconsistent.  I mean I am using bootstrap in order to validate my results. So in the end I end up with meaningful standard errors, frequency of predictors, corrected statistics etc., but that doesn't change the fact, that the main model, defined on the original data, is kind of random depending on the seed.  How does one "solve" this ? Run the "main model" (i.e. original data) with 100-1000 different seeds to see which predictors are most frequently chosen, and then chose the model that actually has these predictors, or is there anything else that could be recommended ?
117577
I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc)  In many cases hyperparameter tuning on the whole dataset using either Grid or Random search is too costly time-wise.  I started using the following technique   Sub sample my dataset Use the obtained fraction to tune the hyperparameters on Use the obtained parameters to train a model using the whole dataset   To evaluate each set of parameters on the second step I use sklearn's GridSearchCV with cv=10. To evaluate the final model that I create in the third step I use sklearn's cross_val_predict. In that sense I evaluate my models leaving a 10% percent of data out, I train on the rest and measure the predictive accuracy on the 10%, iteratively 10 times, then taking the average of the scores.  What made me worry is that the prediction accuracy I get from training on my whole dataset, is really close to the evaluation I get when tuning the parameters for the best set of parameters (each tested set of parameters outputs a score obtained from averaging 10-fold-cross validation results).  Most of the times the accuracy that cross_val_predict measured using all the training examples (whole dataset) is a little bit above what the evaluation of the best parameters returned.  To illustrate this here is the evaluation of a set of parameters (on a smaller dataset than what I described above but the effect is the same)  Best parameters set found on development set: {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} Scores for all sets of parameters 0.851 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.5} 0.852 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.1} 0.829 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.001} 0.853 (+/-0.006) for {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} ...   And here are the averaged scores (from cross_val_predict) I got from training on my whole dataset using the best parameters  precision    recall  f1-score   support        0       0.86      0.85      0.86     15417       1       0.86      0.87      0.87     16561  avg / total       0.86      0.86      0.86     31978  acc score: 0.863750078179 roc au score: 0.863370490059 [[13147  2270]  [ 2087 14474]]   As you can see training on the whole dataset improves the results. I have also validated that badly tuned model (e.g. using the default values or random values for C and gamma) leads to much worse prediction accuracy.   Overall I think that tuning the hyperparameters on a subset is not ideal but can potentially lead to relatively good results without having to wait too long. I for example before using that approach used optunity package for tuning the hyperparameter on the whole dataset. This procedure would take 3-5 days to complete and would produce results that either had really good precision or really good recall but not both, so although for each class either the precision or the recall was really high (higher than what any of my other classifiers had achieved) the f1 meassure was really low. In the contrary using the later approach leads to some hours of training and a better f1 meassure.   My concerns are:  Do I limit my classification accuracy? Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset? If such a harm of performance is happening is it somehow limited by some factor?
50425
I am a little bit confused by early stopping and in particular by how it can be inserted inside a CV framework. As far as I understand, I can fix the optimal number of epochs (for NN, or number of trees for XGB) by early stopping, that is:   pick a validation set, train with increasing number of epochs until a predefined metric evaluated on the validation set starts worsening that is the optimal number of epochs to prevent overfitting   and that's fine. But then I would like to insert this early stopping framework inside a CV framework: Suppose I have a model with 10 hyperparameters I want to fix via CV. And suppose that there is an eleventh hyperparameter, the number of epochs. My feeling is that one can do like this:   create the K resampled folds, for each of which you have a training and validation set choose a suitable grid for your 10 hyperparameters for each point on the grid train your model in each fold with early stopping, that is use the validation set of the fold to keep track of the preferred metric and stop when it gets worse take the mean of the K validation metric choose the point of the grid (i.e. the set of hyperparameters) that gives the best metric   Questions:   What number of epochs should I choose as optimal? in each of the K folds I have, in general, a different number of stopping epochs.  seems to suggest they take the mean of the K stopping epochs. Is this right? Is it actually a "fair" practice to use the validation metric coming from the early stopping as a proxy of the out-of-sample metric? As  in section 3.4.5, maybe the best thing to do would be:     ...if you want to do early stopping, then in each fold you should take your training set and split it again, holding out a small early-stopping-set to guide the early stopping, and then evaluate the model on the validation set of that fold.   But this seems to me a really intricate process...   Unfortunately, I wasn't able to find references where this issue is presented in a clear and transparent form.
43359
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g- #ROUND 1 param_grid={     'max_depth': [3,4,5],     'learning_rate':[0.1,0.01,0.05],     'gamma': [0,0.25,1.0],     'reg_lambda':[0,1.0,10.0],     'scale_pos_weight':[1,3,5] }  grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid,                            cv = 3, n_jobs = -1, verbose = 2) grid_search.fit(X_train,y_train) grid_search.best_params_  &gt;&gt;&gt;.....   (and now based on the result changing the params...) after this step I choose the best hyperparameters and run the model; clf_xgb=xgb.XGBClassifier(seed=42,                          objective='binary:logistic',                          gamma=0,                          learn_rate=0.7,                          max_depth=6,                          reg_lambda=0.8,                          scale_pos_weight=1,                          subsample=0.9,                          cilsample_bytree=0.5)  clf_xgb.fit(X_train,            y_train,            verbose=True,            early_stopping_rounds=10,            eval_metric='aucpr',            eval_set=[(X_test,y_test)])  The problem is that when I check the model score clf_xgb.score(X_test,y_test)  I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process. Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?
20499
We're working with some logistic regressions and we have realized that the average estimated probability always equals the proportion of ones in the sample; that is, the average of fitted values equals the average of the sample.  Can anybody explain me the reason or give me a reference where I can find this demonstration?
60650
I am in the machine learning business for a long time, but still, this fundamental fact gets me confused, since every paper, article and/or book describe different kind of usages for validation and test sets. I aim to set my mind at ease forever on this issue:   Let's say I have a relatively small Convolutional Neural Network and I want to train it on MNIST dataset. Naturally, I want to learn the best hyperparameters for the given CNN, like the weight decay coefficient $\lambda$, the learning rate $\alpha$, etc. Naturally, MNIST has 60K training images and 10K test images. The basic practice is allocating 10K of the training set as the validation set for MNIST.   Assume that we are doing a grid search on $\lambda$ and $\alpha$ hyperparameters and assume that we have fixed a $(\lambda^*,\alpha^*)$ pair. Then we start training the CNN using SGD, using the 50K training set and measuring the performance on the 10K validation set in the end of each epoch (a complete pass over the 50K training set). Usually we either stop training if a fixed budget of epochs has been depleted or we start to lose accuracy on the validation set. And then, only for a single time in the training process of the CNN with the $(\lambda^*,\alpha^*)$ pair, we measure the real performance on the test set. We pick the one with best performance. This usage is intuitive and does not cause the network to unfairly overfit on the test set. But the problem is, we did not ever used the 10K validation set in the actual training with SGD.  So, now, what is the correct way to use the validation set here? After we finish training with the $(\lambda^*,\alpha^*)$ pair, either by running out of allowed epochs or early exiting due to overfit on validation set, should we merge the validation set and training set, and train a few epochs more? At this point, our learning rate may already be very small due to decaying it gradually. Should we train everything from scratch by using the 60K training + validation set, setting the learning rate to its initial value?   Another alternative may be to train the CNN with every $(\lambda,\alpha)$ pair in consideration of our grid search, using only 50K training samples in the SGD and 10K validation samples for measuring the accuracy. No interaction with the test set is allowed. Let's assume that after we train with every $(\lambda,\alpha)$ pair, we pick the pair which yields the highest accuracy on the validation set. Then we train the CNN with the picked hyperparameters, from scratch, this time using the 60K training + validation samples directly in SGD, for an amount of fixed epochs. After the training ends, we use the test set once and for all to declare our final accuracy performance.  This method, in my mind, would cause the following issue: The hyperparameters we have picked would be the optimal ones for the 50K training samples! Since we have used a smaller subset of the actual training set, most probably the model would more easily overfit, seeing a lower data variation, so our hyperparameter search may tend to find a higher $\lambda$. And with this $\lambda$ we would probably see a lower accuracy on the actual test set with the full 60K training samples, since the model won't be able to compensate for the variation of the extra 10K samples, due to high $\lambda$.  So, I am not certain what the ultimately correct way to use the validation-test sets. What would be the valid - logical approach here?
123350
There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few:    @DikranMarsupial's  to    However, the answers to those threads are fairly generic and mostly highlight the issues with particular approaches to cross validation and model selection.   To make things as concrete as possible, say for example that we are working with an SVM with an RBF kernel:  $K(x, x' ) = (\gamma \, \vert x - x'\vert)^2$, and that I have a dataset of features X and labels y, and that I want to    Find the best possible values of my model ($\gamma$ and $C$) Train the SVM with my dataset (for final deployment) Estimate the generalization error and the uncertainty (variance) around this error   To do so, I would personally do  a grid search, e.g. I try every possible combination of  $C$ and $\gamma$. For simplicity, we can assume the following ranges:   $C \in \{10, 100, 1000\}$ $\gamma \in \{0.1, 0.2, 0.5, 1.0\}$   More specifically, using my full dataset I do the following:   For every ($C$,$\gamma$) pair, I do repeated iterations (e.g. 100 random repetitions) of $K$-fold cross validation (e.g. $K=10$), on my dataset, i.e. I train my SVM on $K-1$ folds and evaluate the error on the fold left, iterating through all $K$ folds. Overall, I collect 100 x 10 = 1000 test errors. For each such ($C$,$\gamma$) pair, I compute the mean and the variance of those 1000 test errors $\mu_M, \sigma_M$.   Now I want to choose the best model (the best kernel parameters) that I would use to train my final SVM on the full dataset. My understanding is that choosing the model that had the lowest error mean and variance $\mu_M$ and $\sigma_M$ would be the right choice, and that this model's $\mu_M$ are $\sigma_M$ are my best estimates of the model's generalization error bias and variance when training with the full dataset.  BUT, after reading the answers in the threads above, I am getting the impression that this method for choosing the best SVM for deployment and/or for estimating its error (generalization performance), is flawed, and that there are better ways of choosing the best SVM and reporting its error. If so, what are they? I am looking for a concrete answer please.  Sticking to this problem, how specifically can I choose the best model and properly estimate its generalization error?
53760
I am building a simple logistic regression model on 2D data. Here is the input I use.    I built a logistic regression model using this data and it successfully is able to find the discriminating line between these two classes. Here is the separating line found by my program:    Also, just for debugging, I plotted number of iterations vs J(theta), which also looks fine:    Now, confusing part comes into play. I am investigating the cost surface of this model that the logistic regressions is doing the gradient descent on. I am considering three different loss functions. Sum-of-squared loss, hinge-loss, log-loss.  Hinge loss:    Log-loss:    Sum-of-squared loss:    As far as I know, both hinge loss, sum-of-squared loss and log-loss should produce a convex surface. When I look at my surfaces, however, sum-of-squared surface does not look convex to me. Why is that the case?  For convenience, here is the two critical functions from my code:  function retval = J(X, y, theta)     M = size(y, 1);     % sum of squared error     retval = (1/(2*M)) * sum((h(X, theta) - y).^2);  endfunction  function retval = h (X, theta)     % logistic     retval =  1 ./ (1+ exp(-1 * (X * theta))); endfunction
86217
In linear regression, the transformations of explanatory variables is done to have maximum correlation with the dependent variable.  What is the best measure of choosing between multiple transformations in logistic regression as dependent variable is binary and not continuous?  The end goal is to maximize the lift (predictive power) of the model.
28205
In answering  question John Christie suggested that the fit of logistic regression models should be assessed by evaluating the residuals.  I'm familiar with how to interpret residuals in OLS, they are in the same scale as the DV and very clearly the difference between y and the y predicted by the model.  However for logistic regression, in the past I've typically just examined estimates of model fit, e.g. AIC, because I wasn't sure what a residual would mean for a logistic regression.  After looking into  a little bit I see that in R there are five types of glm residuals available, c("deviance", "pearson", "working","response", "partial").  The help file refers to:     Davison, A. C. and Snell, E. J. (1991) , eds. Hinkley, D. V., Reid, N. and Snell, E. J., Chapman &amp; Hall.     I do not have a copy of that.  Is there a short way to describe how to interpret each of these types?  In a logistic context will sum of squared residuals provide a meaningful measure of model fit or is one better off with an Information Criterion?
17699
I'm doing regression using Random Forests for predicting prices based on several attributes. Code is written in Python using Scikit-learn.  How do you decide whether you should transform your variables using exp/log before using it to fit the regression model? Is it necessary when using an Ensemble approach such as Random Forest?
88907
This is for my hw, and if anyone can solve the first part of the question it will be great. Here is the question:  Assume a two-class problem with equal a priori class probabilities and Gaussian class-conditional densities as follows:  $$p(x\mid w_1) = {\cal N}\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix},\begin{bmatrix} a &amp; c \\ c &amp; b \end{bmatrix}\right)\quad\text{and}\quad p(x\mid w_2) =  {\cal N}\left(\begin{bmatrix} d \\ e \end{bmatrix},\begin{bmatrix} 1 &amp; 0 \\ 0 &amp; 1 \end{bmatrix}\right)$$  where $ab-c^2=1$.   Find the equation of the decision boundary between these two classes in terms of the given parameters, after choosing a logarithmic discriminant function. Determine the constraints on the values of a, b, c, d and e, such that the resulting discriminant function results with a linear decision boundary.
69991
I am learning logistic regression from The elements of statistical learning: data mining, inference, and prediction, by Trevor Hastie, Robert Tibshirani, Jerome H. Friedman.  Suppose $G$ is a random variable representing response taking class label in $\{1,\ldots,K \}$, and $X$ is a random vector representing predictor taking values in $\mathbb{R}^n$.  After modeling $P(G = k|X = x; \theta), k=1,\ldots,K, x \in \mathbb{R}^n$ by logistic model in , the parameter $\theta$ is estimated from samples $\{(x_i,k_i), i=1,\ldots,N \}$ in section :     Logistic regression models are usually ﬁt by maximum likelihood, using   the conditional likelihood of $G$ given $X$. Since $P(G|X)$ completely   specifies the conditional distribution, the multinomial distribution   is appropriate. The log likelihood for $N$ observations is  $$ℓ(θ) = \sum_{i=1}^N \log P(G = k_i|X = x_i; \theta) $$   My question is how the estimation method is maximum likelihood.   Generally speaking, MLE is to estimate parameter $\theta$ of a single distribution $p(z;\theta)$ from its i.i.d. samples $\{z_i, i=1,\ldots,N \}$ by maximizing $\log p(x_1,\ldots, x_N; \theta) = \sum_{i=1}^N \log p(x_i; \theta) $, the log of joint distribution of the samples.  The estimation for logistic regression as in the quote seems not using MLE, because:   $P(G = k|X = x_i; \theta), i=1,\ldots,N $ are $N$ different distributions, instead of just one distribution as for MLE; in $ℓ(θ) = \sum_{i=1}^N \log P(G = k|X = x_i; \theta) = \log     \prod_{i=1}^N P(G = k|X = x_i; \theta)$, I wonder how to explain that "${ (k_i|x_i), i=1,\ldots,N }$" are independent and therefore $\prod_{i=1}^N P(G = k_i|X = x_i; \theta)$ is their joint distribution, as I don't know how to understand $(k_i|x_i)$ which seems not make sense.   My guess answer is that $\{(x_i,k_i), i=1,\ldots,N \}$ are assumed to be i.i.d. samples of the single distribution for $(X,G)$. So MLE for the distribution of $(X,G)$ is to maximize $$ \log P((x_1,k_1),\ldots,(x_N,k_N))  = \log \prod_{i=1}^N P(x_i,k_i)  $$ $$ = \log \prod_{i=1}^N P(k_i | x_i) P(x_i) = \log \prod_{i=1}^N P(k_i | x_i) + \log \prod_{i=1}^N P(x_i) $$ $P(k_i | x_i)$ is parameterized by $\theta$ as $P(k_i | x_i;\theta$, while $P(x_i)$ is assumed to be independent of $\theta$, so to solve for $\theta$ by MLE for the distribution of $(X,G)$ is equivalent to maximize $\log \prod_{i=1}^N P(k_i | x_i; \theta)$. I wonder if this is a reasonable explanation, and what yours are?  Thanks and regards!
19626
If you have read  tutorial about CRF, on page 4 under the section Classification, it wants to relate CRF to Logistic Regression (or Maximum Entropy, as it is known by this name in NLP community).   The formulation of Logistic Regression is said to be  $ p(y|x) = \frac{1}{Z(X)}exp \{ \lambda_y + \sum_{j=1}^{K}\lambda_{y,j}x_{j}\} $.  On the other hand, on the Wikipedia  for logistic regression, the formulation is  $ f(t) = \frac{1}{1+e^{-t}} $  and the classifier's formulation is the explanation of this formulation which is derived from Logistic Function.  My question is, first, how Logistic Regression and Maximum Entropy Classifier are related to each other and are identical?   Second, the CRF tutorial says Logistic Regression (or Maximum Entropy Classifier) is simple CRF model, how?