I have gone through a tone of links to understand the concept of Kernel linear and rbf but it's still not clear to me along with gamma and C values (I do know for linear kernel we only use C value). From what I have read if the number of features are greater it's better to use linear and if the features are less rbf should be used. I have total 821 instances in data and 19 features and 1 class label, which kernel should I use? Also please explain to me gamma and C in possible easiest way. Thanks
88596
I am trying to fit a SVM to my data. My dataset contains 3 classes and I am performing 10 fold cross validation (in LibSVM):   ./svm-train -g 0.5 -c 10 -e 0.1 -v 10 training_data   The help thereby states:  -c cost : set the parameter C of C-SVC, epsilon-SVR, and nu-SVR (default 1)   For me, providing higher cost (C) values gives me higher accuracy. What does C in SVM actually mean? Why and when should I use higher/lower values (or the LibSVM given default value) of C?
101205
I am seeking clarification on why both the vector $(X_{(1)},X_{(n)})^T$ and $\max\{-X_{(1)},X_{(n)}\}$ are sufficient for $\operatorname{Unif}(-\theta,\theta)$, but only $\max\{-X_{(1)},X_{(n)}\}$ is minimal sufficient, as stated .   Can this be explained by the fact that while both can describe the data adequately, $\max\{-X_{(1)},X_{(n)}\}$ is of lesser dimension and is thus minimal sufficient?    Using the definition of minimal sufficiency ($T(X)$ is minimal sufficient if $T(x)=T(y) \iff \frac{\mathcal{L}(x;\theta)}{\mathcal{L}(y;\theta)}$ does not depend on $\theta$), I run into issues as with either choice of statistic, I need to analyze $$\frac{\mathbb{1}_{[\max\{-X_{(1)},X_{(n)}\}&lt;\theta]}}{\mathbb{1}_{[\max\{-Y_{(1)},Y_{(n)}\}&lt;\theta]}}$$ or $$\frac{\mathbb{1}_{[-\theta&lt;X_{(1)}]}\mathbb{1}_{[X_{(n)}&lt;\theta]}}{\mathbb{1}_{[-\theta&lt;Y_{(1)}]}\mathbb{1}_{[Y_{(n)}&lt;\theta]}}$$ which may be $\frac{0}{0}$.  Even when not $\frac{0}{0}$, I'm having trouble seeing why the former is not dependent on $\theta$, but the latter is dependent on $\theta$ if $T(X)=T(Y)$.
50496
I've seen the other thread  but I don't think the answer satisfied the actual question.  What I have continually read is that Naive Bayes is a linear classifier (ex: ) (such that it draws a linear decision boundary) using the log odds demonstration.  However, I simulated two Gaussian clouds and fitted a decision boundary and got the results as such (library e1071 in r, using naiveBayes())   As we can see, the decision boundary is non-linear.  Is it trying to say that the parameters (conditional probabilities) are a linear combination in the log space rather than saying the classifier itself separates data linearly?
6014
There is the notion of Generalized RBF Kernels, for example in "Towards Optimal Bag-of-Features for Object Categorization and Semantic Video Retrieval" from Jiang (1) or in formula (2.72) in  (2).  They define: Gen. RBF Kernel: $e^{-pd(x,y)}$, where (1) says d can be any distance function and (2) requires it to be a metric.  I now have been wondering for a long time: Is a function defined by $e^{-pd(x,y)}$ where d is a metric always a Mercer's Kernel? I think in (2) it sounds like it, but if it were true, not many people know of it, since they are still looking to prove that $e^{-EMD(x,y)}$ is a kernel, where EMD = Earth Mover's Distance, which is a metric.  If its not a Mercer's Kernel - do you know of any counter example?
104338
I have a generalized linear model that adopts a Gaussian distribution and log link function. After fitting the model, I check the residuals: QQ plot, residuals vs predicted values, histogram of residuals (acknowledging that due caution is needed). Everything looks good. This seems to suggest (to me) that the choice of a Gaussian distribution was quite reasonable. Or, at least, that the residuals are consistent with the distribution I used in my model.   Q1: Would it be going too far to state that it validates my choice of distribution?  I chose a log link function because my response variable is always positive, but I'd like some sort of confirmation that it was a good choice.   Q2: Are there any tests, like checking the residuals for the choice of distribution, that can support my choice of link function? (Choosing a link function seems a bit arbitrary to me, as the only guidelines I can find are quite vague and hand-wavey, presumably for good reason.)
33005
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.  I understand the difference between OHE, LabelEncoder and DictVectorizor in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.    Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?
84273
I have built the following model:  def create_model(conv_kernels = 32, dense_nodes = 512):      model_input=Input(shape=(img_channels, img_rows, img_cols))     x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)     x = Activation('relu')(x)     x = MaxPooling2D(pool_size=(2, 2))(x)     x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)     x = Activation('relu')(x)     x = MaxPooling2D(pool_size=(2, 2))(x)     x = Dropout(0.25)(x)     x = Flatten()(x)      conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)      x1 = Dense(nb_classes, activation='softmax')(conv_out)     x2 = Dense(nb_classes, activation='softmax')(conv_out)     x3 = Dense(nb_classes, activation='softmax')(conv_out)     x4 = Dense(nb_classes, activation='softmax')(conv_out)      lst = [x1, x2, x3, x4]      model = Model(inputs=model_input, outputs=lst)     sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)     model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])      return model   When I do a prediction:  model.predict(X_test)   it works properly. However, when I want to get prediction probability like this:  model.predict_proba(X_test)   my model has no predict_proba function. Why not? Is it because of the multiple-output nature of the model?
100600
I have a very basic question, but it's not really clear to me. When would one choose logistic regression over SVM? Maximum margin property seems more justifiable then whatever log. reg does. Is it because scores or interoperability?
90200
I have been told by a professor that it is not possible to combine different activation functions within a neural network.  And I can't find any examples of anyone doing this.  However, I cannot find any good explanation why.   Conceptually, it seems to make sense:  suppose I have a set of data on some population.  Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data.  We could imagine a logistic model that maps my population data into the categorical variable.  Now suppose I'm training a neural net with one hidden layer.  It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course).  Meanwhile, all the other nodes have a linear activation function.  And then output would be a linear function of all the nodes in the hidden layer.    I have no reason to assume this would improve prediction error.  This is what I want to know:  is it possible to estimate such a model using standard approaches?
85576
I am starting to study Neural Network. I want to build a MLP where I will feed it:   3 features which are continuous one feature which is categorical (48 classes)   How can I do this?  Before adding the categorical feature, I was using 'relu' activation function and 'lbfgs' as a solver in MLPRegrssor in sklearn.  I was wondering: how can I integrate this variable into the network? Should I do one-hot encoding? Should I keep relu as activation function or should I try sigmoid, softmax or something like that?
110463
How can I prove that linear combination of two kernel functions is also a kernel function?  \begin{align} k_{p}( x, y) = a_1k_1( x, y) + a_2k_2(x,y) \end{align}  given $k_1(,)$ and $k_2(,)$ are valid kernel functions.  In general to prove any such results involving dot product , cascading.. etc. , what methodology can be followd to prove RHS is a kernel function given k's in LHS all are kernel?
14190
We find the cluster centers and assign points to k different cluster bins in  which is a very well known algorithm and is found almost in every machine learning package on the net. But the missing and most important part in my opinion is the choice of a correct k. What is the best value for it? And, what is meant by best?  I use MATLAB for scientific computing where looking at silhouette plots is given as a way to decide on k . However, I would be more interested in Bayesian approaches. Any suggestions are appreciated.
72831
have a couple of coefficients which meaning I can not understand.  gamma : float, optional (default=0.0) Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’. If gamma is 0.0 then 1/n_features will be used instead.  and  coef0 : float, optional (default=0.0) Independent term in kernel function. It is only significant in ‘poly’ and ‘sigmoid’.  I tried to google for gamma in support vector machines, but have not found anything relevant. Can anyone explain me what is the meaning of these parameters and reasonable value for them.
56794
I have trained two models using sklearn library in python.. My dataset was about 750 features, 250 features per class (three classes), I trained only one feature dimension (1-D array). This are the results:   SVM   Between training process and testing process (0.20%) I got: 0.029801 sg    KNN   Between training process and testing process (0.20%) - 0.0074096 sg  As we can see K-NN got a shorter execution time ≈ 7 milliseconds and SVM 29.801 milliseconds.  I'm interested in knowing what of this two models are more complex computationally. According to [1] the complexity of SVM (LibSVM) is O(n^3) Sklearn is using libsvm like backend or like solver for svm problems (linear and non-linear)  According to [2] the complexity of K-NN is O(nd)  "Since the large O notation only gives a higher asymptotic dimension, and not an asymptotically adjusted upper bound, we can make statements that at first glance seem incorrect, but that are technically correct. For example, it is absolutely correct to say that the binary search is executed at a time O (n),  That's because the execution time grows no faster than a constant multiplied by n. In fact, it grows slower." [3]  What is more complex?  O(n^3) or O(nd) and Why?  Since my point of view KNN is more less complex in time execution that SVM model. thanks so much.  [1]  [2]  [3]
43277
Why are activation functions of  (ReLU) considered non-linear?   $$ f(x) = \max(0,x)$$  They are linear when the input is positive and from my understanding to unlock the representative power of deep networks non-linear activations are a must, otherwise the whole network could be represented by a single layer.
75064
I'm trying to identify the best model to predict the prices of automobiles, using the prices and features available on automobile classified advertisement sites.  For this I used a couple of models from the scikit-learn library and neural network models from pybrain and neurolab. The approach I used so far is to run a fixed amount of data through some models(machine learning algorithms) and compare there $R^2$ values which was calculated with the scikit-learn metrics module.   Is $R^2$ a good method to compare the performance of different models? Although I got quite acceptable results for models such as Elastic net and Random forests I got very poor $R^2$ values for Neural network models,so is $R^2$ an appropriate method for evaluating neural networks (or non-linear methods)?
87798
I am currently using an SVM with a linear kernel to classify my data.  There is no error on the training set.  I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$).  This did not change the error on the test set.  Now I wonder: is this an error caused by the ruby bindings for libsvm I am using () or is this theoretically explainable?  Should the parameter $C$ always change the performance of the classifier?