I am using sklearn library to train the default SVM classifier on a bunch of data. My data has roughly 30 features. But I realize some of these features are "more important" than others. For example, I want the change in features 1, 2, and 3 to greatly impact the learning process, whereas, features 4 through 25 are useful information, but not as critical.   If I just multiply the numbers in features 1 to 3 by say, 100, it won't make much difference because I believe the algorithm normalizes the features anyway. So my question is, how can I give a weight to the features in an SVM classifier, so that some of them influence the learning and prediction more than others?  Here's the very crude version of the code I'm using:  clf = SVC(probability=True, gamma=0.1, C=1)) data = np.array([1, 4, 1, 6, 2, 5, 7, 2, ... ]) target = np.array([0, 1, 0, 0, 1, ... ]) clf.fit(data, target)
123453
I'm building an object detector using HOG features and linear SVM. Some of the regions of the object are more "distinctive" so I would like to give more weight to the features extracted from those regions. (e.g. imagine we want to detect between mugs and glasses, then the most "distinctive" part is the mug's handle)  How could I do this? For sure there should be a more intelligent way than just replicating the features of those "distinctive" regions.
54253
My input variables have different dimensions. Some variables are decimal while some are hundreds. Is it essential to center (subtract mean) or scale (divide by standard deviation) these input variables in order to make the data dimensionless when using random forest?
87798
I am currently using an SVM with a linear kernel to classify my data.  There is no error on the training set.  I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$).  This did not change the error on the test set.  Now I wonder: is this an error caused by the ruby bindings for libsvm I am using () or is this theoretically explainable?  Should the parameter $C$ always change the performance of the classifier?
104586
I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out.  I have a microarray dataset that is classified into two groups and has 1000s of features.  My aim is to get a small number of genes (my features) (10-20) in a signature that I will in theory be able to apply to other datasets to optimally classify those samples.  As I do not have that many samples (&lt;100), I am not using a test and training set but using Leave-one-out cross-validation to help determine the robustness.  I have read that one should perform feature selection for each split of the samples i.e.   Select one sample as the test set On the remaining samples perform feature selection Apply machine learning algorithm to remaining samples using the features selected Test whether the test set is correctly classified Go to 1.   If you do this, you might get different genes each time, so how do you get your "final" optimal gene classifier? i.e. what is step 6.  What I mean by optimal is the collection of genes that any further studies should use.  For example, say I have a cancer/normal dataset and I want to find the top 10 genes that will classify the tumour type according to an SVM.  I would like to know the set of genes plus SVM parameters that could be used in further experiments to see if it could be used as a diagnostic test.
17699
I'm doing regression using Random Forests for predicting prices based on several attributes. Code is written in Python using Scikit-learn.  How do you decide whether you should transform your variables using exp/log before using it to fit the regression model? Is it necessary when using an Ensemble approach such as Random Forest?
17688
I am beginning to learn how to use scikit-learn and I have a hard time choosing the right model.  Here is my dataset: I have 100 persons. Each person was measured three times: baseline, first event and second event. Each measurement had 100 different markers per person that range from 0.1 to 1000. Additionally I have outcome measurements of each event: outcome can be 0, 1 or 2. My task is to find just a few markers (let’s say 10) that can predict outcome with a good accuracy. If I am right it should be: Supervised learning/Classification problem. What model would be the best?  Thanks for your help!
32233
I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max 20-30 for later use with model training (binary classification - increase / decrease).  I was thinking about using random forests for feature ranking. Is it a good idea to use them recursively? For example, let's say in the first round I drop the worst 20%, second too and so on until I get the desired number of features.  Should I use cross-validation with RF? (It's intuitive for me not to use CV because that's pretty much what RF does already.)  Also if I go with random forests should I use them as classifiers for the binary or regressor for the actual increase / decrease to get feature importances?  By the way, the models I would like to try after feature selection are: SVM, neural nets, locally weighted regressions, and random forest. I'm mainly working in Python.
2122
I am trying to use sequentialfs to do some feature selection in matlab. I have huge dimensional data of 22215 features. When I tried to use sequentialfs with svm as classifier so that it selects the best subset of features, it just keeps on running, probably its because of the huge number of dimensions. However, weka does the same thing very quickly. Weka has wrapper filters and it does it so quickly. What sort of heuristic does it use? Even though I tried with 5000 features in weka since it was not taking 22215 features, it gave me results quickly with wrapper filters. What should I do with sequentialfs in matlab  This is the command I am using in matlab  c = cvpartition(yS1,'k',12); opts = statset('display','iter'); [fs, history] = sequentialfs(@SVM_class_fun, X, yS1,'cv', c, 'options', opts);  SVM_class_fun  function err = SVM_class_fun(xTrain, yTrain, xTest, yTest)   model = svmtrain(xTrain, yTrain);   err = sum(svmclassify(model, xTest) ~= yTest); end   Here the dimension of X is 100x22215 where I have 100 examples each of dimension 22215.
88540
In terms of neural network lingo (y = Weight * x + bias) how would I know which variables are more important than others?  I have a neural network with 10 inputs, 1 hidden layer with 20 nodes, and 1 output layer which has 1 node. I'm not sure how to know which input variables are more influential than other variables. What I'm thinking is that if an input is important then it will have a highly weighted connection to the first layer, but the weight might be positive or negative. So what I might do is take the absolute value of the input's weights and sum them. The more important inputs would have higher sums.  So for example, if hair length is one of the inputs, then it should have 1 connection to each of the nodes in the next layer, so 20 connections (and therefore 20 weights). Can I just take the absolute value of each weight and sum them together?
9309
My understanding is that in machine learning it can be a problem if your dataset has highly correlated features, as they effectively encode the same information.  Recently someone pointed out that when you do one-hot encoding on a categorical variable you end up with correlated features, so you should drop one of them as a "reference".  For example, encoding gender as two variables, is_male and is_female, produces two features which are perfectly negatively correlated, so they suggested just using one of them, effectively setting the baseline to say male, and then seeing if the is_female column is important in the predictive algorithm.  That made sense to me but I haven't found anything online to suggest this may be the case, so is this wrong or am I missing something?  Possible (unanswered) duplicate:
106596
I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.  What is the meaning of Gain, Cover, and Frequency and how do we interpret them?  Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters   Are there any other parameters that can tell me more about feature importances?  From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.  I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.  importance_matrix   Output:             Feature         Gain        Cover    Frequence   1:            xxx 2.276101e-01 0.0618490331 1.913283e-02   2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01   3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01   4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01   5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02  214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05 215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05 216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05 217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05 218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05
68079
I'm training a neural network for regression. The input vector consists of $140$ entries.   For each feature vector entry, I calculate both min and max over the training set. The difference max-min of some entries are smaller than 1e-5. I understand I can normalize such feature entries by say MinMaxScalar. However, if the difference max-min is this small, can I discard such entries since they seem to be so insensitive?
33005
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.  I understand the difference between OHE, LabelEncoder and DictVectorizor in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.    Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?
25422
I want to make my classifier prioritise finding true cases (1) even if that means that a lot of the false cases (0) are also classified as true.  Specifically I wish to find the weights to my features that gives at least 97 % of my true cases as true and as a small fraction as possible of my false cases as false.   What should I be looking for in a logistic regression setting? Of course I can just adjust the intercept but I assume that there must be some better way?
49128
Assume I have a dataset for a supervised statistical classification task, e.g., via a Bayes' classifier. This dataset consists of 20 features and I want to boil it down to 2 features via dimensionality reduction techniques such as Principal Component Analysis (PCA) and/or Linear Discriminant Analysis (LDA).  Both techniques are projecting the data onto a smaller feature subspace: with PCA, I would find the directions (components) that maximize the variance in the dataset (without considering the class labels), and with LDA I would have the components that maximize the between-class separation.  Now, I am wondering if, how, and why these techniques can be combined and if it makes sense.  For example:   transforming the dataset via PCA and projecting it onto a new 2D subspace transforming (the already PCA-transformed) dataset via LDA for max. in-class separation   or    skipping the PCA step and using the top 2 components from a LDA.   or any other combination that makes sense.
19289
I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) &gt; 0.5$, we will assign a class 1, otherwise 0 (let this be a binary classification).   My question is, what if I find out, that if I classify the class as 1 also when the probabilities are larger than, for instance 0.2, and the classifier performs better. Is it legitimate to then use this new threshold when doing classification?  I would interpret the necessity for lower classification bound in the context of the data emitting a smaller signal; yet still significant for the classification problem.  I realize this is one way to do it. However, if this is not correct thinking of reducing the threshold, what would be some data transformations, which emphasize individual features in a similar manner, so that the threshold can remain at 0.5?
56299
I am using both random forest and xgboost to examine the feature importance. but i noticed that they give different weights for features as shown in both figures below, for example HFmean-Wav had the most important in RF while it has been given less weight in XGBoost and i can understand why?