Is it known who originally introduced the term "machine" used in terms like   machine learning, support vector machine or gradient boosting machine?   Could it be defined as an "algorithm that fits a statistical model"?  OLS could then be called "least squares machine"...   Is there an official definition of the word "machine" in statistics?
122015
Why are they called "machines"? Is there an origin to the word "machine" used in this context? (Like the name "linear programming" can be confusing but we know why it is called "programming.")
104586
I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out.  I have a microarray dataset that is classified into two groups and has 1000s of features.  My aim is to get a small number of genes (my features) (10-20) in a signature that I will in theory be able to apply to other datasets to optimally classify those samples.  As I do not have that many samples (&lt;100), I am not using a test and training set but using Leave-one-out cross-validation to help determine the robustness.  I have read that one should perform feature selection for each split of the samples i.e.   Select one sample as the test set On the remaining samples perform feature selection Apply machine learning algorithm to remaining samples using the features selected Test whether the test set is correctly classified Go to 1.   If you do this, you might get different genes each time, so how do you get your "final" optimal gene classifier? i.e. what is step 6.  What I mean by optimal is the collection of genes that any further studies should use.  For example, say I have a cancer/normal dataset and I want to find the top 10 genes that will classify the tumour type according to an SVM.  I would like to know the set of genes plus SVM parameters that could be used in further experiments to see if it could be used as a diagnostic test.
116069
In Element of Statistical Learning, chapter 4-2 about linear regression of an indicator matrix, it is stated that the sum of predictor is equal to 1.  To bring a bit of context: We have $N$ training samples of size $p$ represented by a $N$ x $(p+1)$ matrix (intercept included) $\textbf{X}$. Each training sample $x_i$ belong to a category $g_i \in [| 1, .., K |]$. We use the $N$ x $K$ indicator matrix for the training samples which contains only one 1 per row. We try to fit a linear regression model for the parameter $\textbf{B}$ and we get $\hat{\textbf{B}} = (\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\textbf{Y}$.  For a new data, we compute the fitted output $\hat{f}(x)^T = (1,x^T)\hat{\textbf{B}}$ and the category is then $arg \max_{k \in [| 1, .., K |]} \hat{f}_k(x)$  They say that $\sum_{k \in [| 1, .., K |]} \hat{f}_k(x) = 1$ for any $x$  as long as there is an intercept in the model.  My question is thus why? I try to use the coefficient relation from the regression derivation ($\textbf{X}^T(\textbf{Y} - \textbf{X}\hat{\textbf{B}}) = 0$) but without success.
91934
What is the difference between a one-vs-all and a one-vs-one SVM classifier?  Does the one-vs-all mean one classifier to classify all types / categories of the new image and one-vs-one mean each type / category of new image classify with different classifier (each category is handled by special classifier)?  For example, if the new image to be classified into circle, rectangle, triangle, etc.
99247
Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:  +----------+----------+-------+-----------+ | FeatureA | FeatureB | Label | Certainty | +----------+----------+-------+-----------+ |        2 |        3 | A     | 50%       | |        3 |        1 | B     | 80%       | |        1 |        1 | A     | 100%      | +----------+----------+-------+-----------+   How could we train a machine learning model on this? Thanks.
102428
In the RMSProp algorithm (And similar algorithms) that are used in Machine Learning in the subject of Adaptive Learning Rates, the squares of the gradients are used in the algorithm step.  Is there motivation for why the squares of the gradients are used?  Thanks!
12415
K-means is probably one of the most used algorithms for clustering. I was looking for bibliography for its first use, but it has been around a lot, so what's the first one?  Also, when was the algorithm first named "k-means"?
6353
Every textbook I've seen so far describes ML algorithms and how to implement them.  Is there also a textbook that builds theorems and proofs for the behaviour of those algorithms? e.g. stating that under conditions $x,y,z$, gradient descent will always lead to $A,B,C$?
25786
I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural networks work, it seems like they commonly use another approach to estimate the parameters instead. Why don't we use MLE or is it possible to use MLE at all?
16751
As far as I can tell, Kohonen-style SOMs had a peak back around 2005 and haven't seen as much favor recently. I haven't found any paper that says that SOMs have been subsumed by another method, or proven equivalent to something else (at higher dimensions, anyhow). But it seems like tSNE and other methods get a lot more ink now-a-days, for example in Wikipedia, or in SciKit Learn, and SOM is mentioned more as a historical method.  (Actually, a Wikipedia article seems to indicate that SOMs continue to have certain advantages over competitors, but it's also the shortest entry in the list. EDIT: Per gung's request, one of the articles that I'm thinking about is: . Note that SOM has less written about it than the other methods. I can't find the article that mentioned an advantage that SOMs seem to retain over most other methods.)  Any insights? Someone else asked why SOMs are not being used, and got references from a while ago, and I have found proceedings from SOM conferences, but was wondering if the rise of SVMs or tSNE, et al, just eclipsed SOMs in pop machine learning.  EDIT 2: By pure coincidence, I was just reading a 2008 survey on nonlinear dimensionality reduction this evening, and for examples it mentions only: Isomap (2000), locally linear embedding (LLE) (2000), Hessian LLE (2003), Laplacian eigenmaps (2003), and semidefinite embedding (SDE) (2004).
110068
I was using , implemented in scikit-learn, for my research work. But I have no good understanding of this.   Can anyone please give a simple, good explanation of one class SVM?
48347
Historically speaking, it makes sense to me someone would say run "the computer". Early computers (not a human computer) were mechanical machines with moving parts that could achieve a velocity deemed "running" from the perspective of a human being. I don't know for sure if that is the reason why, but that seems plausible.  But why run the "program", "code" or "software"?
36617
In a SVM with linear kernel, could you explain to me what exactly the C parameter is/represents?  An example why it's important to select a good value for C would also be appreciated.  Thank you.
66966
I usually hear about "ordinary least squares".  Is that the most widely used algorithm used for linear regression?  Are there reasons to use a different one?
114048
While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not imagine an application of a GB that uses linear regression, and in fact when I've performed some tests - it doesn't work. I was testing the most standard approach with a gradient of sum of squared residuals and adding the subsequent models together.  The obvious problem is that the residuals from the first model are populated in such manner that there is really no regression line to fit anymore. My another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.  I also thought about lowering the learning rate or using only a subset of predictors for each iteration, but that could still be summed up to a single model representation eventually, so I guess it would bring no improvement.  What am I missing here? Is linear regression somehow inappropriate to use with Gradient Boosting? Is it because the linear regression uses the sum of squared residuals as a loss function? Are there any particular constraints on the weak predictors so they can be applied to Gradient Boosting?
102043
There seem to be many machine learning algorithms that rely on kernel functions. SVMs and NNs to name but two. So what is the definition of a kernel function and what are the requirements for it to be valid?
44268
When using SVM, we need to select a kernel.  I wonder how to select a kernel. Any criteria on kernel selection?