Support Vector Machines
SVCs are a classification technique that is commonly used in some of the work I do. This generally involves considering the case of Linear discriminant analysis (LDA) for two classes, with data that is perfectly separable.
- Of the candidate separating planes, the “best” one is the one that is “furthest” from each set of observations
- The question then comes to how one can find the best separating plane
- Class separation: basically, we are looking for the optimal separating hyperplane between the two classes by maximizing the margin between the classes’ closest points - the points lying on the boundaries are called support vectors, and the middle of the margin is our optimal separating hyperplane;
- Overlapping classes: data points on the “wrong” side of the discriminant margin are weighted down to reduce their influence (“soft margin”);
- Nonlinearity: when we cannot find a linear separator, data points are projected into an (usually) higher-dimensional space where the data points effectively become linearly separable (this projection is realised via kernel techniques);
- Problem solution: the whole task can be formulated as a quadratic optimization problem which can be solved by known techniques.