Jerome Friedman’s paper titled, “On bias, variance, 0/1-loss, and the curse-of-dimensionality”, provides a great insight in to the way classification errors work. The paper throws light on the way bias and variance conspire to make some of the highly biased methods perform well on test data. Naive Bayes works, KNN works and so do many such classifiers that are highly biased. This paper gives the actual math behind classification error and shows that the additive nature of bias and variance that holds good for estimation error cannot be generalized to classification error.

[youtube https://www.youtube.com/watch?v=uaWA2GbcnJU?rel=0]

Via Letters Of Note Back in 2006, a group of students at Xavier High School in New York City (one of whom, “JT,” submitted this letter) were given an assignment by their English teacher, Ms. Lockwood, that was to test their persuasive writing skills: they were asked to write to their favourite author and ask him or her to visit the school. Five of those pupils chose Kurt Vonnegut. His thoughtful reply, seen below, was the only response the class received.

Via The Rise of Machines There is another interesting difference that is worth pondering. Consider the problem of estimating a mixture of Gaussians. In Statistics we think of this as a solved problem. You use, for example, maximum likelihood which is implemented by the EM algorithm. But the EM algorithm does not solve the problem. There is no guarantee that the EM algorithm will actually find the MLE; it’s a shot in the dark.

[youtube https://www.youtube.com/watch?v=jk9S3RTAl38?rel=0]