Understanding human-level performance

Human-level error as a proxy for Bayes error

Medical image classification example:

Goal is to make a diagnosis classification decision.

• Human error = 3%
• Radiologist error = 1%
• Experianced radiologist error = 0.7%
• Team of experianced radiologists = 0.5%

Given above, Human level error can be considered as a proxy for Bayes error, and since Bayes error $\le$ 0.5%, so we can set the human level error as 0.5%.

Error analysis example

For a medical imaging diagnosis example,

• human-level performance as proxy for Bayes error $\begin{cases} \text{1%} \\ \text{0.7%} \\ \text{0.5%} \end{cases}$

                                    $\Bigg\updownarrow$ Avoidable bias 4-4.5%                                          $\Bigg\updownarrow$ Avoidable bias 0-0.5%

• training error        5%                                                                            1%

                                    $\Bigg\updownarrow$ Variance 1%                                                          $\Bigg\updownarrow$ Variance 4%

• dev error              6%                                                                            5%

                               reduce bias                                                                reduce variance

More on how to reduce bias and variance

Summary

Having an estimate of human-level performance gives you an estimate of Bayes error. And this allows you to more quickly make decisions as to whether you should focus on trying to reduce a bias or trying to reduce the variance of your algorithm. And these techniques will tend to work well until you surpass human-level performance, whereupon you might no longer have a good estimate of Bayes error that still helps you make this decision really clearly.