Back to All Events

"Lies, Damned Lies, and Accuracy Metrics" by Joe Morrison (Azavea)

This session is a beginner’s guide to:

  • Gaining fluency in common statistics used to describe machine learning models and their limitations.

  • Understanding how they can be manipulated, both intentionally and unintentionally.

  • Seeing some real examples of accuracy metrics done right and wrong.

Advances in machine learning technology, especially the sub-field of deep learning, have dramatically accelerated the adoption of computer vision methods in nearly every industry. In the world of drones and satellite imagery, companies are being formed to help with everything from counting trees in an orchard, to calculating the volume of stockpiles, to estimating the rust on aging infrastructure. However, describing the accuracy of a machine learning model is far less straightforward than many of these companies make it seem when advertising their work. In practice, statistical measures used to evaluate machine learning models are often more misleading than they are helpful.

This talk will use real examples projects that leveraged machine learning and drone or satellite imagery to automate a task--these examples will be used to illustrate how different evaluation techniques can tell different stories about the same data, and to suggest best practices for portraying the accuracy of a model... accurately.