Bad Data: Why We Measure the Wrong Things and Often Miss the Metrics that Matter

By Peter Schryvers

'Not everything that counts can be counted, and not everything that can be counted counts' – William Bruce Cameron

Big data has arrived. The concept is so ubiquitous, in fact, that the term may sound anachronistic in just a few more years – 'big data' is just data at this point. Our new obsession with 'datafication' began benignly enough, with data-nerd sports GMs and improved Netflix recommendations, but has metastasized into invasive Facebook harvesting and massive data breaches. In 15 years, we’ve gone from Moneyball to Black Mirror.

Despite the rapid rise and ubiquity of big data today, there is a fundamental question, underlying all of data science, that has so far been mostly ignored: how does the data we choose to collect change our thoughts, values, actions and achievements? After all, every metric we use does not really exist, statistically speaking, until we consciously choose to measure it. The way we measure something – even the choice to simply begin measuring – unavoidably colours how we approach a problem, and often determines whether we solve it or simply transform it into a different problem.

In Bad Data, Peter Schryvers looks at the use and abuse of metrics, including the pitfalls associated with misunderstanding them. There is a dark side to metrics, a blind faith in the power of big data that can lead to death-spiral thinking. He highlights the dangers of this unthinking adherence in our personal, professional, national and global endeavours. Along the way, he shares many stories of metrics-gone-wrong, but unlike similar books on the subject, Bad Data is defined by his dogged pursuit of solutions. Big-data metrics are here to stay; no amount of hand-wringing or cautionary tales is going to put them back in the bottle. Across all spheres of public life – education, health, city development, even the state of our planet – we use metrics to shape our future. As an urban planner, Schryvers has faced countless examples of poorly chosen or constructed metrics, and he has devoted himself to spotting their flaws and solving them.

Bad Data is not a book about statistics, analytics or mathematics. It is a book about metrics – about when they work, when they don’t and when they never will. Ultimately, it seeks to answer a simple question: are we measuring the right thing? Does what we are counting really count?