It was hit or miss with Sir William Thomson, Baron Kelvin of Largs (1824-1907). Thanks to his work in thermodynamics, we understand that there are absolute temperatures, and we use his eponymous scale to express those. An Irishman by birth, he was opposed to Irish home rule, which pleased Queen Victoria and, along with his scientific achievements, gained him British knighthood. He declared X-rays a hoax, saw no future in radio, and flatly stated that “heavier than air flying machines are impossible.”
But it’s the measurement thing that has really caught on for most of us; Lord Kelvin is the one who said, "To measure is to know," and, more famously, "If you cannot measure it, you cannot improve it."
The idea of using objective measurement as the basis for objective, fact-driven analysis and evaluation persists as a fundamental tenet of the hard and social sciences. By focusing on numbers, the theory goes, we focus on facts; observations without numbers are merely opinion.
Over the years, science and technology have made more things observable and measurable. As new means of observation have become available, new methods of calculating the changes we can observe have emerged. Today, our fascination with numbers and measurements and what they tell us shows up, at least in the world of Internet-based communications and social media, as metrics, and even more recently, analytics.
E-Learning professionals are a data- and measurement-oriented lot. We want to iterate and improve the learning products we produce, and we prefer to do that based on objective data rather than guesswork. Whether we’re talking about front-end analysis, formative evaluation, post-implementation evaluation or any other form of examination of a communication program, we continue, however, to encounter resistance to looking at the numbers. Our colleagues in the world of Web analytics have been successful in opening the minds of marketing professionals to the value of measurements and their analysis. In the world of e-Learning, we have the opportunity to borrow from their success, while at the same time tailoring their metrics to our needs.
We cannot just wholesale adopt marketing analytics for our own purposes. Regardless of the moniker, the answers we seek and the numbers we analyze to tease them out are only as good as the questions we ask, the measurements we take, and the numbers we evaluate to arrive at our answers. If we want to understand the effectiveness of a Web page design, we have to articulate our definitions of “effective” and “design,” ask questions that prompt measurements related to our concepts, and analyze the resulting numbers in ways that are meaningful to our question.
Sometimes, the questions we ask don’t automatically generate meaningful or useful measurements. Indeed, the questions themselves can be elusive. Take engagement, for example, a highly desired characteristic sought by almost every Web property. How can we determine whether a learning program is engaging, whether the learners are engaged? And what do we want them to engage with – the site’s content, the site’s publisher, other site users, or others unrelated to the site? In defining the term, we begin to understand how important our definition is.
Once a definition is in place, we are in a much better position to figure out what measurements to take. The measurements correspond with the indicators that our definition points to. Those indicators reflect the observable behaviors we would like to see our learners exhibit when they are taking the actions or being the characteristic we seek. In the case of engagement, we might agree that its indicators are:
-
Recency – the last time Learner came to the site
-
Frequency – how often Learner comes to the site
-
Activity – number of actions/interactions Learner took in response to the content
-
Duration – how long Learner stays on the site
-
Virality – whether Learner forwards a link to the site and/or its content to others
-
Ratings – the rating Learner gives the site and/or its content
In examining the numbers that measurements yield, it’s clear that they only become meaningful when analyzed in the context of each other. Lots of visitors to a site is great, but if their visits are brief and they never return, the site is probably not eliciting the engagement that the site owner desires. Likewise, receiving high ratings from users is nice for the ego, but those ratings aren’t helpful to growing an audience if appreciative users don’t tell friends and colleagues about the value they derive from the site in question.
We don’t all have to be “quant guys” or “numbers geeks.” But numbers matter – to our customers, to our learners, and to us. We won’t get the questions or the analysis right every time –Lord Kelvin sure didn’t – but metrics matter, and they are more readily at hand all the time.
