Measuring Information Resource Value, Part 1: Vendor Supplied Data

One of the guiding principles of this blog is that you cannot determine the value of an information resource if you can't measure its usage or who is using it.

This seems axiomatic, but in practice the problem is more complicated than it first appears:

• What do we mean by measure?
• What do we mean by user? 
• What do we mean by usage? 
• What do we mean by value? 

I will explore each of these concepts in subsequent posts. But to begin, we must consider a simple question: should you rely on the usage data your vendors give you?

Simply put: no! Whenever possible, do not rely on vendor-supplied usage data.

This is not to disparage vendors qua vendors or to suggest there's anything nefarious going on. Your vendors are your partners. They aren't padding your usage stats or anything like that. We're on the same side. Usually.

So what's the problem?

First, vendors do not capture users and usage in the same way, which makes apples to apples comparisons of competing vendors much more difficult. Vendor A may have a different definition of "user" from Vendor B.  There are many ways to define user: is the person merely registered to use the service? Does he count as a user even if he doesn't log in for months on end? What about an employee who simply signed up for marketing collateral but didn't register? Some vendors will (somewhat dishonestly, imo) characterize these people as "users" - particularly if you have an enterprise license to the product.

Defining "usage" is even more complicated than defining users. Some vendors may include newsletter referral clicks as a page view while another vendor may not. Does a user who simply logs in count as usage? Did they visit any pages? Download any content? Some vendors count web page views and downloads differently; some count them the same. Some vendors log records downloaded when you download Excel files; others do not. Some vendors measure time spent, while others simply measure page views. Some measure both.

Second, vendor-only usage data is unreliable or incomplete - some vendors simply may not have the metadata you need or be able to deliver it with the frequency you need it. This is the most important issue in my mind: you don't have the data you need to determine the value of the resources you're buying. Simple as that.

The basic issue with vendor supplied data is this: in order to properly assess value, you can't rely on usage and user data from different sources. And you can't rely on data sources that don't have the metrics you need.

You need to collect your own data. Vendor data won't cut it.

In Part 2 I will address the various means by which we can capture the metrics you need -- in other words, measure your products.

- Kevan Huston