Measuring Information Resource Value, Part 2: Quantitative Metrics

In Part 1 of the Measuring Information Resource Usage (MIRV) series I explained why it's a bad idea to rely on vendor-supplied usage and user data as the basis for measuring the value (i.e. ROI) of a resource.

We now need to review the two categories of metrics you will need to use to determine the value of a resource, both of which you can and should gather internally.

There are two broad categories of metrics: Quantitative and Qualitative. As a rule, Quantitative metrics capture processes, and Qualitative metrics capture impact.

In this post I review Quantitative metrics.

Quantitative metrics are helpful because, as numeric data, they can easily be converted into analyses that communicate the reach, cost and usage of a resource.

Quantitative metrics are typically of three kinds: 

Reach Metrics

Reach metrics show the footprint of a resource (or its data) within your organization. This is as simple as the number of users for a product: YourCo subscribes to Horizon Research and has 5 licenses to access their website. The licenses are assigned to 5 different people and the seats are non-transferable, i.e. they can’t be shared (nor can the content obtained via these seats).

Cost Metrics

Cost metrics are simply the cost to subscribe and own a resource. YourCo’s Horizon Research package costs $60,000 per year. Maintenance and ownership costs are de minimis, as the content is hosted on the Horizon Research website. Total Cost is $60K.

At this point you can already come up with some modestly useful analyses:

Horizon Research cost: $60,000 per year. Number of users: 5.

Cost per user: $12,000/yr. or $1,000 per month.

Seats per Employee: You have 100 employees, so there’s 1 seat for every 20 employees, and the content and licenses can’t be shared internally.

These are very simple metrics, and easily calculated. But they tell us nothing about what a user is using, how he is using it, and to what effect.

Usage Metrics

Here’s where things get fun, and where you can start determining what and how a subscriber is using a given resource. Things like page views, log-ins, time spent, downloads. 

Let's apply some of these to YourCo's Horizon Research subscription.

For our users, we can see there's wide variance in usage and in how they use the product. Joan, clearly, is a casual user with only a handful of logins. Meanwhile, Charlie is a true power user, averaging almost 2 logins a day, and tens of thousands of page views and downloads.

To provide additional analysis you can create ratios from these stats for additional insight into how people are using a resource:

The numbers tell you a slightly different story than the raw data. David, who only logged in 30 times over 9 active months, viewed far more pages per login than any other user - something you might miss just using the raw data in Table 1. We also see that casual user Joan views almost as many pages as Charlie, despite the fact that she rarely uses the service. 

Putting it All Together

The real fun begins when you combine Reach, Cost and Usage metrics in some really clever and 
creative ways: 

Here we can the see economics of each user's activity. It's quite clear from a pure quantitative basis that Joan is paying a lot for this service relative to her usage. The other users' cost ratios are quite similar and the unit costs are low (3 of 5 users have cost / page view under a buck). An observer could easily be convinced that the information service manager should see about reassigning Joan's seat, or upon renewal, simply get 4 seats instead of 5. 

This would be a mistake. While these quantitative metrics are valuable, they don't tell the whole story. 

We have lots of analysis here on what's being used, by whom, when, and how, and what the nominal cost associated with this usage is, but we don't know to what effect. We don't know the business impact of this usage. In other words: we don't have enough information to determine the actual ROI of this resource for each user. 

What are the metrics that will help you with that? That's where Qualitative metrics come into play, which will be addressed in Part 3.  

- Kevan Huston