Measuring Information Resource Value, Part 2: Quantitative Metrics

In Part 1 of the Measuring Information Resource Usage (MIRV) series I explained why it's a bad idea to rely on vendor-supplied usage and user data as the basis for measuring the value (i.e. ROI) of a resource.

We now need to review the two categories of metrics you will need to use to determine the value of a resource, both of which you can and should gather internally.

There are two broad categories of metrics: Quantitative and Qualitative. As a rule, Quantitative metrics capture processes, and Qualitative metrics capture impact.

In this post I review Quantitative metrics.

Quantitative metrics are helpful because, as numeric data, they can easily be converted into analyses that communicate the reach, cost and usage of a resource.

Quantitative metrics are typically of three kinds: 

Reach Metrics

Reach metrics show the footprint of a resource (or its data) within your organization. This is as simple as the number of users for a product: YourCo subscribes to Horizon Research and has 5 licenses to access their website. The licenses are assigned to 5 different people and the seats are non-transferable, i.e. they can’t be shared (nor can the content obtained via these seats).

Cost Metrics

Cost metrics are simply the cost to subscribe and own a resource. YourCo’s Horizon Research package costs $60,000 per year. Maintenance and ownership costs are de minimis, as the content is hosted on the Horizon Research website. Total Cost is $60K.

At this point you can already come up with some modestly useful analyses:

Horizon Research cost: $60,000 per year. Number of users: 5.

Cost per user: $12,000/yr. or $1,000 per month.

Seats per Employee: You have 100 employees, so there’s 1 seat for every 20 employees, and the content and licenses can’t be shared internally.

These are very simple metrics, and easily calculated. But they tell us nothing about what a user is using, how he is using it, and to what effect.

Usage Metrics

Here’s where things get fun, and where you can start determining what and how a subscriber is using a given resource. Things like page views, log-ins, time spent, downloads. 

Let's apply some of these to YourCo's Horizon Research subscription.

For our users, we can see there's wide variance in usage and in how they use the product. Joan, clearly, is a casual user with only a handful of logins. Meanwhile, Charlie is a true power user, averaging almost 2 logins a day, and tens of thousands of page views and downloads.

To provide additional analysis you can create ratios from these stats for additional insight into how people are using a resource:

The numbers tell you a slightly different story than the raw data. David, who only logged in 30 times over 9 active months, viewed far more pages per login than any other user - something you might miss just using the raw data in Table 1. We also see that casual user Joan views almost as many pages as Charlie, despite the fact that she rarely uses the service. 

Putting it All Together

The real fun begins when you combine Reach, Cost and Usage metrics in some really clever and 
creative ways: 

Here we can the see economics of each user's activity. It's quite clear from a pure quantitative basis that Joan is paying a lot for this service relative to her usage. The other users' cost ratios are quite similar and the unit costs are low (3 of 5 users have cost / page view under a buck). An observer could easily be convinced that the information service manager should see about reassigning Joan's seat, or upon renewal, simply get 4 seats instead of 5. 

This would be a mistake. While these quantitative metrics are valuable, they don't tell the whole story. 

We have lots of analysis here on what's being used, by whom, when, and how, and what the nominal cost associated with this usage is, but we don't know to what effect. We don't know the business impact of this usage. In other words: we don't have enough information to determine the actual ROI of this resource for each user. 

What are the metrics that will help you with that? That's where Qualitative metrics come into play, which will be addressed in Part 3.  

- Kevan Huston

Measuring Information Resource Value, Part 1: Vendor Supplied Data

One of the guiding principles of this blog is that you cannot determine the value of an information resource if you can't measure its usage or who is using it.

This seems axiomatic, but in practice the problem is more complicated than it first appears:

• What do we mean by measure?
• What do we mean by user? 
• What do we mean by usage? 
• What do we mean by value? 

I will explore each of these concepts in subsequent posts. But to begin, we must consider a simple question: should you rely on the usage data your vendors give you?

Simply put: no! Whenever possible, do not rely on vendor-supplied usage data.

This is not to disparage vendors qua vendors or to suggest there's anything nefarious going on. Your vendors are your partners. They aren't padding your usage stats or anything like that. We're on the same side. Usually.

So what's the problem?

First, vendors do not capture users and usage in the same way, which makes apples to apples comparisons of competing vendors much more difficult. Vendor A may have a different definition of "user" from Vendor B.  There are many ways to define user: is the person merely registered to use the service? Does he count as a user even if he doesn't log in for months on end? What about an employee who simply signed up for marketing collateral but didn't register? Some vendors will (somewhat dishonestly, imo) characterize these people as "users" - particularly if you have an enterprise license to the product.

Defining "usage" is even more complicated than defining users. Some vendors may include newsletter referral clicks as a page view while another vendor may not. Does a user who simply logs in count as usage? Did they visit any pages? Download any content? Some vendors count web page views and downloads differently; some count them the same. Some vendors log records downloaded when you download Excel files; others do not. Some vendors measure time spent, while others simply measure page views. Some measure both.

Second, vendor-only usage data is unreliable or incomplete - some vendors simply may not have the metadata you need or be able to deliver it with the frequency you need it. This is the most important issue in my mind: you don't have the data you need to determine the value of the resources you're buying. Simple as that.

The basic issue with vendor supplied data is this: in order to properly assess value, you can't rely on usage and user data from different sources. And you can't rely on data sources that don't have the metrics you need.

You need to collect your own data. Vendor data won't cut it.

In Part 2 I will address the various means by which we can capture the metrics you need -- in other words, measure your products.

- Kevan Huston

Are Enterprise Subscriptions Worth it?

Managers are often given the opportunity to contract for "enterprise" or "group" level subscriptions to web-based resources like MergerMarket or Pitchbook.

These subscriptions at first glance appear attractive: there's no per user or flat-rate cost for each seat you use.

Suppose a vendor comes to you with two contract options:

Per-user: $150/mo. per user, no tiered pricing (each new user is $150 pro rata);
Group: $2,000/mo. for Group access. The Group has 20 people in it. That's $100/user per month!

It's a no-brainer: simple math tells you to take Option 2.

No so fast.

Not all your users in the group have roles that benefit from the service. You may provision each person with the resource, but how many actually need the service? And how many will use it? How often?

Let's suppose you have a time machine that allows you to zoom ahead a year and see how the product ends up being used. Your future data shows that while you may have 20 subscribers, only 5 people are front office, and of those, only 4 people used the service at least weekly.

Your data are telling you that of the 20 people licensed for the service, only 4 are meaningful ROI+ users.

So if you were on a per-seat license, you'd be paying $600/mo. versus $2,000 a month for the group license.

With Option 1 you're paying $150 a month per user for four active users. With Option 2 you're paying $500 a month per user for your four active users.

This analysis tells you to take Option 1.

Absent a time machine, what should you do?

My recommendation is to start small with a limited, preferably per seat, license. Your unit costs may be higher in year 1, but you will have good qualitative data on the utility of the service, and, using a service like Research Monitor or OneLog, you'll have quantifiable data on usage patterns as well. Word of mouth from subscribers will potentially drive interest from other members of the Group, thereby justifying a switch to a group license upon renewal.

Start small, force the vendor to demonstrate actual value, before you go enterprise.

And remember, you can always repaper a contract before the term ends! If it really looks like the ROI is there for a group license, renegotiate - nothing is written in stone.

- Kevan Huston