Measuring Information Resource Value, Part 3: Qualitative Metrics

In the second part of the Measuring Information Resource Value series, I laid out the case for gathering quantitative data inputs toward an ROI decision. Now we must look at what I call qualitative metrics.

Where quantitative metrics measure processes, qualitative metrics measure outcomes. I typically divide Qualitative metrics into three categories:

1. Verbatims - these are written testimonials of the value of a service. They can be solicited as part of a formal feedback process, or unsolicited when users simply tell you how much they like a service. 
2. User Surveys - these are carefully designed, methodologically sound instruments you issue to users or a sample of users that gather feedback in a way that can be aggregated, compared and reported on.
3. Productivity Studies - a blend of quant and qual, these metrics assess the time-use and user behavior associated with a resource. These can be survey-based or gathered via network, server or key-stroke means.

I have always argued that information professionals should endeavor to gather measurable (and reportable) data in qualitative analyses. While feedback such as verbatims and anecdotes can be powerful ammunition in support of a purchase, these on their own are insufficient. The reality is that most senior managers respond best to numeric insight.

So how do you assign numeric values to qualitative data? What you're looking for, beyond verbatims, is a way to systematically measure business impact. A user verbatim may read: "this source is tremendously helpful. It has saved me countless hours and has helped us land three deals we wouldn't otherwise have gotten".

How do you record something like this in a way that can be compared, aggregated, analyzed and visualized?

You need to conduct a user survey. This doesn't have to be hard - you're not trying to recreate a Gallup poll or comScore panel! Here's what I recommend.

As part of your renewal process, survey a sample of end users about the product's value. Your survey questions should be laser-focused on outcomes. Don't worry about usage or process stats - you're collecting that already! Some example survey questions:

1. On a scale of 1 to 5, with 5 being the highest, how important would you rate Horizon Research to your job? (Scale)
2. On a scale of 1 to 5, with 5 being the highest, how valuable is Horizon Research to the work product you deliver to clients? (Scale)
3. Your subscription to Horizon costs $12,000 per year. At that price, do you think it is worth renewing this service? (Y/N)
4. Have you landed deals or won business because you have access to Horizon Research? (Y/N)
5. On a scale of 1 to 5, with 5 being the highest, how much more productive are you because you have access to Horizon Research (Scale)
6. On a scale of 1 to 5, with 5 being the highest, how satisfied are you with the quality of the data you get from Horizon? (Scale)

Remember, you're focused on two things with these surveys: business impact and measurable data. Scale and Y/N questions like the above measure impact and be easily aggregated and reported on.

There are limitations to surveys like these of course. They are user reported, and people tend to over-report the value of a resource. Best example: Bloomberg Terminals. Ask any owner of a Bloomberg any of the above questions and the answers will be all 5s and Yes, Yes, Yes. You need to cross reference this feedback with quantitative metrics: when you look at the actual usage data, you see a very different story - sporadic logins, limited usage, and content that's cheaply available elsewhere.

You can also verify survey data with other qualitative inputs. Trust, but verify, is the name of the game. A time use productivity study is a great way to do this.

Suppose you're considering whether to renew a service that helps your employees pull and share regulatory filings. The verbatims you've gathered suggest the main value people get from it is time savings. Upon further investigation, you learn that with the service, users are spending 10 minutes/day pulling filings, and 5 minutes/day sharing the filings through the product. A similar, free service on the web has them spending 20 minutes retrieving and 10 minutes sharing filings.

The mean hourly pay for the user base is $60. There are 20 users.

Under the free service, users are spending $20 in time retrieving info and $10 sharing it. That's $400/day in retrieval and $200/day in sharing. $600 a day on this one workflow for your team!

With the paid service, the time spent and thus the cost, is half that: your team is spending $200/day on retrieval and $100 on sharing, or $300/day on this workflow.

Added up over a 200 day work year, your team spends $120,000 in time for this workflow with the free configuration, but only $60,000 in time annually with the paid service.  This is valuable data to complement your verbatims and survey data.

Qualitative inputs are critical inputs for your ROI analyses. Where quantitative usage data measures processes, qualitative inputs like verbatims, surveys, and time use studies are used to show outcomes and business impact.

- Kevan Huston

Measuring Information Resource Value, Part 1: Vendor Supplied Data

One of the guiding principles of this blog is that you cannot determine the value of an information resource if you can't measure its usage or who is using it.

This seems axiomatic, but in practice the problem is more complicated than it first appears:

• What do we mean by measure?
• What do we mean by user? 
• What do we mean by usage? 
• What do we mean by value? 

I will explore each of these concepts in subsequent posts. But to begin, we must consider a simple question: should you rely on the usage data your vendors give you?

Simply put: no! Whenever possible, do not rely on vendor-supplied usage data.

This is not to disparage vendors qua vendors or to suggest there's anything nefarious going on. Your vendors are your partners. They aren't padding your usage stats or anything like that. We're on the same side. Usually.

So what's the problem?

First, vendors do not capture users and usage in the same way, which makes apples to apples comparisons of competing vendors much more difficult. Vendor A may have a different definition of "user" from Vendor B.  There are many ways to define user: is the person merely registered to use the service? Does he count as a user even if he doesn't log in for months on end? What about an employee who simply signed up for marketing collateral but didn't register? Some vendors will (somewhat dishonestly, imo) characterize these people as "users" - particularly if you have an enterprise license to the product.

Defining "usage" is even more complicated than defining users. Some vendors may include newsletter referral clicks as a page view while another vendor may not. Does a user who simply logs in count as usage? Did they visit any pages? Download any content? Some vendors count web page views and downloads differently; some count them the same. Some vendors log records downloaded when you download Excel files; others do not. Some vendors measure time spent, while others simply measure page views. Some measure both.

Second, vendor-only usage data is unreliable or incomplete - some vendors simply may not have the metadata you need or be able to deliver it with the frequency you need it. This is the most important issue in my mind: you don't have the data you need to determine the value of the resources you're buying. Simple as that.

The basic issue with vendor supplied data is this: in order to properly assess value, you can't rely on usage and user data from different sources. And you can't rely on data sources that don't have the metrics you need.

You need to collect your own data. Vendor data won't cut it.

In Part 2 I will address the various means by which we can capture the metrics you need -- in other words, measure your products.

- Kevan Huston

Establishing a User Base When Using SSO or IP Authentication

For some widely held services it makes sense to migrate from a password-based access program to a single sign-on (SSO) or IP Authentication program:

• The administrative overhead can be lower: depending on the vendor, you may waste a lot of time getting accurate and current user inventory counts;
• Better access controls as you can immediately deactivate user access without relying on the vendor to cancel user accounts;
• Improved compliance from reduced or eliminated password sharing;
• Improved visibility into who is using what services.

But it's important to ensure you have a plan for establishing an accurate user base in the absence of vendor supplied user lists. Don't simply implement a new access program for a resource without also knowing who is accessing the service, and how often. 

Accurate user and usage data is an absolute must, for several reasons: 

• You can't determine the value of a service if you don't know who is using it, and how often;
• You can't negotiate effectively with the vendor if you don't know how many users are using the service;
• You can't offer training and remediation to users who aren't using a service but who should be.

So how do you do this without usage data from the vendor? 

Many SSO utilities will give you data on who is accessing a website or application, but it may not be as granular as you need, or capture all the metadata you want. And with IP Authentication the situation is even more dire: for the most part you're still stuck with using vendor supplied data. 

Instead, I recommend installing a usage monitoring token on your proxy server. Firms that offer these services include Lucidea, OneLog, ResearchMonitor and H&H. With a usage monitoring service you will have detailed information on who you users are and how they are using the service. When combined with a SSO access program, you're not really compromising anything with respect to establishing a user footprint for such services.

No matter what access program (or programs) you use, it's imperative that you have accurate and reliable user and usage data. Don't switch from vendor-managed access control (passwords) to SSO or IP Authentication without having a plan in place to capture this valuable information.

- Kevan Huston