Measuring Information Resource Value, Part 3: Qualitative Metrics

In the second part of the Measuring Information Resource Value series, I laid out the case for gathering quantitative data inputs toward an ROI decision. Now we must look at what I call qualitative metrics.

Where quantitative metrics measure processes, qualitative metrics measure outcomes. I typically divide Qualitative metrics into three categories:

1. Verbatims - these are written testimonials of the value of a service. They can be solicited as part of a formal feedback process, or unsolicited when users simply tell you how much they like a service. 
2. User Surveys - these are carefully designed, methodologically sound instruments you issue to users or a sample of users that gather feedback in a way that can be aggregated, compared and reported on.
3. Productivity Studies - a blend of quant and qual, these metrics assess the time-use and user behavior associated with a resource. These can be survey-based or gathered via network, server or key-stroke means.

I have always argued that information professionals should endeavor to gather measurable (and reportable) data in qualitative analyses. While feedback such as verbatims and anecdotes can be powerful ammunition in support of a purchase, these on their own are insufficient. The reality is that most senior managers respond best to numeric insight.

So how do you assign numeric values to qualitative data? What you're looking for, beyond verbatims, is a way to systematically measure business impact. A user verbatim may read: "this source is tremendously helpful. It has saved me countless hours and has helped us land three deals we wouldn't otherwise have gotten".

How do you record something like this in a way that can be compared, aggregated, analyzed and visualized?

You need to conduct a user survey. This doesn't have to be hard - you're not trying to recreate a Gallup poll or comScore panel! Here's what I recommend.

As part of your renewal process, survey a sample of end users about the product's value. Your survey questions should be laser-focused on outcomes. Don't worry about usage or process stats - you're collecting that already! Some example survey questions:

1. On a scale of 1 to 5, with 5 being the highest, how important would you rate Horizon Research to your job? (Scale)
2. On a scale of 1 to 5, with 5 being the highest, how valuable is Horizon Research to the work product you deliver to clients? (Scale)
3. Your subscription to Horizon costs $12,000 per year. At that price, do you think it is worth renewing this service? (Y/N)
4. Have you landed deals or won business because you have access to Horizon Research? (Y/N)
5. On a scale of 1 to 5, with 5 being the highest, how much more productive are you because you have access to Horizon Research (Scale)
6. On a scale of 1 to 5, with 5 being the highest, how satisfied are you with the quality of the data you get from Horizon? (Scale)

Remember, you're focused on two things with these surveys: business impact and measurable data. Scale and Y/N questions like the above measure impact and be easily aggregated and reported on.

There are limitations to surveys like these of course. They are user reported, and people tend to over-report the value of a resource. Best example: Bloomberg Terminals. Ask any owner of a Bloomberg any of the above questions and the answers will be all 5s and Yes, Yes, Yes. You need to cross reference this feedback with quantitative metrics: when you look at the actual usage data, you see a very different story - sporadic logins, limited usage, and content that's cheaply available elsewhere.

You can also verify survey data with other qualitative inputs. Trust, but verify, is the name of the game. A time use productivity study is a great way to do this.

Suppose you're considering whether to renew a service that helps your employees pull and share regulatory filings. The verbatims you've gathered suggest the main value people get from it is time savings. Upon further investigation, you learn that with the service, users are spending 10 minutes/day pulling filings, and 5 minutes/day sharing the filings through the product. A similar, free service on the web has them spending 20 minutes retrieving and 10 minutes sharing filings.

The mean hourly pay for the user base is $60. There are 20 users.

Under the free service, users are spending $20 in time retrieving info and $10 sharing it. That's $400/day in retrieval and $200/day in sharing. $600 a day on this one workflow for your team!

With the paid service, the time spent and thus the cost, is half that: your team is spending $200/day on retrieval and $100 on sharing, or $300/day on this workflow.

Added up over a 200 day work year, your team spends $120,000 in time for this workflow with the free configuration, but only $60,000 in time annually with the paid service.  This is valuable data to complement your verbatims and survey data.

Qualitative inputs are critical inputs for your ROI analyses. Where quantitative usage data measures processes, qualitative inputs like verbatims, surveys, and time use studies are used to show outcomes and business impact.

- Kevan Huston