Pitfalls of User-Testing: Learn Them Here

As someone who always dreaded "market research" and data, I have to say that over the years I've become much more appreciative of marketing metrics.  Most importantly, measuring consumer behavior, online and offline, has radically improved my marketing results.  Enter the concept of user testing.  In today's guest post Jason Thai explains user testing and the more common pitfalls associated with it.

User-Testing Overview

Anytime a web-based business releases a new feature, several issues pop up, keeping product managers tossing and turning at night:

·         1. Are there any glaring bugs with the new feature?

·         2. How will customers view the new feature? 

·         3. Did we design the feature in the most customer-friendly way?

·         4. Does the new feature impact other unidentified areas in the site?

·         5. And most importantly, how will the feature impact overall conversion?

If only there was a way to test your products before launch…  Luckily, several companies have built fantastic user-testing platforms, providing companies with a pool of unbiased respondents to provide feedback as if they were real customers.  A lot of the sites offer great features such as video walk-throughs with step-by-step user interaction.  Another great aspect of user testing is that there are few constraints on what you can test.  A few of the tests we’ve run include:

·         Testing preferences for  before and after web page designs

·         Testing the ease of our order flow process

·         Testing the usability of new customer features

Also, user-testing is relatively inexpensive, making it a no-brainer for identifying pitfalls prior to product launch.  

What Happens When User-Testing Results Go Against the Grain?

However, are the results of user-testing always gospel?   We, at FoxTranslate, a certified translation service ran into that exact issue.  A few months ago, we redesigned our homepage.  We felt that the redesign more effectively conveyed our key service features and more professionally segmented the content.

Current Homepage	Redesigned Homepage

To validate our intuition, we ran user-testing to compare the two sites, expecting our redesigned homepage to be the clear champion.  Unfortunately, the results of user-testing suggested otherwise.

User-Testing Results

If you needed a document translated, which page would you more likely use?	Common Themes for Picking the Current Page
Here is the split between the "preferred" current page and the "preferred" redesign page.	“The current homepage feels more credible and has a more professional look”           “The current homepage feels more secure”           “The current homepage is much easier to navigate and gather information”           “The current homepage has a much better layout”

The results showed that users clearly preferred the current homepage, meaning it might be time to head back to the drawing board.  

Call it stubbornness, but we proceeded to test the redesign on our actual customers using Google website optimizer.  Google website optimizer tested the effectiveness of our new homepage by sending half our customers to the old homepage and half to the new homepage.  From there, website optimizer calculated conversion rate (# of transaction made / site visitors) and revenue generated, allowing us to gauge which page lead to more transactions.

Results of Current vs. Redesigned Homepage Test

Redesigned Homepage Conversion Rate and Transaction Size Improvement

Over the course of a month, our new homepage generated 17% more revenue vs. the current homepage on similar customer visits.   6% of the increase was due to more converting visitors and 11% of the increase was due to people spending more per transactions.

Our Learnings

So, why didn’t the actual results align with the results from user testing?  Sometimes, it’s hard to replicate your customers – Regardless of the results; we are still advocates of user-testing.  However, sometimes, it’s hard to find everyday people to replicate your customers.  We’re not selling laundry detergent, so generic user panels can only tell us so much because they don’t exactly know what your customers care about.

About the Author: Jason Thai is a marketing manager for FoxTranslate, specializing in document translation of business, legal, immigration and academic documents in over 30 different languages.

You have read the best review article categorized by Jason Thai / user testing and the title Pitfalls of User-Testing: Learn Them Here. You can bookmark or spread this post by using this URL http://howto-howto8.blogspot.com/2011/12/pitfalls-of-user-testing-learn-them-here.html. Thank You!