welterweightUX

Half inspired by the $5 Guerrilla User Test concept, half by the need to quickly and cheaply get user feedback to quell internal debate, @yellowledbedder and I hit up AJ Bombers in downtown Milwaukee last Wednesday afternoon. We were armed only with a pair of laptops and a question: Which approach is more effective, user?

A bit of background
We’ve been working on developing a new section for our consumer-facing websites. The section has refinement tools that allows users to get to more qualified, personalized search results.

In the consumer world, refinements traditionally cascade down the left side of the page. Take, for example, how CarMax does it:

But during our research, we found that some of the biggies, like Zappos, are starting to place these tools across the top of the page, above the products, like this:

The evils of untested innovation
Innovation is good. Don’t get me wrong. Our UX team, some of our designers, and most of the business teams personally preferred the new-fangled horizontal approach.

But UXers, being persnickety as we are, said ardently that our preferences couldn’t make the call. Users had to decide. So we pitched the idea for a guerrilla study and got energetic support from the organization to go ahead.

Here’s what we did
We came up with the idea of testing this way on Friday afternoon, for testing the following Wednesday.

These were our steps, from concept to execution.

Friday:

1. Got approval to buy participants lunch (up to $15) for 15 minutes of their time.
2. Contacted @ajbombers (a Milwaukee restaurant well-connected in the social media realm) to see if we could take up table space for a few hours.
3. Sent out a tweet that included my email address for participants to contact me for more info.
4. Had AJ Bombers retweet my tweet.

Within minutes, I had several responses. I contacted the respondents immediately to let them know I’d be following up Monday with details.

When I got into work Monday, there were several more emails waiting! The rest of the day went something like this:

1. Contacted respondents to let them know available times, asking them to identify the 3 times that worked best with their schedule.
2. Began slotting participants into available times as they responded.
3. Followed up by confirming times and giving details about where we’d be and generally what they could expect.

Testing day, Wednesday, went like this:

1. Tied up all loose recruiting ends, like getting zero hour substitutions for people who could no longer attend. Social media made this super easy.
2. Got to the restaurant when they opened, set up, ordered a soda, and waited for our first participant.
3. Tested 10 participants from between 11:20am and 3pm-ish.

The result
What happened was not at all what we expected. Preference for horizontal vs vertical was split precisely 50/50. There was no trend based on age, gender, profession, etc. It was a pure matter of preference.

Now, this may sound inconclusive. But really, it’s anything but.

We can now confidently tell our clients that they can use the placement that best suits their business needs. Both approaches were understandable, noticeable, and usable.

Oh, and the total cost of the test (aka, the restaurant tab, including tip) was under $200.

How often can you walk away with such inexpensive surety? Not very.

I quickly plugged our findings into an executive summary, which we shared with the larger team Friday morning. They were pleased as punch with the results.

Gorilla vs guerrilla vs go-for-reala testing
My partner-in-UX-crime, Mike, recently wrote a blog post outlining the difference between gorilla testing (old, classic, lab testing) and guerrilla testing (agile, informal, out “in the field”). It’s worth a read.

I agree that different realms of technology require different methodologies. But if you’re looking to get quick feedback on a consumer-facing website or app, meeting users out in the real world— which is rife with verisimilitudinous distraction—is bar-none the best thing you can do. Plus it’s cheap. You get the answers you need, now. No expensive recruiting, no huge pay-outs, no sterile lab settings. Just down and dirty results.

Really, I see this as more than just guerrilla testing. I see it as go-for-reala user testing. Because it’s a real deal (also, I just really like puns). Yo? Ahem…

Shout outs
Thanks to AJ Bombers for being amazingly accommodating! And thanks to our participants for being great guinea pigs and proving the effectiveness of the go-fo-reala approach to user testing.