As promised last week, we’re embarking on some private-label usability tests of blogging. We’re curious as to how some of the various interfaces related to blogs actually stack up when subjected to detailed qualitative reviews. As a result, we’ve been engaged in our typical pre-test preparation: generating screeners, debating moderator’s guides and testing scenarios, considering equipment set up and - most recently - recruiting.

As with many fundamentally interesting or vaguely anthropological exercises, usability testing has its more seemingly mundane aspects. These can range from procuring and setting up testing technology to finding the people to actually test. Mundane as they may appear, however, the test wouldn’t happen without them. Further, if a poor job is done on this level, the test might well be flawed or hard to back up.

Which brings me to recruiting, a peculiarly nuanced but sometimes exhausting activity. Typically, for clients, we enhance our efforts with banks of professional recruiters accessed through the lab facilities where we perform the tests. We create elaborate screening scripts and monitor the candidate responses as they come in - in order to make certain that the outsourcing is producing the right kind of candidate. In this case, however, we’re doing it all in-house, which means JF is casting the net.

In addition to the usual “family/friends” channels, we often use Craigslist when we want a classic cross section of Internet users. Although we have a network of enthusiastic people ranging in age from 23 to 40, it doesn’t compare in responsiveness with Craigslist users. This, of course, is both a strength and a weakness of the site - many of these folks are overly self-selected for “computer gigs” as CL terms things like our usability test. But for casting a wide net fast and cheaply, it’s still hard to beat. Within about twelve hours of posting a generic research request, we had several hundred responses.

But the question at hand today is less how to cast the net, than what to do once you have found your recruits. Because from time to time, an odd dynamic arises: people get pushy. Despite the fact they are volunteers, and despite the fact they are receiving an solid incentive fee, sometimes they insist on more. So, here’s a hypothetical situation: you have 15 test slots, during which 15 people will spend an hour working through various tasks online. Only one slot has been filled. But one of your other recruits is insisting on that same spot. He/she is also requesting parking and subway advice, questioning the fee and asking if the session can be shortened. What should you do?

On one hand, the recruit is part of a relatively small pool of people if he/she has passed through the screener. (In our actual case, only about 20% of those called qualify for the test we wish to conduct.) Therefore, you need him/her - because finding someone else will mean making, on average, some multiple of calls.

But on the other, most other recruits are happy to operate within the carefully-calibrated times and confines of the test. And pre-test behavior is often an indicator for in-test dynamics. Further, it can be an indicator of the kind of person who routinely participates in such research and who may even fib to get past the screener. Not always, of course, but enough to be a bit of a red flag. So, given our hypothetical situation, we’d recommend the following:

1. Review the answers to the pushy recruit’s screener for any additional red flags/qualifications.

4. Consider the overall size of the recruiting pool - have you exhausted all your leads?

If there are some additional red flags - hesitation over some answers, frequency of behavior that’s within tolerance range but maybe different from the ideal candidate - the yield is 20% or higher, no one else is having similar scheduling issues and you have some leads left, decline the candidate. Experience says it’s better to find an accommodating person than struggle through with someone who can’t or won’t give the test his/her full attention.

JF

This entry was posted on Wednesday, June 22nd, 2005 at 2:32 pm and is filed under Usability Testing, Diary of a Business Blog, Usability Research.