Sputnik 11

Leisure Suit Larry was a somewhat racy graphic adventure released in 1987. I never played it—I wasn’t even aware of its existence until years later. Recently I learned that it was a huge hit, not only because of its risqué content, but because it was actually a good game. Its author attributes this to the fact that it was the first game the publisher ever ran through user testing.

Most app developers I know don’t do user testing, either. And then they wonder why they have no customers.

Dumb Excuses for Not Testing

Did this man take my laptop?

As I said in my last post, my laptop was stolen while I was performing coffee shop user testing for a client. I didn’t actually say define ‘user testing’, so here we go: you ask people to try your app, then you see if they can figure it out, and if it solves their problem.

Here are the excuses I get for not doing it:

What could I possibly learn?

I get this all the time. Some entrepreneurs are so sure of their Grand Vision for the Future that they don’t believe they can learn anything from users.

That doesn’t sound very scientific.

It isn’t. Qualitative research identifies problems; quantitative research measures them.

What do you have against data? Let’s do a survey.

You’ll get plenty of data when you launch, and it’ll be way more relevant than survey data. But you will learn different things from user testing.

I don’t want to delay the launch to do user testing.

So do it after launch. Unless you’re planning to do a big marketing splash to coincide with go-live—and you probably shouldn’t—this is fine.

I am afraid that people will hate it.

OK, nobody says this, but it’s what they’re thinking.

You’ve convinced me. Let’s do this the right way.

Nope: let’s half-ass it instead. Performing user testing at a coffee shop implies that you’re not that particular about who is participating, and that your main focus is on ensuring that it actually gets done and doesn’t become this bizarro never-ending project of its own. (If you work for a giant corporation, then go ahead and do this stuff. For you, eking out small improvements can yield big dividends, and the cost of a mistake is huge. For entrepreneurs, speed is paramount.)

Eight User Testing Tips

I’ve had my picture taken over a dozen times while I’m testing. A college professor uses an image of me in his class. Tourists laugh and say, “Only in San Francisco!” Yelp’s offices overlook the café where I set up shop, and they sent a member of their product team down to investigate what was going on. (Yes, I made her take the test.) So I’ve done enough user testing that I feel comfortable sharing some advice:

The sign I taped to my chair

1. Write a script. If you try to wing it, you’re wasting everyone’s time. Test it with two people outside of your organization.
2. If what users type into the device—not just their comments—is important, then instrument the app so you can collect their data without looking over their shoulders.
3. Find a local coffee shop with decent foot traffic. Family-run ones are best: you want one where the person behind the counter can make the decision. (If you go to Starbucks, they’ll give you a little card with the number for their Intergalactic HQ.) Once you find a place, go there every time. Sometimes prospective testers don’t have time the first time they see you, but they’ll look for you next visit.
4. 

   I’ve tested with people who were on meth. If they can understand your app, you know it’s easy to use.

   Offer money. $10 for 10 minutes is far more compelling than $20 for 20, so break your testing into small chunks. Lots of people will do it just for the money, and that’s OK.
5. Tell them that you don’t care if they like the app or not. I like to say that I’m testing it for a friend.
6. Ten people is enough. I know this sounds ridiculously small, but by the time you talk to the last person, you should be hearing the same complaints over and over. That means it’s time to stop testing and go fix things.
7. If it’s below 65 F, it is too cold for San Franciscans to be outside.
8. Nobody wants to go first: people would walk up to me, get nervous, and back away. I am always very thankful for the people who volunteer. By the time they finish, someone else is inevitably waiting.

Done Is Better Than Perfect

I’m a big believer in strategy-focused product development. Part of that entails being laser-focused on who your customers are (and are not). So how can I recommend something as sloppy as coffee shop user testing, where you’re profoundly unconcerned with the demographic of your testers?

Because the alternative for entrepreneurs is usually not doing it at all. Someday you’ll be big enough to worry about getting users in the right demo, but that time is not before you launch.