Building a user testing program from scratch: how to

Step 1: Convince People.

I have found that it can be difficult to convince folks to start user testing at big companies. Here are some tips and tricks to convincing people that you need to do user testing:

Step 2: Determine your Audience.

Once you have interest or agreement to do user testing, you will need to figure out who will be testing your site with you.

You'll need at least 5 people per each segment (Source), and if you testing on multiple devices you'll need 5 people from each segment per device.

Step 3: Recruitment.

We recruited existing customers, since most of our revenue came from returning customers. We did this by emailing users that we had in our email list (other ways to recruit users and the pros and cons of each method here).

If you work in a large organization, sometimes getting work done from different teams takes longer than what you would expect. Make sure you know how long each step in the process will take. Because getting an email out was the longest step in the process, I kicked off the email portion of the project first. You will have to look at your own company to figure out the timing of the steps. Things you will need to determine before you write the email:

Step 3.5: Write a screener

You will need to write and build a screener survey: this should be the landing page for your email. Make sure you remember to ask folks when they are available and their preferred method of contact. Make a nice thank-you page with when they can expect to be contacted.

Step 4: Build your tasks and questions.

This should be fairly simple: identify the major tasks on your site and write them down. For example, an eCommerce site task could be something like: "Find a product you are interested in." Here's the Nielsen Norman Group's post on how to write tasks if you are unsure.

For each task make sure you have a method to evaluate the success of the task or not. This can be pretty tricky. If your office thinks in numbers and analytics, you will want to measure the outcomes of your test quantatively. Plan how you are going to do that at this step. Here's a book on the subject.

Step 3: Acquire Materials.

Purchase or make use of what is in your office. When I was doing this, I was lucky enough to find all the materials in house. I asked our video team to lend me a mini-tripod for our QA team's iPhone (which I used to film the sessions). Mini tripods are available at Best Buy (or just do a google search).

At this point you will need to think about how you will record the user reactions and what is on the screen. We set up our user testing so that users faced the interviewer and the camera. The device they were using was plugged into our projection system. The projection was behind them. That way the camera captured the screen and their reactions to the device. I don't have any advice on screen recording software - so if you do - message me and I'll update the article.

Step 4: Book Interviews.

I was lucky enough that the admin assistant for my department did this for me. Make sure you build time into your schedule to do this work (it takes awhile).

Step 5: Prep for the Testing.

Don't forget about ports, questions, release for video if you filming, run a practice session, have someone taking notes / keeping track, decide if you want to broadcast the interviews live, and don't wear a low-cut shirt because sometimes you might have to bend down in-front of the camera (true story).

Thanks for reading, and write to me if I missed anything. Back to top