What’s the real difference? Face-to-face versus Remote user testing

Have you ever wondered what the real difference is between remote unmoderated user testing (like usertesting.com and whatusersdo) and face-to-face user testing? Which method should you use, when and what for? There is a huge difference in these two techniques and the end results so let’s take a look at them both in greater depth.

Face-to-face

Interviews conducted face-to-face with target users, often in a lab setting or in-context, for example in the person’s home to make them feel more at ease and uncover more realistic feedback and issues. The interviews are a two-way conversation guided by a trained researcher.

The positives

• Can be conducted at any stage of the design process, even before you have any UI in place. Gathering very early feedback will save time and cost later!
• Conducted by a researcher as a two way conversation, giving the opportunity to question answers given and behaviour observed
• Very rich data. The opportunity to question users enables you to discover deep insights and more potential barriers to conversion
• Greater team collaboration. When observing lab research, the team are together in an accompanying observation room. This increases design collaboration and buy-in for changes
• See the user and their reactions. Face-to-face interviews use technology that can record the user as well as their interactions, so you can see their facial expressions and real time responses. This can make the videos much more powerful – it’s harder to ignore the feedback if you can see a real person and you know they’re just sat in the next room!
• Discover deeper problems and opportunities. The researcher can spend a portion of the interview understanding behaviour and underlying psychological motivations, these can often open up greater opportunities than just examining the UI in isolation
• If a user stops thinking out loud (this is quite common), the researcher can prompt them or follow up with another question
• Gain environmental, contextual and personality insights (if the research is conducted in context)

When should I use this method?

• You want to understand current user behaviour, needs, wants and concerns
• You want feedback on early concept ideas, which you should pursue and why
• You want to test sketches or lo-fidelity mockups
• Throughout a project when you need to more deeply understand user behaviour and interaction
• When you need to get stakeholder buy-in. They’re much more highly influenced when they’re sat in a room together, observing real people struggle to use their UI
• If you want to benefit from the expertise of a UX researcher to run the session and to help recommend changes that will successfully engage users
• When you need to establish rapport with users to encourage trust to discuss things they might not disclose if just speaking into a computer
• If you want more authentic results
• If the tasks and feedback will take longer than 15 minutes

The drawbacks

• Your research is only as good as the person conducting it. Face-to-face relies heavily on the research capabilities of the researcher. If someone with little experience runs research, there is a greater risk of them asking leading questions, not establishing rapport and missing key identifiers for insights to further pursue. They are also more likely to express interviewer bias – where they impose their beliefs and expectations subconsciously on the user and therefore influence their responses. It takes years of practice to conduct great user interviews but great interviews give better output
• If your testing requires a large number of participants, face-to-face testing will quickly become very expensive. It works best for small scale, qualitative research (5-15 people)
• It generally takes longer than remote testing

Remote unmoderated testing

Tasks and questions are planned in advance so that the user can complete them in a small amount of time (usually 15 minutes) in their own time at home. This is a one way method (no researcher involved) with users required to think aloud as they complete tasks.

The Positives

• Cheap
• Quick
• User is in their environment on their device (note: face-to-face can also be performed in context)
• Cheaper for collecting large samples
• No researcher needed (note: this could also be seen as a negative)

When should I use this method?

• If you need very quick results
• Your budget is too small for face-to-face
• If you just need to test a small aspect of your UI, something that can be explored in 15 minutes
• You need a larger number of responses to backup qualitative face-to-face research with greater significance
• You aren’t researching difficult, complex flows or sensitive issues
• You’re main focus is on identifying easy to find usability issues and not deeper problems, insights or psychological barriers to conversion
• When your UI is higher fidelity

The drawbacks

• One way method, no active questioning. If the user misinterprets the question or gets stuck on something they can’t recover from, no one can put them back on track and help them. You also have to try and pre-empt the problem they might encounter in order to ask the right questions in the first place, this allows no chance to pursue interesting things they may do or say.
• Limited to around 15 minutes. This limits the amount of feedback you can gather and the depth of questioning. It can feel like the user’s just got themselves settled into the research and they’re giving some great feedback, then before you know it, the session has ended.
• Doesn’t record the user so you won’t get to see their facial reactions or body language
• Higher risk of expert and money-motivated users. Some of the users may participate in many tests therefore becoming experts, others may be highly motivated by the payment and they’ll fly through the tasks without getting into the correct mindset.
• Users focussed on usability. As people sign up through the testing website, people are sometimes aware that the focus is on usability and user experience and you’ll hear them mention ‘usability’ and other terms a real consumer just wouldn’t pick up on.
• Thinking out loud can be quite a difficult skill to maintain. Once their cognitive resources are concentrated on the task in hand it’s easy to forget to think aloud. In face-to-face the interviewer can prompt them, but in remote, the user is on their own.

Feel the fear and do it anyway!

Understandably it feels very comfortable doing remote testing. You are more in charge, you can control the tasks and questions and even the end videos – if one user is particularly scathing there may be the temptation to not show that video to the team.

However, if it’s the only method you use you will always be limiting yourself to the top level issues you can find based on the tasks and questions you define (do you see how you’re already biasing your own results with your own expectations?). There’s also the limitation of the 15 minute sessions, as well as your own ability to analyse this sessions appropriately and suggest recommendations that will give the most effective results.

If you’ve never tried face-to-face research we’d strongly advise you give it a go. Yes some of it might be uncomfortable to hear but mostly, it will be an eye opening and inspiring experience! You’ll feel more connected to your customers, you’ll feel more connected as team, and you’ll go away with tonnes of new ideas to make your user experience and conversion much better!

Need help?

If you are in any doubt as to which method is right for you, or if you would like to find out more about how face-to-face could help you, feel free to get in touch and one of our experts will happily help you.