Usability testing is an enlightening technique for designing products people are likely to buy and enjoy. Yet this method has one great hidden trap that is hard to find.

How irrational is rationality

As human beings, we tend to think that the way our mind works is driven by rationality.. If it was not for cogito ergo sum, how would the Western Culture look like, you might say. Yet owing to the research by Daniel Kahneman and Amos Tversky, since late 70’s there are proofs that this conviction is far from accuracy.

They found out that as human beings we are prone to so-called heuristics and biases.

Heuristic and biases are simple, yet efficient rules that we are provided with by evolutionary processes.

They influence the way people make make decisions, solve tasks and create judgement.

Why heuristics matter in the way we conduct UX research

Although this theory seems quite distant from everyday working reality of UX people, it casts a light on why the conclusions we make during user experience research often do not improve the performance of product. It might sound controversial to say, but it does actually happen.

Not every usability testing session, user recording analysis and conversion funnel overview leads to conclusions that actually make the product perform better.

It happens not because these UX research methods are faulty. It does not happen as well because you cannot follow rules and do a usability study. It is because of the tendency of our mind for inner sabotage that is called fundamental attribution error.

Why are we skipping important information and do not notice that?

The heuristic called fundamental attribution error is the tendency of the mind to asses that human actions are driven by internal motivations and characteristics and underestimate the influence of external factors on their behaviour. Because of this tendency

we believe that from pure observation we can drive complete conclusions on the motivations behind user behaviour.

This idea can easily lead to misconceptions about the motivations users have, because we are unable to understand what are the random external factors that interrupt or modify their behaviour.

Fundamental attribution error can be harmful for such areas of user experience research as:

1. Analysis of conversion funner from Google Analytics
2. Analysis of recordings from user sessions (via Hotjar)
3. Unmoderated usability testing
4. And even moderated usability testing if not performed with enough care

For example, the website we analyse has relatively shallow visit depth. From that we can drive a conclusion that users are not engaged. Yet, it does not necessarily has to be the one and only right conclusion. Do you consider such external factors as irritating content of remarketing campaign and banners scattered all over the page? Or the fact that the main banner lead to external URL? Or just the fact there is no need for people to explore the site deeply, because it is a culinary blog and what they need is a recipe for particular dish such as duck with apples, and not “other recipes that contain apples?”.

The external factors are there, but there are ways to get to know them and do not drive false conclusions from UX research

How can we draw better conclusions?

The scientific research is done in scientific and unified environment on a defined group of people, so it can be repeated to clarify whether the results are right or wrong. Hovewer, user experience research in everyday work is often more fluid and has a different goal than pure scientific research. It is not to find a truth about human behaviour in general, but to optimize a certain behaviour.

Users never experience digital products in laboratory environment.

There are always tones of external factor that can influence their experience: time, place and social situations that accompany the usage of digital product and their own beliefs and values.

It’s important not to undervalue the element of serendipity and random that appears in the non-static digital products (such as an e-commerce or newspaper website).

We cannot probably win over the tendency in human mind to search for inner causes of behaviours, but what can help not to draw wrong conclusions is getting to know more about the context of usage that can hypothetically influence the experience.

The only way to do it is to talk do people — ask about their opinions and habits (where and how they use a product, what other things are they doing at the same time). Identify their opinions using in-depth interviews.

And if the funds are limited — go guerilla.