UAT vs Usability Testing: Can UAT Really Replace Usability?

User acceptance testing (UAT) – also known as beta testing – is strictly a technical step and does not replace the breadth and depth of knowledge acquired through user experience research and usability testing.

There is a distinct difference between testing for bugs – UAT, or beta testing – and measuring how well users are able to interact with a design, or usability testing. While it is important to capture how well the site technically functions in beta, it is also imperative to determine whether the product truly meets the needs and expectations of normal people and improves the customer experience.  Picture yourself preparing for a performance of the Superbowl half-time show.  Think of UAT as an “inspection” of the lights and stage and UX testing as a “dress rehearsal.”  You need both if you want things to go smoothly.  If only UAT testing were performed, our dear entertainer can have the perfect acoustics and lighting but be thoroughly confused as to where to stand on stage and how to accommodate her fly dance moves in her specially-made outfit.

In this clip from the HBO show Silicon Valley, the software engineers contact their other programmer friends to test a newly developed product. The idea is to get feedback from fellow engineers who will try to break the beta version of the software and weed out any functionality issues. The product receives glowing remarks.

When the product goes to a focus group, however, it is torn apart because it is not user friendly. The user interface is impossible to navigate (the users report that they were “totally freaked out” by the UI), and the developers hadn’t picked up on this because they were too close to the code and technical side of development to think about user flows and the customer experience. As the character Monica says, “You are trying to sell the product to regular people, but you never put it in the hands of regular people.”

Take this as a cautionary tale: UAT does not replace usability testing – an important part of UX best practices. However, both processes do have their place and time in digital product development.

Let’s look more closely at how they compare to each other. If we were performing a UAT on a purchase funnel, we would want to ensure the SSO is  functioning properly, that the product is being added to the cart, that security measures have been set up for payment and that product access is working bug-free without much regard to the target audience, like the programmer friends in the Silicon Valley example. Examples of questions to ask the tester: Can you click this button? Can you hover over this drop down?

However, usability testing would focus on the navigation, information architecture, content, and interaction all from the perspective of the end user’s intended performance to ensure user friendly interactions. Examples of questions to ask:  Where are you in this process? What do you expect to happen next?

Which comes first? Ideally, usability testing is an activity that happens throughout the project lifecycle. However, as the project gets closer to the beta testing stage, it would be wise to complete UAT and fix any issues before running the next usability test. This would reduce the distractions – such as bugs and errors – in the site when it is sent to the participants so they can focus on usability testing tasks.