App development is a complex process. As designers move through numerous iterations, there’s a fair amount of testing that has to be done before an app is ready for release. After system tests to check that the app works pretty much as expected, the last stage of testing is user acceptance testing (UAT). UAT is one of the final opportunities for developers and the QA team to identify issues and make significant changes to the app before its release.
Defining User Acceptance Testing
User acceptance testing (UAT), sometimes known as alpha/beta testing or end-user evaluation, is the last phase of testing that an app goes through before it is released. Developers also use UAT after an app is released to test new features or design changes.
Although UAT is occasionally referred to as alpha or beta testing, this doesn’t tell the full story. Alpha and beta testing are just part of the full gamut of tests that make up user acceptance testing. These tests aim to establish whether an app is fit for release. Although they will likely catch a few bugs, it is assumed that by the time an app reaches this stage of development, it should be functioning correctly.
The Purpose Of User Acceptance Testing In Development
UAT is used to check how a product functions in real-world situations. It's invaluable in app development as it helps development teams to understand how users may technically respond to the final product and its features, catch any last-minute bugs, and test new features. As the name suggests, the basic goal is to establish whether a theoretical user would find the app or feature at hand acceptable. Within that goal, the various test types each have their own purpose.
At the end of the UAT process, once any recommended changes have been made, the QA and development teams and the client should be sure that the app is fully functional and is ready to be released.
In short, the objectives of UAT are to confirm that the app:
- Meets client requirements
- Meets users' technical requirements
- Complies with relevant regulations
- Is ready for release
Why Do You Need User Acceptance Testing?
There are a lot of apps out there. Users often have a huge variety of options for any given task, so for your app to stand out and retain users, it’s not enough for the app to be functional; you need to make it as user-friendly and pleasant to use as you can.
UAT helps to ensure that your app is not only functional, but also user-friendly. Yes, acceptance might sound like a low bar, but, if done correctly, UAT will enable you to improve the user experience to the point where users not only accept the app, but embrace it. It's a critical milestone for good UX.
Catch last-minute bugs
Often, when we design and build products, we are so close to the product that we struggle to see it clearly. Having QA professionals and developers test it out and provide their feedback allows us to view the product with fresh eyes and often surfaces issues that we never would have spotted or thought of.
Sending apps out into the world without performing UAT can be enormously costly as there are always problems that went previously unnoticed. Fixing these after release is much more expensive than tackling them beforehand. Then there is the added problem of users switching off – if they have a bad experience once, they are unlikely to give the app a second chance.
Types Of User Acceptance Testing
There are several types of tests that make up UAT, performed at different times during the development process:
Alpha testing is carried out by internal staff, often members of the client organization, when the app is approaching completion. The idea is that alpha testers experience the app as an end-user would and give feedback to allow developers to identify bugs and make changes.
Alpha testing is usually conducted in a development environment and is the first chance to spot problems outside of the testing environment. The development team does not take part in the alpha test, but observes the testers and records the results, before making necessary changes.
Beta testing, or field testing, comes after alpha testing, but before the app is released. Testing takes place in a real operating environment, by specially selected users who are not part of the client organization.
Beta testing allows developers and the QA team to discover any remaining bugs, understand how users interact with the app, and collect feedback on the user experience. Beta testing is also used to test new features or significant changes to already-released apps.
Black Box Testing
Black box testing, also known as behavioral testing, is carried out by a tester familiar with the functional specifications of the app, but unfamiliar with the code. They test that the app works as desired, and, typically, many iterations are carried out throughout the development process, at every stage in the testing cycle.
Black box testing helps to identify basic problems with the functioning of the app, without the tester having access to the code. It is sometimes classed as a type of functional testing.
Contract Acceptance Testing
At the beginning of the development process, a contract is drawn up between the client and the developer that lays out the expectations and requirements for the finished product. Contract acceptance testing determines whether the terms of the contract have been met.
Operational Acceptance Testing
Operational acceptance testing, or production acceptance testing, checks whether the app works smoothly and reliably with all parts of its operating environment. This includes maintenance, security and update systems, user training and workflows, and fallback plans in case of malfunction.
Regulation Acceptance Testing
Regulation acceptance testing, or compliance acceptance testing, is concerned with whether the app conforms to relevant industry, governmental and legal regulations and laws.
Who Is Involved And Participates In User Acceptance Testing
User Acceptance tests are typically designed and conducted by the development and QA teams, with input from the client.
The roles in the UAT process can vary, depending on the factors like the types of tests being performed and the size of the operation, but the roles and responsibilities generally look something like this:
- Client: Liaises with the development and QA teams to establish requirements and expectations. May allocate employees to be part of alpha testing. Signs off on the work when requirements are fulfilled.
- Development team: Designs and implements tests, may observe testers and review data. Makes required changes to code.
- QA team: Trains and observes testers, collates and analyses data, passes on results and further requirements to the development team. Presents results to the client.
- End-users: Selected users perform beta testing and report their results to development team.
When User Acceptance Testing Is Performed
UAT comes towards the end of the development process when the app is nearly ready to be released. Several other types of testing come before UAT in the app development life cycle, including system testing and usability testing. UAT is also performed after the app is released, as it develops with new or altered features.
The different types of UAT are carried out at different stages of development and the app life cycle:
- Before the initial release: Black box testing, alpha testing, operational, regulation, and contract acceptance testing
- Patch or update tests before release: Beta testing, black box testing
- After release: Beta testing of new features, black box testing
9 Essential Principles When Conducting User Acceptance Testing
To conduct UAT successfully, you can break the process down into nine steps. Let’s look at them in detail:
1. Business Requirements Analysis
First, everyone involved needs to understand what is required of this round of testing. A meeting between relevant parties (lead developers, business analysts, QA engineers, project manager, client, etc.) with all requirements, objectives, and expectations properly recorded, is essential. The more detailed and explicit, the better. During testing, uncertainty is the enemy – questions and problems need to be clearly defined.
Before moving on to the next step, everyone should be clear on:
Scope: What the tests will measure
Objectives: Why the tests are being performed
Requirements: The functional and business requirements of the end product
Responsibilities: Who will take charge of each part of the process
2. Creating The Test Plan
Next, it's time to create a detailed plan. Using the requirements gathered in the first step, lay out the steps to be taken, the how, and the why. The plan needs to cover the scope, timeline, and objectives of the testing, as well as roles and responsibilities for each part of the process.
A few of the things that a good plan should cover include:
- Timescale: When the tests will take place and how long they will take.
- Environment: Where the tests will take place.
- Project methodology: What form the project will take – i.e, waterfall or agile.
- Roles, responsibilities, and hierarchy: Who does what and who reports to whom.
- Communication protocols: Who testers should report to and when, how information will be shared among teams.
- Results recording and analysis: How results will be gathered and analyzed, including tools to be used.
- Entry and exit criteria: The conditions to be met before testing begins and ends.
3. Identifying Test Scenarios
Identify as many hypothetical functional issues that users could face while using the app and use them to create simple test scenarios. It’s vital to stay within scope here, or the number of scenarios could quickly become unworkable. To make the test scenarios easy for non-technical testers to follow, use plain language to describe them rather than overly technical terms.
A scenario might be something like:
- Check subscribe button works as intended
- Check in-app camera functionality when using dark mode
- Check timer start and stop function
4. Create UAT Test Cases
Once you have your test scenarios, it’s time to create test cases. Test cases correspond with the scenarios, breaking down the details in clear, simple steps. Be specific – create a short user guide for each test case that tells the tester exactly what they need to do and ideally, what to expect when they perform each step.
A test case might look like this:
1. Switch app to dark mode
2. Use camera button to take photo
3. Check photo
The user guide for this test case might look like this:
1. Open the app.
2. Navigate to settings using the hamburger button in top right corner.
3. Switch app to dark mode.
4. Use camera button at the bottom of the main screen to take photo. A popup will appear with a thumbnail of the photo and a ‘Photo Saved!’ message.
5. Tap thumbnail on popup. Gallery app will open showing saved photo.
5. Preparation Of Test Data
Once the test cases and scenarios are in place, it’s time to make final preparations before beginning testing. Perform a final check of the test components, then the scenarios and relevant data should be uploaded to the testing platform, and management and user access established.
It's a good idea to take testers through some training to make sure that they are familiar with what's expected of them, how they perform the tests, and how they record the results. Let them know who they should contact in case of a problem – likely different people, depending on the issue.
6. Execution Of Test Cases
Now it’s time to begin testing. Depending on the type of tests, environment, time frame, etc., what the testing looks like and how long it takes will vary a lot. The testers should know what they are doing, but make sure that someone is on hand to help them if they need it. Work methodically through the test cases until all have been completed.
7. Recording Test Results
For testing to be useful, the results must be recorded effectively. Testers should record results as they go, with leads addressing any issues immediately. Using a testing platform specifically designed for the purpose will make the process much easier than relying on spreadsheets or note-taking apps. There are plenty of options available, including software like Usersnap, FitNesse, and Bugwolf. Crowdsourcing sites like UserTesting are another option, and many project management tools like Trello also have UAT functionality.
Testers should record in a standardized way, and include information like:
- Name of tester
- Date of test
- Test case name and number
- Test case steps taken
- Test pass/fail
8. Incorporating Test Feedback
Once the test results have been collated and examined, the QA team can pass on the feedback to the development team, who will then make any necessary changes. Another round of testing must then be carried out to ensure that the changes have been implemented successfully.
9. Confirming Business Objective
Once all changes have been made and retested successfully and final checks against the requirements have been carried out, the QA team produces a closure report which is submitted to the client with the finished product.
If the established requirements have been met, the app is a “Go” and is deemed ready for release. If deeper issues have become apparent during testing and the business objectives cannot be met, the app is a “No-go” and needs more work before it is ready for release.
Optimizing The Effectivity Of The User Acceptance Testing Process
Create An Elaborate And Well-Designed Plan
We’ve all heard the phrase, “failing to plan is planning to fail.” This goes for UAT as with other things in life. Without a proper plan, UAT will not be as effective as it should be and could even be a complete waste of time and money. Don’t be tempted to rush or skimp on time here – UAT is one of the most important parts of the development process, and the plan is the most important part of UAT.
Its important that the test plan is in place well before the time comes to begin testing so that there is no chance of rushing any part of the process. If other parts of the development process overrun, time for UAT reduces so the last thing you want is to have to spend time making a plan. It’s also likely that if the UAT test plan is left until the last minute, it will be less than effective. The plan may also need to be tweaked or updated occasionally.
Don’t Over or Under Scope
If you’ve planned properly, you should be clear on the scope of the UAT. It’s important to make the scope clear to everyone involved from the start, and not to try to do too much in a single round of testing. Laying down the scope initially combats the risk of scope creep, where the test is never finished because the scope keeps changing. Equally, the scope has to be wide enough to be useful.
Evaluate and Monitor
There's no point in carrying out testing if you don't pay attention to the results. Throughout the testing process, pay attention to what testers are doing. Make sure that they are recording valid data, and tackle any issues as soon as they arise.
Consider using a designated testing platform to centralize the process. It will be easier for testers to record their results, easier for you to monitor them, and easier to evaluate them at the end of the test.
Document Everything Clearly
That means everything. It might be tempting to think you’ll remember something important later, but it’s easy to forget. Don’t take the risk – document everything. Proper documentation also helps others understand your method and results, and ensures that the test will provide valuable data.
Using a designated testing platform will allow testers to input their results in a standardized way. It will also be easier to collate the data at the end of the round and access and publish the results.
Other Tests Involved In The Development Process
UAT is just one type of testing involved in the app development process. Before UAT, a whole array of tests will have been carried out, including:
As the name suggests, system testing tests that an entire system is functional and meets requirements. There are many kinds of system tests, each concerned with a different facet of operation.
A type of system testing, functionality testing checks that the app meets its specified functional requirements.
Another type of system testing, usability testing comes towards the end of the development process. The aim is to establish whether users can use the app to perform activities and identify any problems with usability.
Regression testing is performed often, after any changes to the code are made. The aim is to check that the app still functions as intended. If it doesn’t, the app has “regressed.”
Successful UAT Helps You Attain Your Business Objectives
User acceptance testing is the final stage in the lengthy app development cycle. Although it comes right at the end of the process, UAT is essential and its importance should not be underestimated. It's the final opportunity to make sure that an app is as good as it can be before release.
Releasing an app without conducting proper UAT can have disastrous consequences for the organization involved, its finances, and its reputation. Conversely, conducting a well-planned and executed UAT helps an organization improve the user experience significantly, making users more well-disposed towards the app and the organization, and in turn achieving its business objectives.
Learn more about the development process in our blog section.
Abdul has helped over 40 Fortune 500 companies make informed user-centered design decisions through evidence-based user research and UX best practices. As an Adjunct Professor, Abdul has taught in DePaul University’s graduate UX programs and for nine other universities.