Automated testing with Playwright, NUnit and Azure Pipelines
Hello, world! My name is Jelle, and in this blog I am diving into setting up professional, dependable, and automated end-to-end tests for modern applications. Since we are talking about testing, I have divided the blog into three sections: Arrange, Act, and Assert.
Arrange
Firstly, I think it is important to make sure that the quality of my software meets the standard. Therefore, I think testing - especially automated testing – my software is very important. From my experience I also know that automated UI testing frameworks tend to come with certain annoyances and drawbacks that make the entire process of UI testing feel shaky and inconsistent.
When I write a unit test or back-end integration test, and the test fails, I know one of the following is true. Either I made a mistake in my code (I found a bug), or I made an error in my test. Or both.
However, when I write a UI test, and the test fails, I do not experience this same certainty. Sometimes, if I run the test again, it suddenly passes. Sometimes, the error is in the test, but it’s not intuitive why it fails. And other times, I have to add a dreaded Wait-like command to the test to allow the test to pass. When I encounter this, it feels messy and unpredictable. It feels like guesswork.
Side note: this problem is often related to the fact that much CPU is consumed by the running test on one hand, and the (headless) browser used to run the test on the other. This causes the browser to slow down, and the test to fail due to a time out. Browsers tend to be CPU-hungry beasts.
Consequently, the quality of my UI tests feels lacking. I trust my code more than my tests. A deadly sin that provokes either a lot of investment in UI tests to make sure they run as smoothly as my other automated tests; or the abandoning of UI tests altogether. Nevertheless, both are outcomes I would like to prevent.
Enter Playwright.
Playwright is an automation testing framework for end-to-end testing. This means it can also test UIs (comparable to Selenium or Cypress). Its most eye-catching feature is that it was created recently (as of May 2022), meaning it has less… lets call it baggage. It focuses on testing the currently relevant browsers (Chromium, Firefox, and Webkit), application types (like SPA), and features (uploads, downloads, etcetera). It’s all async, which helps a lot with the “random” time-out issues I mentioned above.
With Playwright I can write tests in my beloved .NET, but there is equal support for Java, Node.js and Python. In .NET, the preferred testing framework for Playwright is nUnit. However many developers and testers I know prefer xUnit, which could also work. Playwright is an Open Source, so I expect support for many different testing frameworks which will pop up like mushrooms. However, nUnit is preferred because it is very suited for helping with the division of cores to prevent CPU scarcity and run the Playwright tests in parallel.
Act
Let’s see it in action! As always, when trying out a new package, it is a good idea to start out in a clean project, and start out small. We create a new nUnit project, run the out-of-the-box test as a sanity check, see it pass, and then add the Playwright dependency by adding references to Microsoft.Playwright and Microsoft.Playwright.NUnit. After that, it’s time to turn our basic nUnit test into a Playwright test.
So, what are we looking at here? Well, first of all, we can see that the test class now inherits from the PageTest class. There are other options like ContextTest, BrowserTest and PlaywrightTest, but PageTest is the logical parent class for many of the common UI testing scenario’s. Of course, we have rewritten the test so it’s async, which really should be the standard for any test. I also added a simple line of code: await Page.GotoAsync(“https://playwright.dev”);. The Page object has Locators (for finding UI elements) and Goto and Navigate methods. Basically, we can treat it as the main hub for our UI tests.
This looks clean! With Playwright, we can just assume that when the test runs, the browser will have been spun up, we will have a tab in there and we can navigate using that tab. Very intuitive and easy to learn. Of course, with enough wrapper code, other UI testing frameworks allow for this kind of abstraction, but with Playwright, it’s out of the box!
However, It’s not all sunshine and rainbows. When we run this test, we get an error.
What’s going on here? It’s simple, actually. Playwright does not target our actual browser. It makes use of browsers specifically for running tests, that need to be installed upfront. These browsers can be headless or not, depending on your configuration. Playwright makes the installation of these browsers as easy as possible – it automatically added a (Powershell) script to the build folder for us to run. After running the script, we have another successful test.
The comprehensive documentation will help us to write tests that are cross-browser, dependable, and parallelizable. So, let’s build enough to get some code coverage. Yet, with all those tests, it’s not feasible to run all of them manually each time we make a change. Nonetheless, we do want to run them, again and again, to make sure there is no unintended regression. In other words, we want to run those tests as part of an automated build. So, let’s make the assumption that we’re working with Azure DevOps and are using YAML. The classic pipeline or even Actions in GitHub will work very similarly.
What we’re going to do is run our tests on a Virtual Machine – not an installed agent. There are some advantages to installed agents, but suppose we have the requirements that any new, clean machine can run our tests. When we pull in the latest image of the browser, all tests should succeed. If the latest version of a browser removes a feature, I want to be aware of that as soon as possible. It’s hard to check that with an installed agent. Supposing all that, how would we go about setting this up?
Before we move on to the complicated part, let’s look at the start of the YAML file.
Nothing special here. The trigger for these tests is to run whenever a change to the main branch is made. And these tests will run in .NET 6.0 – as they did locally – so we need to install that on our agent. Let’s skip to the end of the YAML file to see what our goal is.
Again, nothing fancy here. We just want to run those tests that are in our test project. We can put the specific browser to use in the arguments.
Now, for the slightly harder – and less intuitive – part. We’ve skipped 23 lines in the middle. Remember how we had to use a script in our build folder to make sure the browsers are installed? If I run my tests on a brand new machine, I need to run that script during each execution of this YAML file. How do we get that to work? Unfortunately, a little bit of magic is needed.
We want to install Playwright, but we need a manifest, because we need to do a tool install in this brand new environment. So we can set that up.
Next, we can install the .NET tool Playwright within that manifest, because we do not intend to use a global .NET install option here. Why would we, when next time we run the tests, we get a brand new machine anyway?
Then, we build the test project. With the project built, we have a bin directory, and within, we have the script to install the browsers Playwright needs. Of course, this time we happen to be running on Linux, not Windows, so it is not a Powershell script.
Note, that it is imperative that the script is run in a directory which has a test project in it, or in a directory which has a solution with a test project in it. If we run the run playwright install command in another directory, this run command will fail.
If we now run the entire YAML file, we’ll see our tests being run in an automated way. Excellent. We’re just missing that cherry on top. One of the most popular features of any testing framework that can run UI tests: the ability to see images or videos of (failed) tests.
With Playwright, that’s really easy to set up.
using Microsoft.Playwright;
using Microsoft.Playwright.NUnit;
using NUnit.Framework;
using System.Threading.Tasks;
namespace PlaywrightTest1;
public class Tests : PageTest
{
[SetUp]
public void Setup()
{
}
public override BrowserNewContextOptions ContextOptions()
{
var options = base.ContextOptions() ?? new();
options.RecordVideoDir = "videos";
return options;
}
[Test]
public async Task Test1()
{
await Page.GotoAsync("https://podcast.betatalks.nl/");
await Page.Locator(".episode-list--play").First.ClickAsync();
Assert.Pass();
}
[TearDown]
public async Task TearDown()
{
if (Page?.Video != null)
TestContext.AddTestAttachment(await Page.Video.PathAsync());
}
}In any class that inherits from PageTest, we can override a ContextOptions method, which is a hook for all kinds of configuration. And, in the Teardown of the test, we can attach the video file to the test results. All from code!
Assert
We’ve created a nUnit test project to use Playwright so we could write end-to-end tests and UI tests. Then, we ran those tests using YAML in Azure DevOps. So, now we reach the one question that matters. Does Playwright solve all my problems with automated UI testing? Of course, it’s tough to answer such a question; but I’m willing to bet: no. At least not fully. The pipeline needed a bit of magic, and pipelines are notoriously bad wizards. However, there are a few signs that Playwright is going to contribute to the quest for quality. It certainly feels more solid and more reliable than most other similar testing frameworks. I’ve gotten all excited at the prospect of using Playwright in production code, which is always a good sign.
Creating and updating tests should be quick, and a test failure should indicate a need to update (test) code, not a need to rerun the test. I feel that Playwright is a step in the right direction for both of these principles. I do not have to fiddle around with installing drivers during my test, or add weird manual Wait() methods in the middle of my code while I hope the browser finishes… something.
And – like a lot of current .NET code – there’s very little boilerplate code to write. So we can convince others in our team who don’t like writing tests that this time, they should write the tests. It takes very little effort and - if written and maintained well – those tests will save a lot of time and prevent a lot of frustration.
In short, I would advise you to try out Playwright and see if it’s a fit for your testing needs.
Automated testing with Playwright, NUnit and Azure Pipelines
Automated testing with Playwright, NUnit and Azure Pipelines
