Tag Archives: neodymium

Accessibility, Automation, Java, Open Source

Automated Accessibility Testing with Google Lighthouse and Neodymium

2025-03-06 Jonas

TL;DR: Accessibility is a crucial aspect of web development that ensures everyone, regardless of their abilities, can access and use your website. Web Content Accessibility Guidelines (WCAG) provide a standard for web accessibility, and adhering to them is not only ethically important but also increasingly becoming a legal requirement.

One valuable tool for checking WCAG compliance is Google Lighthouse. It can help identify accessibility issues during development. And when integrated into your test automation project with tools like Neodymium, it can significantly streamline your accessibility testing and monitoring process.

What is WCAG and Why is it Important?

WCAG, or Web Content Accessibility Guidelines, are a set of standards developed to make web content more accessible to people with disabilities. These guidelines cover a wide range of disabilities, including visual, auditory, motor, and cognitive impairments. By following WCAG, you ensure that your website is usable by a larger audience. Approximately 15% of the world’s population has some form of disability.

WCAG compliance is also gaining traction legally, with governments in the US and EU enforcing accessibility standards. Non-compliance can lead to lawsuits and of course to less revenue and visitors.

The Role of Test Automation

Automating WCAG checks using tools like Lighthouse can help find and fix accessibility flaws early in the development cycle, saving time and resources. It also reduces the burden on manual testers by automatically generating accessibility reports.

Most importantly, test automation maintains the achieved level of accessibility of the application through regression testing.

If you have an existing or planned test automation project, it is crucial to incorporate WCAG compliance into your testing strategy. This is why we have integrated Lighthouse into Neodymium, allowing you to easily extend your test cases with accessibility checks.

How to Use Lighthouse with Neodymium

Here’s a simplified example of how to integrate Lighthouse with Neodymium:

Prerequisites: Ensure you have Neodymium set up and Lighthouse CLI installed. Lighthouse only works with Chrome or Chromium-based browsers.
Triggering Lighthouse: Neodymium provides the LighthouseUtils.createLightHouseReport(name) method. Call this method at the desired point in your test to generate a Lighthouse report.
Analyzing Results: After the test run, the Lighthouse report will be attached to the final test report (e.g., Allure report).

@NeodymiumTest
public void testHomepage() throws Exception
{
    var homePage = OpenHomePageFlow.flow();
    LighthouseUtils.createLightHouseReport("Homepage");
}

@NeodymiumTest

public void testHomepage() throws Exception

{

var homePage = OpenHomePageFlow.flow();

LighthouseUtils.createLightHouseReport("Homepage");

}

Showing how a Lighthouse report in Neodymium looks like. It shows some test case step descriptions and result data such as performance 90%, Accessability 85% and more — Lighthouse report in Neodymium’s Allure report

Important: Creating a report requires Lighthouse to refresh the current web page. That approach doesn’t work with some types of pages or states.

Lighthouse Validation

Neodymium allows you to validate Lighthouse scores and specific audits programmatically.

Scores

Lighthouse provides scores for Performance, Accessibility, Best Practices, and SEO. You can set thresholds in Neodymium to automatically fail tests if these scores fall below a certain value.

To properly verify those scores, we implemented configuration properties in the neodymium.properties file.

neodymium.lighthouse.assert.thresholdScore.performance = 0.5
neodymium.lighthouse.assert.thresholdScore.accessibility = 0.5
neodymium.lighthouse.assert.thresholdScore.bestPractices = 0.5
neodymium.lighthouse.assert.thresholdScore.seo = 0.5

neodymium.lighthouse.assert.thresholdScore.performance = 0.5

neodymium.lighthouse.assert.thresholdScore.accessibility = 0.5

neodymium.lighthouse.assert.thresholdScore.bestPractices = 0.5

neodymium.lighthouse.assert.thresholdScore.seo = 0.5

Audits

You can also validate specific Lighthouse audits. For example:

neodymium.lighthouse.assert.audits = aria-roles aria-text

1	neodymium.lighthouse.assert.audits = aria-roles aria-text

This property is used to identify which audit IDs should be evaluated. The example checks for errors in the Lighthouse audits for aria-roles and aria-text. If either or both of these audits find an error, an exception will be thrown. For a complete list of existing audit IDs and their corresponding titles, please refer to our documentation.

Limitations of Automated Accessibility Testing

While automated tools like Lighthouse are incredibly helpful, they can’t catch all accessibility issues. Some aspects, like the meaningfulness of alt text or the logical flow of keyboard navigation, still require human judgment.

It’s important to note that it is possible to build an inaccessible website that still gets a high Lighthouse score. Automated tools provide a good starting point, but they should be complemented by manual testing and user testing with people with disabilities.

Conclusion

Integrating Google Lighthouse with Neodymium into your test automation project is a great way to improve your website’s accessibility. It helps catch issues early, automates report generation, and enables validation of accessibility scores and audits. While not a complete solution, it is a powerful tool in making the web more accessible to everyone.

More information on how to integrate Lighthouse in Neodymium are available at https://github.com/Xceptance/neodymium/wiki/Accessibility

Automation, Open Source, Testing

Neodymium 5.1 – Get more done faster

2025-02-11 Rene

TL;DR: Neodymium 5.1 has been released with a host of impressive new features. These include full-page screenshots in reports, enhanced control over JSON assertions, accessibility testing via Google Lighthouse, and simplified session handling and URL validation.

The newest enhancements primarily focus on these key areas to boost testing efficiency and user experience.

Enhanced Reporting: We’ve significantly improved the quality and usability of test reports. You now get more informative and insightful results with features like full-page screenshots, enhanced JSON assertions for easier data comparison, and a streamlined report structure for improved readability.

Accessibility Testing: Recognizing the growing importance of web accessibility, we’ve integrated Google Lighthouse to automate accessibility checks. This allows you to easily identify and address accessibility issues within your web applications, ensuring compliance with WCAG standards.

Streamlined Workflows: We’ve introduced several features to streamline your testing processes. These include improved session handling for cleaner test environments, enhanced configuration options for greater flexibility, and robust URL validation to prevent unintended access to sensitive systems and ensure test stability.

These improvements are all about making your testing smoother and more dependable. You’ll get better results and learn more from them, so you can build even better software! So, let’s get into the details, shall we?

Enhanced Reporting

Neodymium 5.1 brings significant improvements to its reporting capabilities, providing you with more insightful and informative test results.

Improved Screenshots

You can now capture full-page screenshots with highlighted viewports for better visibility of the test context. Additionally, the last element, the test has interacted with, will be highlighted. This makes it easier to pinpoint the exact location of an interaction and understand the state of the application at that time.

# enable advanced screenshots and full-page screenshots
neodymium.screenshots.enableAdvancedScreenshots = true
neodymium.screenshots.fullpagecapture.enable = true
neodymium.screenshots.fullpagecapture.highlightViewport = true

# enable advanced screenshots and full-page screenshots

neodymium.screenshots.enableAdvancedScreenshots = true

neodymium.screenshots.fullpagecapture.enable = true

neodymium.screenshots.fullpagecapture.highlightViewport = true

Shows a screenshot of a report with an embedded screenshot showing the marking of the viewpport — Screenshot within Viewport Marking

Test Data Overview

Gain a deeper understanding of your test data and streamline the debugging process by accessing test data in JSON format for each test case. This detailed data analysis capability allows you to quickly identify and resolve potential issues.

Screenshot showing how test data is shown in an Allure report — Test Data within the Allure Report

JSON Assertions

The JSON file comparison feature has been enhanced. It now provides improved accuracy in identifying differences between expected and actual results.

// assert that the files are equal
JsonAssert.assertEquals(expectedJson, actualJson, JSONCompareMode.STRICT_ORDER);

1 2	// assert that the files are equal JsonAssert.assertEquals(expectedJson, actualJson, JSONCompareMode.STRICT_ORDER);

A screenshot with test results on the left and a JSON comparison example on the right — JSON Comparison Within the Allure Report

Environment Information

We’ve added the browser used for testing to the report’s environment section to give you more context about the test results. Also, you can now add custom values to tell the difference between environments or profiles in your test runs, making it easier to figure out if any setup or deployment problems are causing issues.

# enable custom environment data
neodymium.report.environment.enableCustomData = true

# add the entry 'customData' with value 'customValue'
neodymium.report.environment.custom.customData = customValue

# enable custom environment data

neodymium.report.environment.enableCustomData = true

# add the entry 'customData' with value 'customValue'

neodymium.report.environment.custom.customData = customValue

Step Information and URLs

The clickable URLs included in each step of the report simplify and streamline the process to reproduce test failures and navigate to specific application pages at various points during test execution.

Reduced Report Clutter

The report now features fewer INFO sections, providing a cleaner and more concise overview with less visual noise, allowing for easier identification of critical information.

Accessibility Testing

Neodymium 5.1 now includes Google Lighthouse to help you identify and fix accessibility issues in your web applications. This integration supports your compliance with WCAG standards, and shows that we understand how important web accessibility is.

Shows a screenshot of a report with an embedded Google Lighthouse report — Lighthouse Report Embedded in Neodymium’s Allure Report

Additionally, you can assert against Lighthouse report data to track changes in state or outcome.

// create a lighthouse report and assert the defined scores and audits
LighthouseUtils.createLightHouseReport("lighthouseReportName");

1 2	// create a lighthouse report and assert the defined scores and audits LighthouseUtils.createLightHouseReport("lighthouseReportName");

# set the thresholds for each metric
neodymium.lighthouse.assert.thresholdScore.performance = 0.9
neodymium.lighthouse.assert.thresholdScore.accessibility = 0.9
neodymium.lighthouse.assert.thresholdScore.bestPractices = 0.9
neodymium.lighthouse.assert.thresholdScore.seo = 0.9

# set specific audits to assert
neodymium.lighthouse.assert.audits = is-on-https link-name

# set the thresholds for each metric

neodymium.lighthouse.assert.thresholdScore.performance = 0.9

neodymium.lighthouse.assert.thresholdScore.accessibility = 0.9

neodymium.lighthouse.assert.thresholdScore.bestPractices = 0.9

neodymium.lighthouse.assert.thresholdScore.seo = 0.9

# set specific audits to assert

neodymium.lighthouse.assert.audits = is-on-https link-name

Furthermore, this enables WebVitals measurement and verification capabilities.

Workflow Enhancements

Several features help to streamline the testing process and improve overall test automation efficiency.

Popup Blocker

The improved popup blocker boosts test stability by minimizing disruptions from unexpected popups, which can lead to inconsistent test results.

# define locator of the close button for each pop up to handle
neodymium.popup.cookieBanner = #privacy-message .close
neodymium.popup.anotherOne = #someId button

# define locator of the close button for each pop up to handle

neodymium.popup.cookieBanner = #privacy-message .close

neodymium.popup.anotherOne = #someId button

Session Handling

Neodymium now utilizes a dedicated annotation to run @BeforeEach and @AfterEach methods in individual, fresh browser sessions. This guarantees a consistent environment for setup and tear down, and minimizes any influence from prior test steps.

@StartNewBrowserForSetUp
@BeforeEach
public void setUp()
{
    // perform the setup using a dedicated browser
}

// annotation to force a new browser for cleanup
@StartNewBrowserForCleanUp
@AfterEach
public void cleanUp()
{
    // Cleanup code using a dedicated browser
}

@StartNewBrowserForSetUp

@BeforeEach

public void setUp()

{

// perform the setup using a dedicated browser

}

// annotation to force a new browser for cleanup

@StartNewBrowserForCleanUp

@AfterEach

public void cleanUp()

{

// Cleanup code using a dedicated browser

}

Properties

Environment variables can now be used to override Neodymium properties, providing greater flexibility and simpler integration with CI/CD pipelines. This allows for more dynamic test adjustments based on the specific environment or build.

URL Validation

Restrict test access to certain sites only by defining include and exclude lists. This prevents accidental access to live sites or sensitive data, mitigating the risk of unintended modifications. Any attempt to access a URL not included in the defined list will result in an immediate test failure, providing early feedback and preventing further interactions.

# List of included URLs separated by whitespace
neodymium.url.includeList = https://www.xceptance.com/en/$

# List of excluded URLs separated by whitespace
neodymium.url.excludeList = https://posters.xceptance.io/.*Dining

# List of included URLs separated by whitespace

neodymium.url.includeList = https://www.xceptance.com/en/$

# List of excluded URLs separated by whitespace

neodymium.url.excludeList = https://posters.xceptance.io/.*Dining

Driver Command-Line Arguments

The configuration now allows for direct pass-through of driver command-line arguments, which streamlines driver setup, facilitates customization, and enables fine-tuning of driver behavior and test execution to meet your specific requirements.

Dependencies

We also updated numerous dependencies to keep them up-to-date and secure. Examples are Selenide, Allure, Log4J, and JUnit.

Summary

Neodymium 5.1 is now available! We value your feedback on new features and any defects encountered. Your input helps ensure Neodymium remains a powerful tool for your needs. You might even find a newer version of Neodymium when you read this article, depending on when you read it.

Automation, Open Source, Testing

A new magnetic force: Neodymium 5.0.0

2024-06-26 Peggy

TL;DR: We proudly announce the release of Neodymium 5.0.0. Neo is a Java-based test library for web automation that utilizes existing libraries (Selenide, WebDriver, Allure, JUnit, Maven) and concepts (localization, test multiplication, page objects) and adds missing components such as test data handling, starter templates, multi-device handling, and other small but useful everyday helpers.

Our R&D team has been busy brewing! So today we finally get back on the major release train and present you the new Neodymium 5.0.0. It comes with a lot of new little convenience features, giving you more control and possibilities on your test automation. A better browser control, improved configuration possibilities and a bunch of new annotations like @DataItem, @WorkInProgress, @RandomBrowser, will help you set up and configure your test automation to your specific needs. Even if a picture says more than 1000 words sometimes it’s a pain to see on a single screenshot, why the automation journey broke, to help you in such cases, we introduced video recording, to exactly see what happened during the whole user journey.

See below for what you can find inside the big box of updated code:

Continue reading →

Automation, Java, Open Source, Testing

Neodymium – An Open Source Framework for Web Testing

2019-02-26 Rene

TL;DR: Neodymium is a Java-based test library for web automation that utilizes existing libraries (Selenide, WebDriver, Allure, JUnit, Maven) and concepts (localization, test multiplication, page objects) and adds missing components such as test data handling, starter templates, multi-device handling, and other small but useful everyday helpers.

Motivation

As a company focused on quality assurance and testing, Xceptance always needs test automation software, especially end-to-end automation software. Several years ago we built a Firefox add-on that was designed to create and run browser automation. The tool was primarily used by people who didn’t necessarily have a strong background in software development. Today, the landscape is a bit different: Mozilla cut the cord on the APIs we were using and standard programming languages have largely taken over test automation because they are more flexible and less proprietary. These changes convinced us it was time to implement an idea we had already hatched, namely our own Open Source test automation project: Neodymium. It is written in and utilizes the Java platform, it is MIT licensed, and of course you will find it on GitHub: https://github.com/Xceptance/neodymium-library

Basis

There are many libraries out there to aid web automation in Java, so developers are faced with the task of choosing ones they like and somehow making them work together. On top of that, there are tasks that require some custom code to work properly. We identified the overall tooling problem mostly as a hurdle in getting started and setting up a project. Finally, there are always things missing such as test data handling, concurrency, and common patterns which you don’t want to have to develop yourself. We chose JUnit, Selenide, WebDriver, Maven, and Allure for the base tooling.

Selenide provides an easy-to-use API to control Selenium WebDriver. Allure offers good mechanics to generate useful reports based on the assertions and actions you perform throughout your test cases. Maven is used to set up the build and execution environment for our framework and all the test projects. We decided to use JUnit as the test runner since it is the de facto standard in the Java world, but we enhanced the capabilities of JUnit to do even more. At its heart, Neodymium is a JUnit runner that wraps default JUnit behavior and adds significant useful functionality to it.

Multiple Browsers

You want to be able to run the same tests for different resolutions and/or browsers to simulate the browsers most common among your users. Additionally, you need to be able to implement small differences within your test execution to address variants such as responsive designs or progressive web apps. So we added a way to run web browsers with different configurations and retrieve the current device type and resolution from within the test.

Neodymium provides a Java annotation that can be added to your test case, in order to run different browser setups. Neodymium is very flexible in configuring browsers, allowing you to fully leverage the Chrome device emulation offerings.

Test Data

Another common task is the execution of a test case with different data sets, such as testing address forms with all the relevant variations. The basic idea is to have test data and data sets in structured files next to your code, preferably as JSON, XML, property style, or simply CSV. Hence, we introduced an easy-to-use API to access the current data set and retrieve basic types from it. Furthermore, you can configure specific scenarios running only a subset or even no data set at all by adding proper annotations. To complete the picture, Neodymium supports test data on a global and package-level scope.

Localization

Another recurring topic in modern software projects is localization. Most of the web sites that are in need of test automation also support several locales. We decided to provide an out-of-the-box solution.

Neodymium’s localization feature makes use of a central translation file written in YAML format. YAML helps to structure the translations. Additionally, we implemented a simple way to override specific translations for different locales. The localized text can be easily retrieved using Neodymium API methods that are globally available.

Development Support

As it is essential to understand what your test is doing, we added a feature that enables you to slow down the test execution and highlight elements that match the current selector. Since you can chain selectors using Selenide, any chain of elements is also represented by the highlighting. With this feature activated, a developer can track down the cause of test failures much more easily. In addition, we provide information on how to set up logging in your project should you need that. Finally, we decided to use the Page Object pattern to organize the website-related code to reduce the maintenance effort and increase reusability.

Reporting

Allure is a widely used framework to generate reports. When using Neodymium with Selenide, your automation code also contributes report information. Your test classes and methods are listed as well as detailed Selenide automation commands. In case of errors, additional details such as screenshots and source code of the page in question are available. Neodymium also provides means to structure code blocks for reporting purposes.

Continuous Integration

Implementing principles of continuous integration will deliver more reliable software by increasing efficiency, and automation is nothing without a continuous integration environment. Yet in almost every development cycle you will eventually end up needing varied settings due to differences in your setup, which can get complicated. Neodymium provides support for extra configuration files during development to override the standard production settings as needed. Furthermore, the framework supports overriding properties that change the configuration of your test execution by setting environment variables or simply passing Java arguments.

Because automation is supposed to run quickly, Neodymium provides support for parallel test execution and also demonstrates that setup as part of the sample test suite.

Documentation and Templates

Does Neodymium address some of your test automation challenges? Does it sound like a good entry point for your test automation?

Neodymium is hosted on GitHub (https://github.com/Xceptance/neodymium-library), where the accompanying project wiki (https://github.com/Xceptance/neodymium-library/wiki) provides extensive documentation to help you get started and answer your questions.

You might also want to take a look at the comprehensive example projects using Neodymium with Cucumber (https://github.com/Xceptance/neodymium-cucumber-example) or plain Java (https://github.com/Xceptance/neodymium-example). We’ve even provided a template project (https://github.com/Xceptance/neodymium-template) to get you started automating in no time.

License

Neodymium is licensed under the MIT License.

Who Are We

We are Xceptance. A software testing company with strong commerce knowledge and projects with customers from all around the world. Besides Neodymium, we have developed Xceptance Load Test (XLT), a load and performance test tool that is available free of charge and features an extensive range of awesome features to make the tester’s and developer’s life easier.

If you are looking for test automation that also covers the performance side of life, take a look at XLT. You can write and run load tests with real browsers including access to data from the Web Performance Timing API. In case browsers are too heavy, XLT has other modes of load testing to offer as well.

We offer professional support for Neodymium as well as implementation and training services.