PSO-testing

Tables of contents

• About PSO-testing
• Technologies recommended
• Playwright
• Structure
• Writing tests
  • Keywords
  • Background
  • Steps tense
  • Scenario outline
• How to Run tests
• How to generate html report
• How to use Checkstyle linter
• How to add properties file
• Conventions
• CI Pipelines
  • Pipeline Stages
  • Merge Requests Pipeline
    • Recommended Branch Naming Conventions
  • Scheduled Pipeline

About PSO-testing

The PSO-testing is an automated test framework designed allow to automate test cases about Commission European projects.

Technologies recommended

• Java 11
• Maven 3.9.6
• IntelliJ Community
• maven-atf-reports:1.0.0 (request to testing tower)

Playwright

Playwright (https://playwright.dev/) is a framework used to automate tests on web pages. It interacts with page elements (named locators) to check their behavior. Playwright uses core version of the browser (chromium, webkit and stable version of Firefox). It allows testing on stable browser versions which are not polluted by branding content and which are the most used. The version of those browsers will be the same for all people using the test project. In addition, it can also run tests on the local chrome and edge version.

Structure

Configuration file is split in data and locators (src/main/java/configuration):

• locators: all locators should be defined here. Each ui page should have its own java class.
• data: all test data should be defined here. Each ui page should have its own java class.

Helpers file for all helpers that supports our stepDefinitions (src/main/java).

Features file where should be defined all scenario tests. Use scenario outline if needs but avoid define test data here (src/main/java).

Framework file where should be defined all code to allow run the tests (src/test/java).

StepDefinitions folder where should be defined all cucumber steps (src/test/java).

TestRunner folder where should be defined the cucumber test runner (src/test/java).

resources should contain config.properties file with sensitive data:

//SIMPL-LABS
URL=https://poc-simpl-labs.dev.simpl-europe.eu/data-spaces
USERNAME=
PASSWORD=
SECRET_KEY=

//SIMPL-OPEN
URL_WIZARD=https://sd-creation-wizard.uatpso.com

//COMMON
IS_HEADLESS=false

Writing tests

Writing test cases in a correct way requires you to respect some easy rules. Our test cases are written with Gherkin syntax (https://cucumber.io/docs/gherkin/reference/).

All scenario and steps name should start with a lowercase letter.

Keywords

Gherkin as three main keywords for steps:

• Given for prerequisites steps
• When for actions steps
• Then for expectations steps

Keywords shouldn't be mixed: You should have this order of keywords: Given, When, Then. Note that a test couldn't have any Given steps if there is no prerequisites and in some rare cases it could happen that a test has no When steps.

E.g.:

Scenario: a user logs in
Given a user opened the login form
When the user fills the login form
And the user validates the login form
Then the user name should be displayed

Background

If all tests of your feature share some common steps, you can put them in a background. All background steps should be of Given type.

E.g.:

Background:
Given the website is displayed
And the user accepted the cookies

Scenario: a user logs in
Given a user opened the login form
When the user fills the login form
And the user validates the login form
Then the user name should be displayed
But the login button shouldn't be displayed

Steps tense

Each type of step has its own tense rule.

• Given steps should be written in past tense most of the time as they represent prerequisites actions that have to be donne for the test's main action to be executed. But sometimes when the step is written in a passive way, given steps could be written in present tense
• When steps have to be written in present tense as they represent the main actions that will be performed in the test
• Then steps have to be written in conditional tense as they represent an expected behavior that could be not right when tested

If you have a step that is defined in a certain type and could be used as another type in another test, the step should be duplicated and written with the type format.

E.g.:

Scenario: a user logs in
Given a user opened the login form
When the user fills the login form
And the user validates the login form
Then the user should be logged in

Scenario: a user logs out
Given a user is logged in
When the user clicks the log out button
Then the user shouldn't be logged in

Scenario outline

In some cases, a scenario can be used for several test cases with only some variations. In such a situation you can use a Scenario outline which will allow you to execute tests with parameters. To do that, you'll have to use create a * Scenario Outline* instead of a Scenario and define your parameters into your steps by putting them between **<> ** (parameters should be written in pascal case). Then you'll have the define the parameters combinations in the Examples section.

E.g.:

Scenario Outline: a user <userStatus> logs in
Given a user opened the login form
When the user fills the login form
And the user validates the login form
Then the user <loginResult> be logged in

Examples:
| userStatus | loginResult |
| valid      | should      |
| invalid    | shouldn't   |

How to run tests

ATTENTION: BEFORE RUN ANY TEST CONFIGURE PROPERTIES FILE

Run the following command

To simplOpen project

// to api testing
mvn clean test -Dtest=ApiSimplOpenRunner

// to ui testing
mvn clean test -Dtest=UiSimplOpenRunner

To simplLabs project

// to api testing
mvn clean test -Dtest=UiSimplLabsRunner

To run a test by tag it just needed to add the following option to the previous command (consider the tag @TCW01 as an example):

-Dcucumber.filter.tags="@TCW01"

How to generate html report

To generate the report of the first time it´s mandatory install the dependency maven-atf-reports. It´s not available in the internet. Talk with testing tower to get it

Run the following command

mvn com.qa.atf.reports:maven-atf-reports:1.0.0:generate@execution

After run the command with success, the html report will be available in target/pluma-report/overview-features.html

To add some information to report type the following code to the cucumber step:

 scenario.attach(<information to add>, "text/plain", <title>);

To add a screenshot to report type the following code to the cucumber step:

byte[] screenshot = page.screenshot(new Page.ScreenshotOptions());
scenario.attach(screenshot, "image/png", <title>);

How to use Checkstyle linter

Checkstyle is a development tool that helps programmers write Java code that adheres to a coding standard. It automates the process of checking Java code for adherence to a set of rules, which can include formatting, naming conventions, and other coding standards. By enforcing these rules, Checkstyle helps maintain code quality, readability, and consistency across a project.

Run Checkstyle locally: To run Checkstyle validation locally, use the following Maven command:

mvn checkstyle:check

How to add properties file

• Go to directory src/main/resources
• Add a file with name config.properties
• Add the information that is needed (see the chapter Structure)

Conventions

• cucumber tags: each feature file should be a tag with feature name and each test should be tag with test case number and initial letter of feature name (e.g. L01 - first scenario to login.feature).
• write typing: <<<<<<< HEAD
  • java class - pascal case
  • data and locators - snake case + upper case
  • feature files and methods name - camel case

CI Pipelines

Pipeline Stages

The following stages are included in the CI Pipeline for PSO Test Automation Repo:

• static-code-analysis --> includes jobs for running Checkstyle and SonarQube verifications for the repo
• test --> for running automation tests during Merge Requests and Scheduled Pipeline executions
• report --> for generating reports based on the test results in the previous stage

Merge Requests Pipeline

• test followed by report stages of the defined pipeline will be executed during Merge Request events
• It has been agreed that for every merge request, a subset of test cases (marked with tag '@smoke') will be executed
• TestRunner and Tags to be used to run the Automation tests will be derived from the branch name
  • xvfb-run mvn clean test -Dtest=<TESTRUNNER_TO_USE> -Dcucumber.filter.tags="<TAGS_TO_USE>"

Recommended Branch Naming Conventions

For Features (test cases):

The following branch naming convention is recommended when creating or updating test cases:

feature/<TestRunner_Identifier>-<JIRA_TestCaseId>

• <TestRunner_Identifier> could be any of the following values:
  • SOUI - Implies feature is related to SimplOpen UI → SimplOpen UI TestRunner is to be used for test execution
  • SOAPI - Implies feature is related to SimplOpen API → SimplOpen API TestRunner is to be used for test execution
  • SLUI - Implies feature is related to SimplLabs UI → SimplLabs UI TestRunner is to be used for test execution
  • SLAPI - Implies feature is related to SimplLabs API → SimplLabs API TestRunner is to be used for test execution
• <JIRA_TestCaseId> is the test case identifier in JIRA. e.g. SIMPL-1234

NOTE:

• Remember to tag the newly created or updated test case with the JIRA id. e.g. @SIMPL-1234
• If the TestRunner_Identifier is not provided in the branch name, SimplOpen UI TestRunner will be used as default
• If the JIRA_TestCaseId is not provided in the branch name, "@smoke" tag will be used as default

Illustrations:

1. feature/SOUI-SIMPL-111 --> For MRs, Tests will be executed with SimplOpen UI TestRunner and Tags "@smoke or @SIMPL-111". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=UiSimplOpenRunner -Dcucumber.filter.tags="@smoke or @SIMPL-111"

2. feature/SOAPI-SIMPL-222 --> For MRs, Tests will be executed with SimplOpen API TestRunner and Tags "@smoke or @SIMPL-222". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=ApiSimplOpenRunner -Dcucumber.filter.tags="@smoke or @SIMPL-222"

3. feature/SLUI-SIMPL-333 --> For MRs, Tests will be executed with SimplLabs UI TestRunner and Tags "@smoke or @SIMPL-333". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=UiSimplLabsRunner -Dcucumber.filter.tags="@smoke or @SIMPL-333"

4. feature/SLAPI-SIMPL-444 --> For MRs, Tests will be executed with SimplLabs API TestRunner and Tags "@smoke or @SIMPL-444". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=ApiSimplLabsRunner -Dcucumber.filter.tags="@smoke or @SIMPL-444"

5. feature/SOAPI-some-feature --> For MRs, Tests will be executed with SimplOpen API TestRunner and default Tag " @smoke". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=ApiSimplOpenRunner -Dcucumber.filter.tags="@smoke"

6. feature/some-feature --> For MRs, Tests will be executed with default SimplOpen UI TestRunner and default Tag " @smoke". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=UiSimplOpenRunner -Dcucumber.filter.tags="@smoke"

7. this-is-some-feature --> For MRs, Tests will be executed with default SimplOpen UI TestRunner and default Tag " @smoke". > >

• Following command will be executed in pipeline:

• xvfb-run mvn clean test -Dtest=UiSimplOpenRunner -Dcucumber.filter.tags="@smoke"

For Spikes:

The following branch naming convention is recommended for spikes and changes not related to test cases:

spike/<short-description>

• <short-description> is a text description about the spike
• During MRs for spike branches, Tests will be executed with default SimplOpen UI TestRunner and default Tag "@smoke".
• xvfb-run mvn clean test -Dtest=UiSimplOpenRunner -Dcucumber.filter.tags="@smoke"

Scheduled Pipeline

• test followed by report stages of the defined pipeline will be executed during Scheduled Pipeline events
• It has been agreed that for every Scheduled Pipeline execution, regression test cases (marked with tag '**@regression **') will be executed
• TestRunner and Tags to be used to run the Automation tests have to be provided in the Scheduled Pipeline definition
• Any number of Scheduled Pipelines can be created as needed. e.g. Dedicated Daily Regression tests for different applications (SimplOpenUI, SimplOpenAPI, SimplLabsUI, SimplLabsAPI etc)
• Targets for Scheduled Pipelines can be the main 'develop' branch or any other branch
• Scheduled Pipelines can be created using the UI (necessary permissions needed): pso-test-automation --> Builds --> Pipeline schedules--> New schedule

Example illustration:

• Setting up a daily regression for SimplOpen UI: >

• Running daily at 19:00 UTC+2

• With target branch 'develop'
• Using TestRunner 'UiSimplOpenRunner' (via variable RUNNER_TO_USE)
• Matching tags '@regression or @wizard' (via variable TAGS_TO_USE)