Link Search Menu Expand Document

Automated Tests

Automated tests can be added to your Interactive Labs to verify they are working as expected. Each time the Lab is changed, the tests are re-run and the results displayed in the dashboard.

Cypress

Cypress.io is a complete end-to-end testing tool that can be used to verify that your interactive labs are working as expected.

You can learn more about Cypress at https://docs.cypress.io/guides/overview/why-cypress.html.

Cypress version 10.4.0 is currently supported.

Cypress Test Helpers

A collection of Cypress test helpers has been created to take the heavy lifting out of creating tests for your Labs. These helpers capture the core functionality required for a Cypress test to complete an interactive lab and verify its behavior.

Function Details Example
startScenario Start and visit the Lab being tested in the browser cy.startScenario();
terminalType Type and execute commands within the Terminal cy.terminalType("uname");
terminalShouldContain Wait for the terminal to contain the given text cy.terminalShouldContain('Linux');
terminalNotShouldContain Wait for the given text to be removed from the terminal cy.terminalNotShouldContain('some text');
terminalDoesNotContain Assert that the given text is currently absent from the terminal, without waiting as terminalNotShouldContain does cy.terminalDoesNotContain('some text');
contains Wait for the Lab page to contain the given text cy.contains('Start Scenario');
terminalShouldHavePath Wait for the path to be accessible from the terminal cy.terminalShouldHavePath("/tmp/build-complete.stamp");
terminalValueAfter Return the first space-delimited string after the last occurrence of the given text cy.terminalValueAfter("exit status=");
stepHasText Wait for the Lab instructions to contain the given text cy.stepHasText("Step 1: ...");

These helpers can be combined with the built-in functionality of Cypress to meet your requirements. More details of the built-in helpers can be found at https://docs.cypress.io/api/api/table-of-contents.html

Creating The First Test

  1. Within the lab you want to test, add a folder called .cypress.
  2. Create a file with the suffix _spec.js (such as test1_spec.js). This will contain your Cypress tests you want to run. The underlying Katacoda platform supports splitting your tests over multiple files, but subdirectories are not supported.
  3. Create your Cypress test using the helpers above. Ensure that cy.startScenario() is called to load your lab.
  4. Commit the test to Git and push to the remote Git Repository to trigger Katacoda re-processing.
  5. Visit the Dashboard to view the results.

A complete example can be found at https://github.com/katacoda/scenario-examples/tree/main/automatedtest

Automatically-Generated Tests

In order to reduce the effort required to get initial test coverage for your labs, we are working towards drafting tests that you may then customize to your needs. They will be delivered to you as a Gitlab Merge Request so that you can adjust them before merging if needed.

These tests are generated based on the commands embedded in your lab. They tend to work best for labs that can by completed solely by running the commands embedded in them. If your lab requires the learner to perform a manual step such as editing a file, you will need to adjust the generated test.

They currently support running commands in a Bash shell or in a Python REPL.

Example

The following is an example of a test to verify the interactive lab:

describe('My First Test', () => {
  before(() => {
    // This would start and visit the Lab being tested. Without
    // this the test will fail.
    cy.startScenario();
  });

  it('finds the content "Start Scenario"', () => {
    cy.contains('Start Scenario');
  });

  it('runs commands', () => {
    cy.terminalType("uname");

    cy.terminalShouldContain('Linux');
  })
})

A complete example can be found at https://github.com/katacoda/scenario-examples/tree/main/automatedtest.

More examples of using Cypress can be found at https://docs.cypress.io/examples/examples/recipes.html#Fundamentals.

View Results

The results of the test execution can be viewed in the Dashboard. You can find test runs for your organization at https://dashboard.katacoda.com/content/testruns. If you have tests in place, a test run will be triggered automatically whenever changes are pushed to your labs’ repository. You can also re-run all of your tests at once by clicking the “Force Retest” button.

Katacoda Testruns Dashboard

When you go into the testruns logs, you will see the output from Cypress execution. This will help you debug the issue if there are failing tests.

Katacoda Testruns Logs

Under the Covers

Want to see what’s happening under the covers? Here’s a video of a test execution:

This happens on every change to your repo to ensure your labs and environments are ready for your learners.