Automated Tests
Automated tests can be added to your Interactive Labs to verify they are working as expected. Each time the Lab is changed, the tests are re-run and the results displayed in the dashboard.
Cypress
Cypress.io is a complete end-to-end testing tool that can be used to verify that your interactive labs are working as expected.
You can learn more about Cypress at https://docs.cypress.io/guides/overview/why-cypress.html.
Cypress version 10.4.0 is currently supported.
Cypress Test Helpers
A collection of Cypress test helpers has been created to take the heavy lifting out of creating tests for your Labs. These helpers capture the core functionality required for a Cypress test to complete an interactive lab and verify its behavior.
Function | Details | Example |
---|---|---|
startScenario | Start and visit the Lab being tested in the browser | cy.startScenario(); |
terminalType | Type and execute commands within the Terminal | cy.terminalType("uname"); |
terminalShouldContain | Wait for the terminal to contain the given text | cy.terminalShouldContain('Linux'); |
terminalNotShouldContain | Wait for the given text to be removed from the terminal | cy.terminalNotShouldContain('some text'); |
terminalDoesNotContain | Assert that the given text is currently absent from the terminal, without waiting as terminalNotShouldContain does | cy.terminalDoesNotContain('some text'); |
contains | Wait for the Lab page to contain the given text | cy.contains('Start Scenario'); |
terminalShouldHavePath | Wait for the path to be accessible from the terminal | cy.terminalShouldHavePath("/tmp/build-complete.stamp"); |
terminalValueAfter | Return the first space-delimited string after the last occurrence of the given text | cy.terminalValueAfter("exit status="); |
stepHasText | Wait for the Lab instructions to contain the given text | cy.stepHasText("Step 1: ..."); |
These helpers can be combined with the built-in functionality of Cypress to meet your requirements. More details of the built-in helpers can be found at https://docs.cypress.io/api/api/table-of-contents.html
Creating The First Test
- Within the lab you want to test, add a folder called
.cypress
. - Create a file with the suffix
_spec.js
(such astest1_spec.js
). This will contain your Cypress tests you want to run. The underlying Katacoda platform supports splitting your tests over multiple files, but subdirectories are not supported. - Create your Cypress test using the helpers above. Ensure that
cy.startScenario()
is called to load your lab. - Commit the test to Git and push to the remote Git Repository to trigger Katacoda re-processing.
- Visit the Dashboard to view the results.
A complete example can be found at https://github.com/katacoda/scenario-examples/tree/main/automatedtest
Automatically-Generated Tests
In order to reduce the effort required to get initial test coverage for your labs, we are working towards drafting tests that you may then customize to your needs. They will be delivered to you as a Gitlab Merge Request so that you can adjust them before merging if needed.
These tests are generated based on the commands embedded in your lab. They tend to work best for labs that can by completed solely by running the commands embedded in them. If your lab requires the learner to perform a manual step such as editing a file, you will need to adjust the generated test.
They currently support running commands in a Bash shell or in a Python REPL.
Example
The following is an example of a test to verify the interactive lab:
describe('My First Test', () => {
before(() => {
// This would start and visit the Lab being tested. Without
// this the test will fail.
cy.startScenario();
});
it('finds the content "Start Scenario"', () => {
cy.contains('Start Scenario');
});
it('runs commands', () => {
cy.terminalType("uname");
cy.terminalShouldContain('Linux');
})
})
A complete example can be found at https://github.com/katacoda/scenario-examples/tree/main/automatedtest.
More examples of using Cypress can be found at https://docs.cypress.io/examples/examples/recipes.html#Fundamentals.
View Results
The results of the test execution can be viewed in the Dashboard. You can find test runs for your organization at https://dashboard.katacoda.com/content/testruns. If you have tests in place, a test run will be triggered automatically whenever changes are pushed to your labs’ repository. You can also re-run all of your tests at once by clicking the “Force Retest” button.
When you go into the testruns logs, you will see the output from Cypress execution. This will help you debug the issue if there are failing tests.
Under the Covers
Want to see what’s happening under the covers? Here’s a video of a test execution:
This happens on every change to your repo to ensure your labs and environments are ready for your learners.