Verified Steps in Labs

Ensuring that users have successfully completed certain actions before the proceed can be a useful technique to ensure that are paying attention and they won’t hit problems later on in the scenario.

Verification Script

If you include verification scripts for the step(s) in your lab, when the user clicks Continue, a script is run against the environment. This provides some of the functionality of a challenge to your labs.

NOTE: If you are trying to show/hide content (for example, asking a learner to think about a problem prior to seeing the solution) without the need for explicit verification, please use the show/hide markup instead.

For the user to be allowed to proceed, the script should exit with status 0, which indicates success (mirroring how most commands return status 0 on success and non-zero on error). For example, the following verify script checks to make sure the user has run git init before they can proceed:

#!/bin/bash

[ -d /home/scrapbook/tutorial/.git ]

(See #useLegacyDoneProtocol below for a legacy, deprecated protocol that uses the text “done” to indicate success.)

To include the verification to a step, you add it to the step object inside the "steps" array in index.json. For example:

"details": {
    "steps": [
        {
            "title": "Verified Step Example",
            "text": "step1.md",
            "verify": "step1-verify.sh"
        }
    ]
}

Verification Failure Reasons

Authors are encouraged to provide the learner with a hint about why verification failed. To do this, the verification script should write to the file given by $VERIFICATION_FAILURE_REASON. The script must still exit with non-zero status to indicate failure. For example:

if ! [ -f hello.txt ]; then
  echo "Expected to find hello.txt in `pwd`, but it was not found." > "$VERIFICATION_FAILURE_REASON"
  exit 1
fi

# Additional checks...

exit 0

The message will be shown to the user with the verification failure notification. No formatting is supported; the message will be displayed as a single line within the narrow sidebar column, so it may wrap.

This feature is not available with #useLegacyDoneProtocol.

Answsers

To provide users with more information or context, an “answer” section can be provided inside the step object in index.json:

"details": {
    "steps": [
        {
            "title": "Show Answers",
            "text": "step2.md",
            "verify": "step2-verify.sh",
            "answer": "step2-answer.md"
        }
    ]
}

This creates the “Show Solution” option inside the lab. Here is an example from the learning platform:

Solutions

To aid in testing of labs with verified steps, add a solution script for each step that has a verification script. The solution script should automate the process that the learner is expected to perform and satisfy the verification script.

The solution script will not be offered to learners; it will only be available to authors and editorial staff for the purpose of testing and review.

The solution script will be invoked by Instruction-Guided Tests when they are run. This provides test coverage for executable actions in your instructions as well as your verification script. (Tests will fail if verification passes before the solution is applied.)

If the solution is successful, click “Continue” to retry verification and proceed to the next step. If the solution fails to apply successfully, you’ll have an option to bypass verification, which can be useful to review instructions in subsequent steps before solution scripts are fully working.

To configure a solution script for a step, add a “solution” key that specifies the solution script file name in your lab directory. The solution script should adhere to the same protocol as the verification script: it should exit with status 0 when the solution is successfully applied.

"details": {
    "steps": [
        {
            "title": "Verified Step Example",
            "text": "step1.md",
            "verify": "step1-verify.sh",
            "solution": "step1-solution.sh"
        }
    ]
}

For example, if the verify script checks to make sure the user has run git init, then the solution should do just that:

#!/bin/bash

set -e

cd /home/scrapbook/tutorial/.git
git init

Solutions may be provided with or without an answer.

If no solution is provided, an author or editorial staff member can still bypass verification without applying any solution. Although this allows access to the instructions of later steps for copy editing purposes, it does not allow testing of any commands in the instructions. For this reason, we encourage you to provide solutions for all verified steps that you create.

useLegacyDoneProtocol

NOTE: This protocol is deprecated and support for it will eventually be dropped.

This section outlines the old behavior, which can be enabled for a particular step by setting the useLegacyDoneProtocol key to true in the step object in index.json:

"details": {
    "steps": [
        {
            "title": "Verified Step Example",
            "text": "step1.md",
            "useLegacyDoneProtocol": true,
            "verify": "step1-verify.sh"
        }
    ]
}

For the user to proceed, the script should output the text “done.” For example, the following verify script checks to make sure the user has run git init:

#!/bin/bash

[ -d /home/scrapbook/tutorial/.git ] && echo "done"

Keep in mind that this has been replaced by newer behavior, based on exit status, which is more concise and more closely follows typical unix conventions to indicate success or failure. (See section Verification Script.)

Authors are encouraged to remove the useLegacyDoneProtocol attribute when they are able to update their scripts to use the newer protocol.