Execute & Post Approve Results


Figure:  Run a Test Set for Formal Results Approval


Now that a Test Set exists, and its status is Ready for Execution, a VERA user, with permission to run Test Sets, runs a test on a remote Tosca machine. The integration then updates ALM Run Execution status and attaches a Tosca generated run results report to the Test Set. Approvers approve the Test Set after inspecting the Run results report.
Neither of these user roles requires a Tosca account.

Prepare to Execute

At this point, the Tosca Execution List should be vetted in the Tosca environment – that is, it is tested to make sure it runs as expected on one or more target machines.
Instruct Tosca to send ExecutionLists to specific machines by using Tosca "Configurations". If you do not include a Configuration in the target Test Event, Tosca sends the ExecutionList to ANY available remote Agent for execution.
The Integration requires a Test Event that is named the same as the ExecutionList to reside in the nearest "TestEvents" root folder. The Test Event is required to include the ExecutionList – a Configuration is optional.

Tosca Configurations

See the Tosca Documentation for details on using Tosca Configurations.
Use Tosca Configurations to set parameters that let Tosca know which machine, or subset of machines, should execute your Test Set. Tosca attempts to find machines that match the parameters in your Test Event's Configuration with parameter settings on the Remote Agent machine.
Tosca comes with some default parameters that already exist on Agent machines and Configurations – for example, HostName, Memory, and OperatingSystem. Additionally, your Admin can configure other parameters (and value pick lists) on the Tosca Server. Once defined on the Tosca Server, they automatically appear in Configurations and on Agent machines.
Example:
If you want to have a Test Event only execute on a specific machine, in Tosca, perform the following steps:

  1. On the target Agent, right-click the Agent icon in the system tray and select 'Configure Agent' – copy the value in the HostName parameter. (Set it, if it's blank.)
  2. Checkout the Configuration, or Create one, as necessary.
  3. Type the target machine name in the HostName parameter.
  4. Check out the target Test Event for the Execution List
  5. Add it to your Test Event for the Execution List.
  6. Checkin All

Now, when you execute that Test Event, Tosca will attempt to find an Agent machine that matches the parameter values set in the Configuration. If the Test Event matches a Test Set name in VERA, you can run the Test Event from VERA.
The only remaining preparation is to make sure target remote machines are up and running. Typically, this is done in Tosca Commander, navigating to the correct "Configurations" object, right-clicking, and selecting "Refresh Agents". If your target Agent machine shows Green (Idle) or Orange (Executing), your Agent is ready to accept your request to Run a Test Set.

Steps to Execute a Test Set

In ALM/VERA, execute a Test Set by selecting it, then:

  1. Click the Tosca toolbar button. A Tosca Action Menu displays.
  2. Click 'Run Test Set'. The Tosca Test Set Execution dialog displays. If this button is disabled, hover the mouse over it to see why.
  3. Make sure it is the correct Test Set and click the 'Run' button.
  4. In the Status window, messages are logged, as the Run progresses. In order to see detailed progress on the remote Agent, use Tosca's Monitor screen from a browser.
  5. When the run has completed, dismiss the dialog.
  6. Click the Refresh button and inspect the attached Run report in the 'Attachments' tab.


Once a Test Set has runs, it cannot be updated – a VERA "Revise" operation is no longer allowed. This is to maintain CFR 21 Part 11 compliance in ALM, as a revision risks removing a Test Instance which causes ALM to automatically delete all associated runs.
Additionally, note that, post execution, a Tosca Test Set will have Run instances with Execution Status, but will not have Step results. All step results are contained in the attached Run report.