Running tests on EPL apps

[Apama_team]

Introduction

Following on from the first blog post that introduces our new Apama EPL Apps tools SDK, the aim of this article is to take you through the steps involved in creating and running tests on your EPL apps. If you have not already read the previous article or checked out the EPL Apps Tools SDK on GitHub, I recommend you do this before continuing.

In this demonstration I have an EPL app that I wish to test. I have also already written a test (in EPL) for that app. If you have yet to write any tests for your EPL apps, a guide for how this can be done can be found here. The sample EPL app that I will be testing for correctness, AlarmOnMeasurementThreshold , can be found in the /samples/apps/ directory of the Apama EPL Apps tools SDK. Except for a couple of small adjustments, the contents of AlarmOnMeasurementThreshold.mon is identical to the “Raise Alarm if Measurement exceeds a threshold value” sample available in Apama EPL Apps.

The behavior of AlarmOnMeasurementThreshold is simple: it listens for measurement events, and if it receives a measurement with a value over 100.00, it will raise a MINOR-level alarm in Cumulocity IoT. AlarmOnMeasurementThresholdTest is a test script that can be found in the Input directory in any of the sample test cases (for example, /samples/TestInEPLApps/Input/AlarmOnMeasurementThresholdTest.mon ). The test sends some mock measurements above and below the threshold value and checks whether an alarm is raised in each case.

Running the test manually in Apama EPL Apps

Before exploring how we can use the test framework, I will first demonstrate how to manually run the test in the Cumulocity IoT interface. This should give you insight into some of the things that are automated by the PySys test framework. If you are only interested in the test framework, feel free to skip this section and head straight to the “Running the test in EPL apps using the PySys framework extension” section below.

To run the test manually, we must first upload the EPL app and the test to your Cumulocity IoT tenant. You can either do this using the ‘Import EPL’ button in the Apama EPL Apps web interface, or you can use the CLI eplapp.py tool. Use the commands below with your real tenant details instead of the placeholders.

On Windows:

cd <path-to-your-apama-eplapps-tools-checkout>\apama-eplapps-tools

scripts\eplapp.cmd deploy -c <your c8y url> -u <your c8y username> -p <your password> -f samples\apps\AlarmOnMeasurementThreshold.mon -d "Listens for measurements and raises an alarm if a measurement with a value of over 100.0 is received."

scripts\eplapp.cmd deploy -c <your c8y url> -u <your c8y username> -p <your password> -f samples\TestInEPLApps\Input\AlarmOnMeasurementThresholdTest.mon -n TestApp -d "Test for AlarmOnMeasurementThreshold. Sends mock measurements above and below the threshold and checks to see whether an alarm is raised in each case." --inactive

Or on Linux:

cd <path-to-your-apama-eplapps-tools-checkout>/apama-eplapps-tools/

./scripts/eplapp.py deploy -c <your c8y tenant url> -u <your c8y username> -p <your c8y password> -f ./samples/apps/AlarmOnMeasurementThreshold.mon -d "Listens for measurements and raises an alarm if a measurement with a value of over 100.0 is received."

./scripts/eplapp.py deploy -c <your c8y tenant url> -u <your c8y username> -p <your c8y password> -f ./samples/TestInEPLApps/Input/AlarmOnMeasurementThresholdTest.mon -n TestApp -d "Test for AlarmOnMeasurementThreshold. Sends mock measurements above and below the threshold and checks to see whether an alarm is raised in each case." --inactive

Now go to Apama EPL Apps and you should see that the .mon files have been uploaded. Make sure AlarmOnMeasurementThreshold is active, then activate TestApp to run the test:

Deciphering test results

Now that the test has been executed, how can we determine the outcome?

Go to the Alarms tab in the Cockpit – a MINOR alarm should have been raised by AlarmOnMeasurementThreshold from when TestApp sent a measurement above the threshold:

The key point is that no MAJOR alarms were raised because the test passed. If the test had failed, we would observe a new MAJOR alarm relating to this failure.

If we visit the Apama-ctrl application log, we can also see the log messages from TestApp (for information on how to check the log files of the Apama-ctrl microservice, see Managing applications in the Cumulocity IoT User guide):

Here we can explicitly see the messages that confirm that both of our test cases, belowThresholdTest and aboveThresholdTest , have passed.

What happens when a test fails?

To demonstrate what a test failure looks like, I will deliberately introduce an error into AlarmOnMeasurementThreshold . By commenting out the line in AlarmOnMeasurementThreshold that sends an alarm will cause the aboveThresholdTest test case in TestApp to fail as there will no longer be a resulting alarm when one is expected:

Save the change to AlarmOnMeasurementThreshold and activate it again. Then reactivate TestApp to re-run the test. This time we can see that a MAJOR alarm was raised with the message “aboveThresholdTest: ThresholdExceededAlarm not raised when one was expected – FAIL”:

We can also see the this message logged at ERROR in the application log:

Running the test in EPL apps using the PySys framework extension

The steps involved in running the test manually may seem simple enough… but imagine if we had a collection of several tests that we needed to run, or if we wanted to rerun our tests at a later date. If we had to upload and activate files to EPL apps, then check in the Cumulocity IoT interface for alarms every time we wanted to run a test, this would quickly become cumbersome and messy. This is where the PySys test framework steps in to make life much easier. Furthermore, once you have done the initial configuration for a PySys project – something you only have to do once for each collection of tests – the framework should significantly simplify the case of running a single test too.

The PySys test framework extension gives you the option of running your tests either in the Cumulocity IoT cloud, or with a locally running correlator connected to the Cumulocity IoT platform (which can save cloud resources). For this demonstration I will be running my test in the cloud, but information on how you can run your tests with a local correlator can be found here.

If you haven’t already got Pysys installed and you wish to follow along, details on how to install it can be found here. There you can also find instructions for how add PySys to your path, for easy invocation of the PySys scripts.

Creating a PySys project and adding a test configuration

A PySys project is simply a group of tests that share a pysysproject.xml file that configures how those tests should be run.

You can create a new PySys project with the command pysys makeproject :

This creates a pysysproject.xml file in the current directory. To configure the PySys project for use with the EPL Apps Tools extension, open the new XML file and add the following properties:

<!-- User-defined properties -->
<property name="EPL_TESTING_SDK" value="${env.EPL_TESTING_SDK}"/>
<property name="EPL_APPS" value="${env.EPL_APPS}" default="${env.EPL_TESTING_SDK}/samples/apps"/>

<property name="CUMULOCITY_SERVER_URL" value="${env.CUMULOCITY_SERVER_URL}" default="https://mytenant.myorg.com" />
<property name="CUMULOCITY_USERNAME" value="${env.CUMULOCITY_USERNAME}" default="myUserName" />
<property name="CUMULOCITY_PASSWORD" value="${env.CUMULOCITY_PASSWORD}" default="myPassword" />

Here you can replace the default values for each property with your own test configuration and Cumulocity IoT tenant credentials:

• The EPL_TESTING_SDK property should be set as the path to your copy of the EPL Apps tools SDK. As I checked out the repository in C:/dev , I will add a default value of C:/dev/apama-eplapps-tools .
• The EPL_APPS property should be set as the path to the directory containing the .mon files that will be deployed to Apama EPL Apps and tested. I have created a folder, C:/dev/EPLAppsTests/apps , that contains AlarmOnMeasurementThreshold.mon , so I will set the default value of the EPL_APPS property to C:/dev/EPLAppsTests/apps .
• The default values of the remaining three properties should be set to the credentials of your Cumulocity IoT tenant.

Alternatively you can choose to set each of these properties using environment variables before you run the test (see below), instead of changing the default values in pysysproject.xml .

Creating a PySys test

Now we have created and configured the PySys project, we are ready to create a PySys test. A basic PySys test has the following structure:

• An appropriately named test folder. This should be in the same directory as the pysysproject.xml file. This test folder will then contain:
  • An Input sub-folder:
    • This should contain any test .mon files (in our case, AlarmOnMeasurementThresholdTest.mon ).
  • A pysystest.xml file that provides configuration for the individual test (such as whether to run the test in the cloud or locally, and the particular .mon file that the test is validating).

Thus to create a test, I first need to make a new test folder called ThresholdTest in the same directory as the pysysproject.xml file. In ThresholdTest , I then create another sub-folder called Input containing AlarmOnMeasurementThresholdTest.mon . In this demonstration, we have just one test .mon file but it is possible for a test to have several. If there are multiple test files in the Input folder, each one will be injected to the correlator after the previous one has finished executing.

I will then add a pysystest.xml file to the test folder – the easiest way to do this is to simply copy the pysystest.xml file from one of the samples. As this demonstration will be running the test in Apama EPL Apps, I will copy the pysystest.xml from the TestInEPLApp sample test.

Open the pysystest.xml file and have a look at its contents. In the description block, you can add a test title and fill out the purpose tag for later reference. I also would like to draw your attention to the data block:

<data>    
    <class name="EPLAppsSimpleTest" module="${EPL_TESTING_SDK}/testframework/apamax/eplapplications/basetest"/>
    <user-data name="EPLApp" value="AlarmOnMeasurementThreshold"/>
</data>

The class tag determines how the test is executed. The module attribute points to the test framework’s basetest class; while the name attribute determines how to run the test. We will be running the test in the cloud so the name attribute should be set to EPLAppsSimpleTest . The user-data tag determines which EPL app in the EPL_APPS folder is the subject of the test (see “Creating a PySys project and adding a test configuration” above). In this example we will set the value to AlarmOnMeasurementThreshold , as this is the name of the .mon file that this test is checking for correctness. Defining the user-data tag is optional – if it is omitted, by default all of the .mon files in the EPL_APPS directory will be deployed as part of the test.

Running the test

To run the test, open a new terminal and navigate to the directory containing the test folder. If you did not change the default property values in the pysysproject.xml file as was discussed in the “Creating a PySys project and adding a test configuration” section, remember to set each of these properties as an environment variable now. Then execute the test using the following command:

pysys run [name of test(s)]

Providing the name of the test(s) is optional. If you do not provide this, all of the tests in the directory will be run successively.

Notice that running the test has created an Output folder in the test directory. It contains a run.log and platform.log file which is especially useful for debugging any test failures. If we now go to Apama EPL Apps in the Cumulocity IoT interface, we can also observe that the framework has automatically deployed our two .mon files as EPL apps:

Notice how the test framework deactivated both of the EPL apps deployed by the test after it finished executing. It is also worth noting that if we were to run another PySys test in the cloud, these apps would then be deleted (before the new test is run) as part of a pre-test cleanup.

What happens if a test fails?

To demonstrate a test failure, again I will deliberately introduce an error to AlarmOnMeasurementThreshold.mon so an alarm is no longer raised:

I’ll save the change, and re-run the test:

As you can see, the framework greatly simplifies the process of running a test, and reports the outcome in a readable format.

Summary of steps involved in creating and running a PySys test for an EPL app

• Start with a .mon file that is an EPL app to test for correctness (for example, AlarmOnMeasurementThreshold.mon ).
• Create an EPL test following the conventions set out in this guide (for example, AlarmOnMeasurementThresholdTest.mon ).
• Create a PySys project by doing the following:
  • In a terminal, run the pysys makeproject command.
  • Add the configuration properties EPL_TESTING_SDK , EPL_APPS , CUMULOCITY_SERVER_URL , CUMULOCITY_USERNAME , and CUMULOCITY_PASSWORD to the pysysproject.xml file that was created by the makeproject command. Make sure to set the value of the EPL_APPS property as the path to the folder containing the EPL files being tested.
• Create a PySys test case by doing the following:
  • Create an appropriately named test folder in the same directory as the pysysproject.xml.
  • Inside the test folder, add a folder called Input containing any EPL test files (for example, AlarmOnMeasurementThresholdTest.mon ).
  • Add a pysystest.xml file to the test folder.
    • Make sure the description and data block in pysystest.xml are set correctly.
• In a terminal, navigate to the directory containing the test folder and pysysproject.xml and execute the command pysys run [test folder].