Introduction
There’s a new Docker image in the Docker Hub registry for Software AG’s 10.3 release called Apama Builder. It differs from the Apama Correlator image in that it can be used to build Apama projects into new images via multi-stage builds. In this blog we will explore how leveraging the Apama Builder along with other common tools, we can easily create an Apama application in a container that is self-testing, self-deploying, and subject to continuous integration.
The Apama Builder image contains a suite of tools that enable the creation of images derived from the Apama Correlator image containing tested and deployed Apama applications. It provides the usual tools such as engine_deploy , engine_inject , the correlator and PySys used during the normal workflow. Additionally, it contains tools such as Ant, Java and Python and can easily be extended to contain any elements required by your build by installing them as part of the Docker build process.
To help explain the process and to demonstrate how Continuous Integration (CI) can be implemented using Travis, this article will refer to a project in GitHub that can be found here from a community contributor:
This uses a simple Java plug-in which needs to be built from Java sources, combined with an Apama application that uses the plug-in and a PySys test that will verify the application. The key to all of this is the authoring of a Dockerfile for your project which uses the Apama Builder image to produce the desired image.
Creating the Dockerfile
For your own project you will need to author a Dockerfile to produce the desired application image.
ARG APAMA_VERSION=10.3
ARG APAMA_BUILDER=softwareag/apama-builder:${APAMA_VERSION}
ARG APAMA_IMAGE=softwareag/apama-correlator:${APAMA_VERSION}
# Use the build environment
FROM ${APAMA_BUILDER} as builder
#build steps, tests, and deployment
RUN ant
RUN cd tests && pysys run
RUN engine_deploy
#create the working application image
FROM ${APAMA_IMAGE}
COPY --from=builder ${APAMA_WORK}/deployed ${APAMA_WORK}/deployed
#example startup command
CMD ["correlator", "--config", "deployed"]
The above example uses substitution variables to define the image names used to generate the final image. It defaults the version to 10.3 and the images to the Docker Hub versions for convenience. The various steps required to build, test and deploy the application need to be inserted into the Dockerfile.
Alternatively if you use Software AG Designer (available on Windows), there’s a new context menu in 10.3: Apama > Add Docker Support . This will add a Dockerfile to the top level of your project. For the majority of EPL-only Apama projects, the default Dockerfile this creates is all you need. This will correctly sequence injections and event files at startup, and includes all your configuration, with no changes required. The Designer Dockerfile will be similar in structure to the example above, and will default the images and version.
Building the image
Once you have added a Dockerfile, you can use the Docker client to build an image from the project. On Windows, you can use the Docker client for Windows and either a local Linux Docker installation or a remote Docker server. It can also be done from Linux as part of a CI pipeline.
By default, this will use the public Docker Hub images for Apama:
docker build -t apama_app ProjectDirectory
However, you can change this by using the build_arg parameter
docker build -t apama_app --build-arg APAMA_VERSION=latest ProjectDirectory
The advantages of this process are that steps such as the build, the tests and deployment will all cause the process to fail if they fail. This can be used to ensure that the final image is fully working before it is generated. Additionally, this method of multi-stage building is efficient because it will discard the intermediate layers which are not included in the final image.
Apama project
Regardless of Windows or Linux operating system, and whether or not Designer is used to create the project, you can automate the apama-builder image in a GitHub project repository and integrate it with TravisCI using the following example as a guide: GitHub - CaribouJohn/ApamaBuilderExample: Simple build sample, test sample, build image example
There is no need to install Software AG Apama locally on your machine, and you do not need a Java JDK (or JRE) installation locally either, as these are only needed inside the container where the Apama correlator will run the sample, and the Docker image already includes them.
Follow the instructions in the README.md in the GitHub repository, which cover all the steps required to build, deploy and run the sample in the container. They are more or less repeated here for convenience:
- From the command line, navigate to a chosen working directory and
git clone(orsvn checkout) the URL which can be copied from the “Clone or Download” control on the GitHub URL above.
git clone https://github.com/CaribouJohn/ApamaBuilderExample.git
Or:
svn co https://github.com/CaribouJohn/ApamaBuilderExample.git
They both have a similar effect. The svn checkout command creates an ApamaBuilderExample.git folder under which is a trunk folder, while git clone creates an ApamaBuilderExample folder which contains the same files as the Subversion trunk folder. Navigate into that, and build (and test) the image with:
docker build -t builder-test:latest .
- Run the image, picking a number for a port, in this case 40000, for the container which you can then access from another command prompt, and another, 15903, for the correlator inside it:
docker run -it --rm -p 40000:15903 builder-test:latest
When the container runs, the complex.yaml file in the project causes the correlator to be initialized with the complex_plugin.jar built and copied into the APAMA_WORK directory, and the copied ComplexPlugin.mon (see the Dockerfile).
- In another prompt, use
docker ps
to confirm the presence of the running container, and discover its name and container identifier. Doing
docker logs <container id>
will show the same output as on the screen in the first prompt, and is comparable to ComplexPluginSample.txt reference. The single PySys test is pretty rudimentary and only checks that after the injection, there are no ERROR or FATAL loggings. It doesn’t compare the output from the plug-in with the text file reference, for example.
- Finally, stop the container cleanly with
docker stop <container id>
Linking to Travis for continuous integration.
Linking the project to Travis ( https://docs.travis-ci.com/user/tutorial/ ) means that whenever changes are made to the project in GitHub, the whole build, test and deploy process is triggered automatically. Failure at any stage will cause the ci process to fail which can then trigger actions to identify what commit caused the failure. The process of adding Travis is as easy as registering with the site, linking your Github repository and adding a configuration file to the project at the top level.
services:
- docker
script:
- docker build -t apama_builder_example .
before_install:
- echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
The example above triggers Travis to use docker to build the project using the supplied script and adds a specific command to do a docker login so that the images can be pulled for the build. The variables named in the before_install line are defined on the Travis website and are not shown in logs or output from the build. See Use Docker in Builds - Travis CI for more details.
If you want a quick start for your own project you can fork the example repository into your own GitHub account. This appears on the surface to take Travis configuration with it, but it requires further setup. Create a Travis account synchronized with your GitHub account, and activate Travis for the forked project.
Note that the grey/green link to Travis from the GitHub project page is linked to the original project’s Travis setup, and is not instantly useful in the forked repository. Edit the URL behind it (accessing Readme.txt via the GitHub UI and the edit button may be the most convenient way) to be the URL for your Travis CI build for the project, and commit. This edit is likely to be to the two instances of username and project name in the line at the top of the file, which starting with the encoding “[![Build Status]”.
Go the the project in Travis, and under “More Options” select Settings and add environment variables called DOCKER_PASSWORD and DOCKER_USERNAME (see .travis.yml) and assign values to them (with “Display value in build log” turned off). The Docker username and password just need to be for any Docker account (create one if necessary) as this account will be sure to have access to the (public) Software AG Apama 10.3 Docker images, including apama-builder.
Clicking the “Restart build” button for the project in Travis will now build and test in the copied repository, and the build status link should now be for your project (if you edited it correctly) and be grey/green build/passing. Follow the link to the Travis log to confirm things (URL, date/time when the build and test ran, etc). Another way to trigger a Travis build is to commit a change to one or more files in the repository.
Working on the copy in the local file system, edit the EPL and Java plug-in code, add further modules, event files, extra plug-ins, configuration etc. as desired. Make sure the project basically does what you want when it runs. Also edit and add to the PySys tests, and debug those until they all run too. Submit the changes as an update to the new repository on GitHub, and see if the subsequent Travis build succeeds and passes.
Debug as required, commit more changes, add more features, and let the spec creep and code bloat begin!
Additional tips
Consult Travis, Docker, Apama and Git documentation on other forums for full guidance.
Useful commands
To check which version of Java the container is running, run
docker ps
as per instructions above to confirm that the container is running, and note its randomly assigned name, under the NAMES column, and do
docker exec <random_name> java -version
For 10.3, it will show you OpenJDK version “1.8.0_181”.
You can also do
docker exec <random_name> correlator -V
to reveal the full version number of Apama which is installed in the container.
If you do happen to have a local Software AG installation, and source apama_env (in the Apama/bin directory), then while the container is running (and outputting status lines) you can use a separate connection to
cd <working folder>/ApamaBuilderExample.git/trunk
engine_inject -v ComplexPlugin.mon -p 40000
to inject the EPL again and see the same output again. Use
docker logs <container id>
to view all the output. To inject different EPL files to the same correlator in the container, though, even ones existing in the local Apama/samples/correlator_plugin/java folder, you would first need to include them in the image build.
Finally, instead of
docker stop <container id>
or
docker stop <random_name>
doing
engine_management -p 40000 -s bye!
will stop the correlator within the container, and since the correlator is running as PID 1 in the container, the container will also stop.