In Apama 10.2 we have upgraded the versions of Docker that we use. We have moved to Docker v17.09.0-ce, Docker Compose v1.16.1 and Compose API 3.3. In addition we have also introduced support for using Kubernetes. Our Docker samples now include Kubernetes orchestration files that are the equivalent of the the YAML files supplied for Docker Compose.
Assumed Knowledge
- How to start a correlator, including using a YAML configuration.
- Using Docker containers.
- Basic knowledge of docker-compose.
- Basic knowledge of Kubernetes.
You will also need Docker and Kubernetes servers installed, or access to a suitable cloud provider with Docker and Kubernetes support.
For all these examples we will use the Apama Docker container available from Docker Hub.
Constructing a ‘Hello World’ Example
This simple example will deal only with getting a basic correlator example up and running and proving itself by outputting Hello World to the log. The example is in many ways redundant as you can do this from the command line using docker run . However, it does allow the introduction of some of the extra steps required by using Kubernetes.
EPL
The first thing we require is the simple EPL file that the correlator will run to prove that it has started correctly and can load EPL. To do this we will create a file called HelloWorld.mon containing the following simple monitor:
monitor HelloWorld
{
action onload()
{
on all wait(1.0)
{
log "Hello world!" at INFO;
}
}
}
The monitor simply logs a Hello world! message every second.
Correlator configuration
We will also need a correlator configuration YAML file that will load this EPL into the correlator when it is started, which we’ll call init.yaml:
correlator:
initialization:
list:
- ${PARENT_DIR}/HelloWorld.mon
Dockerfile – copy files
For creating the container which contains this simple application we need a Dockerfile which defines how we build the container, in this case located in the same directory as the two files we are copying into the image. For this example we’ll base it on the Apama image in Docker Hub.
FROM softwareag/apama-correlator:10.2
COPY init.yaml HelloWorld.mon /apama_work/HelloWorld/
WORKDIR /apama_work/HelloWorld
CMD ["correlator", "--config", "init.yaml"]
Line 1 makes the apama-correlator image in Docker Hub our starting point.
Line 2 copies the two files we created above into this image into the /apama_work/HelloWorld folder.
Line 3 defines that same folder as the working directory.
Line 4 gives the default command to be run for the container, namely the correlator using the init.yaml configuration file.
Dockerfile – deploy
Alternatively we can use the deployment functionality provided by Apama, which allows us to take a SAG Designer project and deploy it for use. The end result will be the same.
engine_deploy --outputDeployDir HelloWorld_deploy <HelloWorld Designer project>/config/launch/HelloWorld.deploy
The deployment file is generated by the SAG Designer when the project is built.
FROM softwareag/apama-correlator:10.2
COPY HelloWorld_deploy /apama_work/HelloWorld
WORKDIR /apama_work/HelloWorld
CMD ["correlator", "--config", "/apama_work/HelloWorld"]
Line 2 has been altered to copy in the deployed Designer project.
Line 4 tells the –config to use all the YAML files that it finds within the path provided, instead of explicitly providing a YAML file. The YAML files are generated for the project when it is built within SAG Designer prior to deployment.
Docker build & Docker run
Given the simplicity of this example, we don’t actually need docker-compose or Kubernetes at all. We can prove this and check what we have so far just by using the docker command itself:
docker build -t helloworld .
docker run -it --rm helloworld
You will see “Hello world!” logged every second from from the displayed log. Ctrl-C will stop the running container.
With docker-compose
To run this example with docker-compose we need a compose YAML file, which we’ll call docker-compose.yml. This will build the image if necessary, using the Dockerfile above, and then run it.
version: '3.3'
services:
# Single application container
application:
build: .
To kick off the HelloWorld app this way use
docker-compose up
Run as above you will see the logs output to the terminal. However, you can also access the logs at will by using the container name or container id. The container id can be found by using:
docker ps | grep helloworld
Thus to get log use one of the following two commands:
docker-compose logs <container id>
docker-compose logs helloworld_1
in a directory containing both the yaml file and the DockerFile. To shut down the container use
docker-compose down
With Kubernetes
Kubernetes is a little more complicated, and we must build the container ourselves, as we did above.
docker build -t helloworld .
Then we must add the image to your Docker registry so that Kubernetes can find it:
docker tag helloworld <your registry>/helloworld
docker push <your registry>/helloworld
Now we can use Kubernetes. Like docker-compose, Kubernetes also uses a YAML file, although it is not quite as succinct. We will name this kubernetes.yml and place it in the same folder as our other files.
apiVersion: v1
kind: Pod
metadata:
name: helloworld
spec:
containers:
- name: helloworld
image: <your registry>/helloworld
ports:
- containerPort: 15903
For this simple example we only need to define a pod. The image is the one we pushed in the registry. Finally we have defined the container port to be the default port for a correlator. To run this example via Kubernetes:
kubectl create -f kubernetes.yml
To view the logs from this pod:
kubectl logs helloworld
And finally to stop and clear the container:
kubectl delete -f kubernetes.yml
A more realistic example
For this we will examine the Adapter sample supplied with Apama in more detail to examine how it helps orchestrate the running Apama with a separate iaf process for the File Adapter.
This sample can be found on github at apama-streaming-analytics-docker-samples/applications/Adapter at master · Cumulocity-IoT/apama-streaming-analytics-docker-samples · GitHub or in the /Apama/samples/docker/applications/Adapter folder. The sample contains two directories, which contain the extra files that need to be copied into each Docker container, and the Dockerfile to build those containers. The top level has both a docker-compose and Kubernetes orchestration file for running this sample.
Dockerfiles
These are essentially the same as in the simple example, defining a baseline image, copying the extra files required, setting a working directory and defining a default process to be run. There is one difference which is adding the ARG line and using that within the FROM line:
ARG APAMA_IMAGE=apama
FROM ${APAMA_IMAGE}
This means that we are no longer restricted to the base image defined in the Dockerfile, but can replace it with one of our own, using the –build-arg argument for docker build. The default base image is the apama image generated in your local Docker repository when you follow the instructions in the top level sample readme. If we want to use the Docker Hub image, as we did in the simple example, then we don’t need to edit the Dockerfile. Instead we can say:
docker build . --build-arg APAMA_IMAGE=softwareag/apama-correlator:10.2
Unlike the first example, this is however not trivial to run. You would have to build and run both containers separately ensuring that they connect to each other. In this example orchestration becomes useful.
With docker-compose
From the point of view of the end user, this example is identical to the simple example. The user simply uses docker-compose up.
For the developer the docker-compose yml file is a little more complicated:
version: '3.3'
services:
# Runs the Apama correlator with an application that talks to the IAF and issues commands to the File adapter
correlator:
# Build the image for this container from the Dockerfile in ./deployment/, which puts the EPL for this application on top of a base
# Apama image
build:
context: ./deployment
args:
APAMA_IMAGE: softwareag/apama-correlator:10.2
# Starts the IAF running the File adapter, which connects to the contained correlator. The IAF configuration is built into the image, using
# a Dockerfile that builds on top of the base Apama image.
iaf:
# Build the image for this container from the Dockerfile in ./iaf/, which puts IAF configuration on top of a base Apama image
build:
context: ./iaf
args:
APAMA_IMAGE: softwareag/apama-correlator:10.2
depends_on:
- correlator
In concept, it is two of the simple example in the same file. In this case, naming the containers being worked with becomes more important, so we have correlator and iaf. Note the alteration to the build tags from the file file provided in the sample. This allows us to invoke the –build-arg when docker-compose uses the Dockerfiles and redirect the image Dockerfiles towards the Apama image within Docker Hub.
The iaf additionally specifies the depends_on flag. This ensures that the correlator container is always started before the iaf container, making sure that we can control the order in which services are started. It also means that should you ever say docker-compose up iaf, to just start the iaf, then the correlator service will also automatically be started as the iaf service requires it to start successfully.
The final aspect to consider here is that the correlator and iaf containers need to be able to talk to each other. For a relatively simple example such as this, we don’t need to explicitly do anything. docker-compose will automatically create a default network and place all the containers on it, unless otherwise specified. In this case the default network is called adapter-default and can be seen using docker network ls. Further information on the network can be inspected using docker inspect adapter-default.
When you shutdown this sample using docker-compose down, you will see that not only are the containers removed, but the network is also removed:
Removing adapter_iaf_1 ... done
Removing adapter_correlator_1 ... done
Removing network adapter_default
With Kubernetes
Again Kubernetes is more complicated. We have to do the initial docker builds ourselves:
docker build -t adapter_correlator ./deployment --build-arg APAMA_IMAGE=softwareag/apama-correlator:10.2
docker build -t adapter_iaf ./iaf --build-arg APAMA_IMAGE=softwareag/apama-correlator:10.2
Alternatively, to make this build stage easier, you can also do:
docker-compose build
We still need to manually add these images to our registry.
docker tag adapter_correlator <your registry>/adapter_correlator:latest
docker tag adapter_iaf <your registry>/adapter_iaf:latest
docker push <your registry>/adapter_correlator:latest
docker push <your registry>/adapter_iaf:latest
Now we are again ready for our kubernetes.yml YAML file.
apiVersion: v1
kind: Pod
metadata:
name: adapter-engine
labels:
app: "adapter-correlator"
spec:
containers:
- name: adapter-correlator
image: <your registry>/adapter_correlator:latest
ports:
- containerPort: 15903
readinessProbe:
httpGet:
path: /ping
port: 15903
---
apiVersion: v1
kind: Service
metadata:
name: adapter-correlator
spec:
selector:
app: adapter-correlator
ports:
- protocol: TCP
port: 15903
targetPort: 15903
---
apiVersion: v1
kind: Pod
metadata:
name: adapter-iaf
spec:
initContainers:
- name: wait-for-correlator
image: busybox
command: ['sh', '-c', 'until nslookup adapter-correlator; do echo waiting for correlator; sleep 2; done;']
containers:
- name: adapter-iaf
image: <your registry>/adapter_iaf:latest
ports:
- containerPort: 16903
Similarly to the HelloWorld example, we have a pod for each container. These need to be able to communicate, and for this we define a service to allow the iaf access to the correlator. The selector specifies which pods the service applies to, and matches the label we specified for the correlator’s pod. We then also specify that the default correlator port of 15903 is exposed, also as port 15903. This could be mapped to another port if required. Unlike with docker-compose, there is no way of specifying dependencies between pods or a startup order.
To ensure the dependency that correlator has started prior to the iaf, the iaf’s pod has an initContainer specified. This is a container which must run to completion prior to the application container starting. In this case we use a bare-bones image supplied by Docker to check for the existence of the adapter-correlator host using nslookup. This host is provided by the adapter-correlator service. We want to ensure that this service only becomes available when the pod it relies on is available. To achieve this, we specify a readiness probe for the correlator, which in this case simply ensures that port 15903 is accessible, indicating the correlator has started successfully.
With this in place the user again has the same experience as with the HelloWorld example, using:
kubectl create -f kubernetes.yml
Full Documentation
Full documentation on using Docker and Kubernetes can be found in the online documentation for Apama 10.2 under Deploying Apama Applications with Docker
Feel free to post on our forums with any questions you may have about this, or any other, topic.
– Nicholas