Cumulocity IoT Streaming Analytics 10.18.0.5+ or 24+
Introduction
Cumulocity IoT Streaming Analytics allows users to create real-time analytics applications within the platform to process messages being generated by IoT devices. These applications can be developed within the platform directly using EPL apps, or externally using the Apama Streaming Analytics tooling. When developing within the platform, however, you lack many of the normal tools for developing software such as version control, testing, quality gates, promotion, and deployment. This blog post introduces a new tool that you can deploy to your Cumulocity IoT tenant which enables more enterprise development workflows: EPL Apps GitHub Sync.
https://github.com/mjj29/streaminganalytics-github-sync
EPL Apps GitHub Sync (referred to below as just GitHub Sync) is a tool to do bidirectional syncing between GitHub and Cumulocity IoT Streaming Analytics EPL apps.
Setting up GitHub Sync in your tenant
To use GitHub Sync you will need a Cumulocity IoT tenant with the ability to run EPL apps, running at least Streaming Analytics 10.18.0.5 or a 24 or later version from continuous deployment. The GitHub repository must exist and be initialized. It can be on github.com or a private GHES instance. Access is managed via the GitHub REST API and not direct Git access.
You will need to set several tenant options in your tenant. You can set tenant options using the REST API or the Cumulocity IoT Tenant Option Management plugin. For example, if you want to synchronize to the main branch of the repository at github.com/myorg/eplapps-repo you would set the following tenant options:
streaminganalytics.github/owner = myorgstreaminganalytics.github/repo = eplapps-repostreaminganalytics.github/branch = mainstreaminganalytics.github/PAT = <a personal access token with write access to that repository>
Optionally you can also set streaminganalytics.github/githubAPIURL to point to the API endpoint for a server other than github.com.
Lastly, just upload SyncToGithub.mon as a new EPL app in your tenant.
This will:
- Write all EPL apps that exist at that moment in the tenant to the
epl/directory of the GitHub repository (overwriting any earlier versions in the repository) - Deploy any already-existing apps under
epl/from the GitHub repository to the tenant as new EPL apps - Watch for updates to EPL apps going forward and write any changes / remove any deleted EPL apps from the GitHub repository
- Watch for new commits to the GitHub repository and make any changes to match in EPL apps
Suggested workflows
Here are a few ideas of enterprise development workflows which are enabled by GitHub Sync.
Backup and history for EPL apps development
The simplest option you have is to just continue EPL apps development in the web UI as normal and let it be synchronized to GitHub. This gives you a backup of your code automatically on each change in the GitHub repository. If you ever need to restore from the backup, simply add SyncToGithub.mon to a new blank tenant and all your EPL apps will automatically be restored. This also gives you a complete history of all your development changes via GitHub history.
Development to Production tenant promotion
A step up from just using GitHub as a backup, while still only using the Cumulocity IoT cloud development facilities, is to work with multiple tenants for different stages of development. It’s very common to have a tenant for development, another for quality assurance (QAL) and another for actual production. Having each of those tenants tracking a different branch in a repository allows you to promote code between the different stages using pull requests in GitHub. Development on the development tenant syncs to a development branch. Once it’s ready to test that on QAL then a pull request (PR) is used to bring the development version onto the branch for that tenant. Once the code change has been validated on QAL, then you can use another pull request to merge onto the production branch. At each stage, deployment is automatically managed by GitHub Sync, so you never have to have developers manually log into your production tenants and make changes to the code there. If they do, then any changes will be tracked in GitHub and won’t be overwritten by the next release from development, but instead merged by GitHub during the PR merge.
External development with automatic deployment
Some people are using the desktop Apama Streaming Analytics tooling to do local off-line development, and only deploying to the Cumulocity IoT cloud once the code is ready. With GitHub Sync you can do that development locally, committing the changes to GitHub and then rely on GitHub Sync to publish the code out of GitHub into the cloud automatically. As with the previous workflow it allows you to protect production from direct developer access and record any changes made directly to production.
Continuous deployment with automatic quality gates
Lastly, we can also include continuous quality with tests using EPL Apps Tools in PR checks to ensure continuous quality in a full CI/CD pipeline. GitHub Sync provides the final piece for automatic deployment once all the tests have passed and the code change has merged.
Conclusions and more information
As you can see, GitHub Sync provides support for these and other enterprise development workflows. For more information about the things mentioned in this post see:
- GitHub - mjj29/streaminganalytics-github-sync: Automatically synchronize assets from Cumulocity IoT Streaming Analytics with github
- GitHub - Cumulocity-IoT/apama-eplapps-tools: Tooling for uploading apps and testing apps within Apama EPL Apps
- GitHub - Cumulocity-IoT/cumulocity-tenant-option-management-plugin: Easily create, edit and delete tenant options. You can configure an option as encrypted and can decided between text or JSON input.
- https://www.apamacommunity.com/
- https://www.cumulocity.com/
This article is part of the TECHniques newsletter blog - technical tips and tricks for the Software AG community. Subscribe to receive our quarterly updates or read the latest issue.