Introduction
A historian database is a system designed to store, organize, and retrieve historical or time-stamped data, widely used across industries to manage and analyze essential manufacturing process data. It provides secure storage and fast data retrieval, but it is usually limited to local access and applications.
The combination of Historian and Cumulocity IoT makes data remotely accessible and available to more data tools, maximizing the potential of Historian data.
In this article we will explain step by step how to leverage thin-edge.io to securely integrate data stored in an AVEVA PI System with Cumulocity’s IoT platform.
The AVEVA PI System, previously called OSIsoft PI System, is an industrial software platform designed to collect, store, and analyze operational data in real-time. At the core of the PI System is its historian database. For this integration, we used the PI Web API module that provides a REST API giving client applications read access to the PI data over HTTPS.
Thin-edge.io is an open-source, lightweight edge framework designed for connecting IoT devices to cloud platforms and enabling edge computing. It can be deployed on an IoT gateway but also Industrial PCs and provides a secure, modular, and extensible solution for processing and managing data at the edge. Thin-edge.io supports device management, and integration with popular cloud services, making it ideal for industrial IoT applications.
This post is based on the results of a joint project conducted with Solenis. Thanks to Solenis for their support and providing access to a production PI system.
Pre-requisite
- A Cumulocity IoT tenant
- PI System with PI Web API installed
- A PI System user with at least READ permission
- A (virtual) machine with access to the PI System and Internet connection
Preview of the architecture
The setup consists of a data flow implemented by Node-RED between the PI System and the Cumulocity IoT platform. An edge device (a virtual machine) equipped with Thin-edge.io which can securely communicate with the platform via MQTT using TLS (port 8883) and HTTPS (port 443). The cloud Remote access feature of the Cumulocity IoT platform enables an efficient and secure remote maintenance. The next section will show you how to set up this architecture step-by-step.
Steps to follow
1. Install and register Thin-edge.io to Cumulocity IoT
Thin-edge.io is an open-source IoT framework designed for lightweight IoT devices. It can make the cloud connection with the edge devices easier and more secure. And it brings out-of-the-box configuration management and remote access capabilities, both of which will be useful for historian integration.
Follow this documentation to connect your first thin-edge device to the Cumulocity IoT platform:
With thin-edge.io, no direct internet access to the local Historian server needs to be exposed. Only whitelisting for one outgoing connection to IoT platform is needed. Users can take the advantages of all thin-edge.io features to manage the configuration files and software remotely and even get the remote access to the thin-edge device securely.
2. Install Node-RED UI plugin on the Cumulocity IoT Platform
Node-RED provides a browser-based flow editor to visualize the data migration process. It comes with HTTP and MQTT nodes that allow data to be retrieved from the PI system and forwarded to the Thin-edge. Node-RED should be installed inside the same local network where thin-edge.io and PI System are also running.
Remote access cloud HTTP proxy is an open-source Microservice based on Cumulocity IoT’s remote access feature. It allows you to remotely access the locally installed Node-RED UI on the platform. Please follow the readme file of this repository to install this Microservice:
Node-RED has been used here for demonstration purpose. The same logic can be achieved by an application written in any other programming languages. Such application can also be managed on the platform.
3. Prepare data point list and configuration file
When extracting data from the PI system, if you need to obtain specific data points, you need to provide the IDs or other identity information of the data points in the request to filter out the desired data points and their historian. By saving the identity of the data points in a separate file, it is possible to change the specified acquired data points without changing the flow. This also applies to other settings in the flow, such as the frequency of data transfers, links to the PI system, and so on.
Here, as an example, the names and descriptions of the data points are stored in a json file. Other file formats are also possible, as long as they have the matching read logic.
These configuration files can be managed using configuration management, another feature of Cumulocity IoT Device Management application. The various configuration files can be saved in the configuration repository on the platform and can be used to remotely change the configuration of the node-RED flow.
4. Get data from the PI System
Once the datapoint name is known, you can use nameFilter to get the RecordedData link of each datapoint one by one. If the link is already known, this step can be skipped.
Send a GET request to this endpoint with the specified start time to retrieve the historian data stored in the PI System for a certain time period. The time range can also be configured by the configuration file remotely.
5. Publish data to thin-edge
The next step is to send the data to the thin-edge. The data should be in a format supported by the thin-edge and then published to the MQTT broker. Sending data individually or sending it in bulk is slightly different in format and topic.
-
Single measurement
Single measurements can be sent to the cloud by publishing message to thete/+/+/+/+/m/+
MQTT topic. Thin-edge’s Cumulocity mapper will verify and then forward the message to the cloud platform. The orignal timestamp of the historian data needs to be sent along with each message.
For example:
Topic:te/device/main///m/<measurement_type>
{ "time": "<timestamp>", "<measurement_name>": { "<measurement_series>": { "value": 10 } } }
-
Bulk measurements
Measurements can also be sent in bulk using Cumulocity’s JSON topics. Timestamp and type of the measurement are required.
For example:
Topicc8y/measurement/measurements/createBulk
{"measurements":[ {"<measurement_name>": { "<measurement_series>": {"value": 10} }, "type": "< measurement_type >", "time": "<timestamp>"}, ...... ]}
Outlook
Integrating the PI System Historian with Cumulocity IoT opens up great opportunities for IoT applications. Once the data from the Historian is securely transferred to the IoT platform, it becomes a foundation for advanced analytics, operational optimization and even more. This will unlock the power of Historian Data to the users on the cloud platform.