How to set up directories when offloading from Datahub to S3

Forum
,
1

Hi all,

I have successfully offloaded data from Data Hub to S3.
In this case, the directory structure is as follows.

s3_bucket
        ├── alarms
        ├── events
        └── measurements

However, I would like to separate each data for each device as follows.

s3_bucket
       ├── device_A
       │   ├── device_A's alarms
       │   ├── device_A's events
       │   └── device_A's measurements
       └── device_B
           ├── device_B's alarms
           ├── device_B's events
           └── device_B's measurements

Is it possible to dynamically change the offload destination for each device in this way?

2

Hi there,

I understand where you are heading at, but I don’t believe that this is possible right now. Please keep also in mind that you are paying AWS S3 per 1,000 requests. So if you have e.g. 1,000,000 devices x 3 data types x 24 hours x 30 days x some rewrites for optimization / 1,000 * $0.0054, that would be a whopping $23,000 per month in the best case.

Cheers,
André

2 Likes
3

Thank you for your answer.
I understand that this is not realistic in terms of money, but I would like to know if it is technically possible.

For example, when it consists of a small group of devices,
Is it possible to specify a bucket path when offloading to S3?

4

Not that I would know. The name of the offloading job defines the folder, and you could partition devices into different folders through that. That can be used if you have device types that you want to treat totally differently from each other. It does not exactly give you the hierarchy in your example and increases the number of offloaders, which you can only have a limited number of.

Cheers,
André

2 Likes
5

I will try different offload job names.
Thank you for your answer.

6

Thanks to you, I was able to specify the name of the folder.Please let me know if you know.
Is it possible to offload data after combining different data?
For example, I want to combine the data of a managedObject with an event.
ManagedObject data is not included when POST event data.

{
	"source": {
    	"id":"123456" },
    "type": "TestEvent",
    "text": "sensor was triggered",
    "c8y_yamashita": "custom fragment test", // Custom fragments of the event itself
    "time": "2025-06-18T13:03:27.845Z",
    "c8y_addDeviceInfo1": "device info", // add from MO when offloading
    "c8y_addDeviceInfo2": "device info", // add from MO when offloading
}

I also tried “Set additional result columns” which can be set when offloading, but it only adds its own fragments, etc.

src.c8y_yamashita

Sorry, perhaps I should have created a another topic.