Database Synchronization

The Ioto agent database (DB) supports optional transparent synchronization of data to and from the cloud.

Ioto takes the pain out of exporting device data into the cloud by automatically and transparently synchronizing data between your devices and a cloud device database.

Ioto will synchronize data from the device cloud database and the embedded database in your devices. This is similar to AWS Global Tables, but instead of between AWS regions, it is between an AWS region and a device.

The database synchronization is full-duplex in that data can be modified in the device or in the cloud and it will be replicated to the other side. The synchronization is controllable on a per-table basis.

The Ioto database is designed to be compatible with AWS DynamoDB so that it can easily replicate structured device data to the cloud. It is modeled after AWS DynamoDB and the OneTable access library.

Database Replication Features

Transparent replication.
Bi-directional synchronization to and from the cloud.
Per-table synchronization control.
Resilient catch-up should the device or cloud go offline.
Database triggers for notification when changes arrive.

Databases

The Ioto service creates a DynamoDB table called ioto in each device cloud. A device cloud is created by the Embedthis Builder for your account to host your device data in an AWS account and region of your choosing. The ioto database table contains the device data for all connected devices in that cloud.

The Ioto device agent creates a local database to hold structured data for the device.

These two databases are then connected and changes are replicated between device and cloud to keep them in sync.

Why Synchronization

The Ioto Database Synchronization service dramatically eases the task of centralizing the data management of a pool of devices. Each device can store data locally and it will be transparently, efficiently and automatically replicated to the cloud. Once in the cloud, it can be easily interrogated and queried for analytics, operational control and intelligence.

Each side has fast, predictable local access to the data. Coupled with reliable replication semantics you can work locally while Ioto takes care of the replication.

How It Works

To replicate cloud-side data, the Ioto service uses DynamoDB streams to capture modifications made to the table data. The stream is connected to an Ioto service Lambda which posts MQTT messages to a special topic that is only known to the device.

The Ioto device agent subscribes to this topic and receives MQTT messages containing the changed data. The agent then applies those changes to the local database.

In reverse, the Ioto agent registers a database trigger that is called whenever local changes are made to the device's database. When the trigger is invoked, the agent sends an MQTT message with the changes to the Ioto service that updates the cloud database.

When the device is rebooted, it sends a "sync" message to the Ioto service to retrieve all changes made to the cloud database since the last change notification was received from the cloud. In this way, changes made while the device is offline are not lost.

Controlling Replication

Ioto can synchronize changes to the cloud on a per model basis. The schema process property defines how to synchronize each model. You can "enable" a model to be stored in the cloud, in the device or in both places. You can also control the replication direction flow of changes.

For each Model in the schema, a property under process specifies where the model resides and how it should be synchronized.

For example:

{
    process: {
        Status:  { enable: 'cloud' },
        Fault:   { sync: 'up' },
    },
}

In this example: The Status model is only enabled (exists) in the cloud. The Fault model exists in both the cloud and on the device and synchronization occurs only from the device "up" to the cloud.

A model's enable property can be set to "cloud", "device" or "both" to define where the model can exist. The default is "both".

A model's sync property can be set to "up", "down", "both" or "none" to define the directional flow of replication updates. A value of "up" means replicate changes from the device "up" to the cloud. A value of "down" means replicate changes only from the cloud "down" to the device. The default is "none".

Replication Best Practices

Synchronization of model items is done on a per-item basis and not field by field. This means that a change of one field by the cloud can overwrite a change of another field on the device. For this reason, it is best to have the sync direction be either "up" or "down".

You should only select "both" as your sync direction for models that can be updated by both cloud and device at the same time without loss of data or conflict.

Optimizations

To optimize network traffic, database changes will be sent up to the cloud in batches. These are controlled by the database.syncDelay and the database.syncLimit properties. The syncDelay is a delay before sending change batches to the cloud. The syncLimit is a size limit of the change set before sending a change batch. These are set to 5 seconds and 10K by default.

You can force a sync save at any point by calling ioSync.

References

OneTable Schema Spec.