Skip to content

Device Table Synchronization

Ioto Device Tables takes the pain out of synchronizing device data into the cloud. It automatically and transparently synchronizes data between your devices and the cloud.

The database synchronization is full-duplex in that data can be modified in the device or in the cloud and it will be replicated to the other side. The synchronization is controllable on a per-table basis.

Databases

The Ioto service creates an AWS DynamoDB database table called ioto in each device cloud. The ioto database table contains the device data and cloud management data for all connected devices in that cloud.

Similarly, the Ioto device agent creates a local embedded database to hold structured data for the device.

These two databases are connected and Ioto replicates changes between device and cloud to keep them in sync.

Why Synchronization?

The Ioto Database Synchronization service dramatically eases the task of centralizing the data management of a pool of devices. Each device can store data locally and it will be transparently, efficiently and automatically replicated to the cloud. Once in the cloud, it can be easily interrogated and queried for analytics, operational control and intelligence.

Each side has fast, predictable local access to the data. Coupled with reliable replication semantics you can work locally while Ioto takes care of the replication.

How It Works

To replicate cloud-side data, the Ioto service uses AWS DynamoDB streams to capture modifications made to the cloud-side table data. The stream is connected to a Lambda function that encodes and sends the updates via MQTT messages to a special message topic that is only known to the specific device.

The Ioto device agent subscribes to this topic and receives MQTT messages containing the changed data. The agent then applies those changes to the local database.

In reverse, the Ioto agent registers a database trigger that is called whenever local changes are made to the device's database. When the trigger is invoked, the agent sends an MQTT message with the changes to the Ioto service that updates the cloud database.

Ioto keeps track of the last update sent and received and will retransmit lost updates.

When a device is rebooted after being offline, it sends a "sync" message to the Ioto service to retrieve all changes made to the cloud database since the last change notification was received from the cloud. In this way, changes made while the device is offline or disconnected are not lost.

Controlling Replication

The cloud-side Ioto DynamoDB table (ioto) and the Ioto agent database both use a OneTable schema that defines application data entities, how items are stored, what indexes are used and how to process table data. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
const DeviceSchema = {
    process: {
        /*
            Where the item is stored (cloud, device, both. Default is both)
            Synchronization direction. Up to the cloud, down or both. Default is none.
        */
        Status: {enable: 'cloud'},
        Fault: {sync: 'up'},
    },
    models: {},
}

The "process" property of the schema defines how to process table data and how tables are synchronized between the device and cloud.

For each database item type in the schema, you can create an entry in the "process" property that specifies where items of that type should be stored and how they should be synchronized.

The "enable" property can be set to "cloud" if items of that type should exist only in the cloud database. Set the property to "device" if the items should exist only in the device. Set to "both" if they should exist in both databases. The default is "both".

The "sync" property defines the synchronization direction. Set to "up" to indicate the device data should be synchronized from the device up to the cloud. Set to "down" to indicate the cloud data should be replicated down to the device and set to "both" to replicate in both directions.

To design for effective synchronization, it is best to have the "sync" direction be either "up" or "down" and not "both". You should only select "both" for item types that are not transactionally critical, as changes from the cloud or device may overwrite changes coming from the other directions.

Synchronization is done on a per-item basis and not field by field. This means that if you update a field in an item, the entire item will be updated on the peer side.

Requirements

Any model that defines a sync direction of "up" or "down" must include the following properties in the model definition:

Property Definition Description
updated {type: 'date', readonly: true} Timestamp for when the item was last updated
_source {type: 'string', hidden: true} The source of the last update

The updated timestamp property is used to determine the set of updates that have been made to the database since the device last connected to the device cloud. The _source property is used to determine the direction of synchronization for an item.

The bin/make-schema script that is used when building the Ioto agent and apps will automatically add these properties for synchronized models.