Skip to content

Database Schemas

Ioto uses database schemas to define how items will be stored in the database.

A schema specifies the database item types, entity fields, indexes and other data access and replication parameters.

Ioto uses the DynamoDB OneTable schema format, implementing the OneTable Schema Spec.

The same schema is used for the cloud side AWS DynamoDB table and for the device side embedded database.

Schema Definition

Schemas look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
const DeviceSchema = {
    format: 'onetable:1.1.0',
    version: '0.0.1',
    indexes: {
        primary: {hash: 'pk', sort: 'sk'},
    },
    process: {
        /*
            Where the item is stored (cloud, device, both. Default is both)
            Synchronization direction. Up to the cloud, down or both. Default is none.
        */
        Status: {enable: 'cloud'},
        Fault:   { 
            sync: 'up' 
            metrics: [{
                namespace: 'Embedthis/Device',
                fields: ['temperature'],
                dimensions: [{Device: 'deviceId'}]
            }]
        },
    },
    models: {
        Status: {
            pk: {type: 'string', value: 'status#'},
            sk: {type: 'string', value: 'status#'},
            parameters: {type: 'object'},
            version: {type: 'string'},
            updated: {type: 'date'},
        },
        Fault: {
            pk: {type: 'string', value: 'device#${deviceId}'},
            sk: {type: 'string', value: 'fault#${id}'},
            deviceId: {type: 'string', required: true},
            id: {type: 'string', generate: 'ulid'},
            timestamp: {type: 'date', required: true},
            source: {type: 'string', required: true},
            severity: {
                type: 'string',
                required: true,
                enum: ['info', 'warn', 'error', 'critical', 'fatal'],
            },
            subject: {type: 'string', required: true},
            message: {type: 'string', required: true},
        },
    },
    params: {
        isoDates: true,
        timestamps: true,
    },
}

Schema Purpose

AWS DynamoDB can operate most efficiently at scale when using a single table with multiple application entities stored in the one table. To make this design pattern easy to implement, application data schemas such as the one used by DynamoDB OneTable provide the mapping between application level entities and raw database items.

Ioto uses the OneTable schema in both the DynamoDB table and in the Ioto agent database.

When coupled with a high-level database API such as that offered by OneTable and by the Ioto Agent Database, accessing and managing device data is much easier.

DynamoDB Schema

Ioto uses the DynamoDB OneTable NodeJS library for all database access to the Ioto DynamoDB database.

Please consult the OneTable library and OneTable Documentation for full details.

Updating the Schema

Updating the Schema

Schema Properties

The valid properties of the schema object are:

Property Type Description
format string Reserved. Must be set to onetable:1.1.0
indexes object Hash of indexes used by the table.
models object Hash of model entities describing the model keys, indexes and attributes.
params object Hash of properties controlling how data is stored in the table.
process object Hash of model entities describing where the model should be and how it should be replicated.
version string A SemVer compatible version string.

The format property specifies the schema format version and must be set to onetable:1.1.0.

The indexes property specifies the key structure for the primary, local and secondary indexes.

The models property contains one or more item definitions with attribute field descriptions. The models collections define the attribute names, types, mappings, validations and other properties.

The params property defines additional parameters for table data formats.

The process map defines how database items should be processed, stored, replicated and whether metrics should be calculated from the data stream.

The version property defines a version for your database. It must be a SemVer compatible version string. The version string is used by tools and consumers of the schema to understand compatibility constraints for the schema and stored data.

Schema Models

The schema defines a model for each item type. For example, consider a music example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
    album: {
        pk:     { type: 'string', value: '${_type}:${name}' },
        sk:     { type: 'string', value: '${_type}:' },
        name:   { type: 'string', required: true },
        songs:  { type: 'number' },
    },
    artist: {
        pk:     { type: 'string', value: '${_type}:${name}' },
        sk:     { type: 'string', value: '${_type}:' },
        name:   { type: 'string', required: true },
        address: {
            type: Object, schema: {
                street: { type: 'string' },
                city: { type: 'string' },
                zip: { type: 'string' },
            },
        },
    }
}

For each item type, all the entity fields are defined by specifying the field type, validations and other operational characteristics (uniqueness, IDs and templates).

The valid types are: array, binary, boolean, date, number, object, and string.

The database will automatically add an item type via the _type attribute to each model. This is set to the name of the type. You can modify this via the params.typeField setting.

Value Templates

The value of a field can be computed based on the value of other fields and a formatting "value template". This is useful for decoupling your key structure from your entity fields.

The value template is like a Javascript string literal where the value of fields are expressed in a "${field}" format. At runtime, the field references are expanded with the item's value for the named field.

In the example above, the primary key "pk" is calculated from the entity type _type and the name field by using the value template: ${_type}:${name}.

Nested Schemas

For object attributes, you can define a nested schema for the object properties, as in the example above (repeated below).

A nested schema uses the schema property to define a nested map of attributes. Schemas can be nested to an arbitrary depth.

1
2
3
4
5
6
7
address: {
    type: Object, schema: {
        street: { type: 'string' },
        city: { type: 'string' },
        zip: { type: 'string' }
    }
}

References