Managing JSON Data with Ioto

cluster

Ioto incorporates a high-performance JSON parser, serializer, and query engine, which stores JSON data in memory-based trees for efficient processing, querying, storage, and exporting.

The JSON engine allows Ioto to swiftly and seamlessly handle HTTP and MQTT requests with JSON payloads, while delivering responses in JSON format. This helps create more efficient and secure manageable devices.

This article represents the eighth installment in a series focused on the Ioto embedded web server component, and specifically explores managing JSON data within Ioto.

The posts in the series are:

Background

Ioto extensively employs and capitalizes on JSON. This format is utilized for all Ioto configuration files and accommodated in HTTP and MQTT request and response data. Additionally, JSON serves as the storage format for data items within the Ioto embedded database.

By consistently adopting a singular high-performance data format and engine, Ioto minimizes its memory footprint through code reuse, while simultaneously maximizing security and performance.

Json Features

JSON/5

Ioto employs a human-readable extension of JSON known as JSON/5. JSON/5 enhances JSON, simplifying the creation, readability, and maintenance of configuration files in JSON format.

Ioto is compatible with both JSON and JSON/5.

JSON/5 introduces the following JavaScript features to JSON:

{
    // single-comment
    /*
        Multi-line comment
    */
    unquotedKey: 42,
    singleQuoteString: 'The "lazy brown fox" jumped...',
    multiLine: "Line one
        line two
    ",
    hex: 0x42,
    trailingComma: {
        one: 1,
        two: 2,
    },
}

Configuration Files

All Ioto configuration files use the JSON/5 format. The primary configuration files are:

Profiles

Configuration files can provide multiple property profiles and modes that can be selected at runtime. When Ioto is run, it executes with selected mode and profile. This is typically a profile of prod for production and dev for development, and a mode of local or cloud.

The Ioto mode is defined by the mode property and the profile by the profile property. These can be overridden by command line options.

The configuration modes and profiles are defined under the conditional property. The relevant configuration properties are selected by the current Ioto mode and profile.

For example:

conditional: {
    profile: {
        dev: {
            limits: {
                stack: '64k',
            }
        },
        prod: {
            log: {
                path: 'ioto.log',
            }
        }
    }
}

Here is a sample config/ioto.json5:

{
    version: '1.1.0',
    profile: 'dev',
    mode: 'local',
    services: {
        database: true,
        keys: true,
        logs: false,
        mqtt: true,
        provision: true,
        serialize: 'auto',
        shadow: false,
        sync: true,
        url: true,
        web: false,
    },
    logs: {
        files: [ {path: '/var/log/sys*log', group: 'ioto', stream: '{hostname}' } ],
    },
    limits: {
        stack: '16k',
    },
    log: {
        path: 'aws',
        format: '%D %H %A[%P] %T %S %M',
        types: 'error,info',
        sources: 'all',
    }
}

API Quick Tour

Ioto has a powerful and flexible JSON API for you to manage JSON data.

Here is a quick API tour:

To parse a JSON string, call the jsonParse API:

Json *json = jsonParse("{Weather: 'Sunny'}", 0);

This parses the given JSON text and returns a Json instance which represents the in-memory parsed JSON tree.

Data for parsing can be supplied in either JSON or JSON/5 format strings. JSON/5 allows for a more lenient JSON format, wherein property keys can exclude quotes and single quotes can be utilized for values. Additionally, values can be multi-line strings, and back-ticks may be used as an alternative to quotes for delimiting strings. Lastly, a trailing comma can be used to terminate the final item in an object map.

Getting Values

To query a value from the JSON tree, use jsonGet:

cchar *value = jsonGet(json, 0, "weather", NULL);

This will retrieve the weather value (Sunny) as a static string that does not need freeing.

When calling jsonGet, you can provide keys with dots:

cchar *value = jsonGet(json, 0, "address.city", NULL);

and you can provide a default value to be returned as the last parameter if the property is not defined in the JSON tree.

cchar *value = jsonGet(json, 0, "weather", "rainy");

The second argument (0) in these calls defines the starting point from which to search. This is a numeric node ID. Don’t worry about it for now, but you’ll appreciate it later when searching deeper in a JSON tree.

You can use the jsonGetInt and jsonGetBool APIs to return data as an integer or boolean data type.

Setting Values

To update or set a value, use the jsonSet API:

jsonSet(json, 0, "weather", "sunny", 0);

This will update the value of the given property in-memory. Again, you can use dotted key properties of any depth and the intermediate properties will be created as required.

Data Types

The last parameter to jsonSet can be used to specify a data type for the value. In general, the Ioto JSON API can sleuth the data type for you. But as data values are supplied as strings, you may need to specify a type if you want the value to be stored as a number.

The valid types are: JSON_OBJECT, JSON_ARRAY, JSON_STRING and JSON_PRIMITIVE. The primitives are false, true, null, undefined and numbers.

To remove a property, use jsonRemove:

jsonRemove(json, 0, "weather");

When you are finished with a JSON tree, remember to free it with jsonFree to prevent memory leaks.

jsonFree(json)

This will release allocated memory.

Serialization

To save your in-memory tree as a string, use jsonToString:

char *string = jsonToString(json, 0, 0, 0);

This will save the in-memory tree as JSON/5.

To save in strict JSON, use:

char *string = jsonToString(json, 0, 0, JSON_STRICT);

To save a sub-set of the tree, specify the root node to export:

char *string = jsonToString(json, 0, "address", JSON_STRICT);

This will save all properties under the “address” (including address) property.

Working with Files

You can parse a JSON file with the jsonParseFile API:

Json *json = jsonParseFile("./file.json", NULL, 0);

and you can save the in-memory JSON tree using jsonSave:

jsonSave(json, 0, "data.json", 0644, 0);

Navigating and Iterating

Properties in the JSON tree are stored as nodes. These nodes can be identified by a numeric ID or by node reference. Nodes and node IDs are used to efficiently traverse and enumerate property values.

To get a node ID, use jsonGetId:

int id = jsonGetId(json, 0, "address");

You can retrieve a node using another node as a starting point. This enables you to begin a property search from a point deep in the tree.

int addressId = jsonGetId(json, 0, "address");
int cityId = jsonGetId(json, addressId, "city");

In fact, most APIs such as jsonGet and jsonSet take a node ID as a parameter that specifies a root of the tree to base the operation. For example, these are equivalent:

int addressId = jsonGetId(json, 0, "address");
int cityId = jsonSet(json, addressId, "city", "Seattle", 0);

// and

jsonGetSet(json, 0, "address.city", "Brisbane", 0);

Similarly, you can retreive a node reference:

JsonNode *node = jsonGetNode(json, "address", "city");

You can iterate over nodes using the ITERATE_JSON macro:

JsonNode *child;
int id;
for (ITERATE_JSON(json, 0, child, id)) {
    printf("Property %s has value %s\n", child->name, child->value);
}

Debugging

Use the jsonPrint API to print a JSON tree to the console:

jsonPrint(json);

Memory References

The JSON engine retrieves values as static pointers within the in-memory tree. Values are returned as const char (cchar) values, which eliminates the need for duplicating strings. This approach leads to a reduced memory footprint and increased efficiency.

It is important not to cast returned values to (char*) or to modify the referenced strings using “dirty” programming to break the const typing protection.

Optimizations

When JSON processes text, it tokenizes the original content and utilizes it for individual property keys and values. As a result, the Ioto JSON parser does not require reallocating the JSON text, significantly reducing the memory footprint and enhancing performance.

JSON nodes are allocated in a single block, which may need expansion if new properties are inserted into a JSON tree. Consequently, JSON node (JsonNode*) references should not be stored persistently. Although JSON node IDs remain stable despite the growth of node storage, node references will be re-based. To store JSON node references, it is advisable to save IDs rather than node references.

JSON with the Ioto Web Server

The Ioto embedded web server fully supports HTTP requests with JSON bodies and responses.

If an HTTP request has a Content-Type of application/json, Ioto will automatically parse the JSON data into an in-memory store referenced by web->vars. From there, the Ioto JSON API can be used to query posted data.

For convenience, Ioto wraps the JSON APIs and provides a few convenience APIs that take the Web handle as an operand. These include:

For example:

cchar *username = webGetVar(web, "username", 0);

If the client request comes from a standard HTML form and uses a Content-Type of application/x-www-form-urlencoded the form data will converted by Ioto to JSON and stored for uniform access.

To send a JSON response, use the webWrite API after converting a JSON object to a string using **jsonToString.

For example:

webWrite(web, jsonToString(json, 0, 0, 0));

JSON with MQTT Messages

To send JSON messages over MQTT to the cloud follows a similar pattern using mqttPublish and jsonToString.

mqttPublish(ioto->mqtt, jsonToString(json, 0, 0, 0), -1, 1, MQTT_WAIT_NONE, "/mytopic");

To parse incoming JSON messages, use jsonParse and then jsonGet to retrieve a value.

mqttSubscribe(ioto->mqtt, incoming, 1, MQTT_WAIT_NONE, "ioto/%s/db/device/+", ioto->thing);

static void incoming(MqttRecv *rp)
{
    cchar   *command;
    Json    *json;

    if ((json = jsonParse(rp->data, 0)) == 0) {
        rError("sync", "Cannot parse message");
    } else {
        command = jsonGet(json, 0, "command", 0);
        dbUpdate(ioto->db, "State", DB_PROPS("command", command), 0);
        jsonFree(json);
    }
}

Summary

This scratches the surface of what you can do with the JSON engine in Ioto.

References

JSON5 Spec.

Want More Now?

To learn more about EmbedThis Ioto, please read:

Posted by Embedthis on Apr 6, 2023 in Ioto.

Comments

{{comment.name}} said ...

{{comment.message}}
{{comment.date}}

Make a Comment

Thank You!

Messages are moderated.

Your message will be posted shortly.

Sorry

Your message could not be processed at this time.

Error: {{error}}

Please retry later.

OK