Monitoring and Defending

ESP includes a monitoring and defensive response framework to mitigate denial-of-service (DOS) and vulnerability survey threats. This framework monitors key server metrics and triggers defenses should any metric be outside anticipated norms. Monitored metrics include the number of requests that result in an error, the number of SSL errors, the total number of requests and more.

Defensive responses invoke remedies that can be implemented to ward off the threat. Remedies include delaying or banning the offending client, notifying network administration or running an external program to implement a custom defense or recovery action.

Embedded Systems

Embedded devices typically have a well defined, understood and anticipated work load. Unlike their enterprise cousins, which have highly variable work loads, embedded systems usually serve a specific purpose with known clients that follow a more predictable access pattern. This allows an embedded system to define a range of operation that is considered "normal" and to trigger alerts and defenses if access is outside of this "normal" operation.

For example: an embedded system may expect to be accessed by at most 1 or 2 clients at a time. ESP can then be configured to regard it as threat if the system is accessed by more than 2 clients.

Monitors

Monitors are defined in the ESP pak.json configuration file via the http.montiors directive. The monitor directive takes the form:

monitors: {
    "NAME": {
        "expression": "EXPRESSION",
        "period": "FREQUENCY to check",
        "defenses": [ "DEFENSE" ],
    },
},

A monitor defines an expression to be regularly tested. The period defines how often it will be tested. If the expression is true, the list of defenses will be invoked to address the threat. For example:

monitors: {
    missing: {
        "expression": "NotFoundErrors > 50",
        "period": "1minute",
        "defenses": [ "block" ],
        "enable": true,
    },
},

This will test each minute if any client has more than 50 errors in the preceding period. If so, the "block" defense will be invoked.

Monitor Expressions

Monitor expressions are simple comparisons between a server Counter and a limit wrapped in quotes. They take the form:

"Counter [<|>] Limit"

Although a monitor can only access one counter, multiple monitors can be created.

Counters

Counters are metrics defined and maintained by ESP for monitoring purposes.

ESP provides three classes of of counters:

• Server-Wide Counters
• Per-Client Counters
• Custom Counters

Server-Wide counters pertain to the whole server and are not maintained per client. Per-Client counters are maintained per client IP address (not per TCP/IP socket connection). Custom counters are defined by the user developer using the httpAddCounter API.

Server-Wide Counters

The server-wide counters are accumulated for the entire ESP server and not per-client IP address. The value examined by Monitors is the current value of the counter.

Name	Description	Notes
ActiveClients	Count of client IP addresses recently accessing the server.	The ActiveClients counter is pruned of old clients whenever a monitor runs to examine its value or every 5 minutes whichever is more.
Memory	Amount of memory used by ESP.	The Memory counter is not representative of an outside security threat, but is provided for as it does impact the ability of the server to provide service.

Per-Client Counters

All the per-client counters are accumulated for each client IP address. For all counters except ActiveConnections and ActiveRequests, the value examined by Monitors is the change in the value over the monitor period. ActiveConnections and ActiveRequests are instantaneous values.

Name	Description
ActiveConnections	Count of active sockets.
ActiveRequests	Count of active requests.
BadRequestErrors	Count of requests that were badly formatted.
Errors	Count of requests that resulted in an error of some kind.
LimitErrors	Count of requests that violated the defined limits in esp.json.
NetworkIO	Total network I/O in bytes performed.
NotFoundErrors	Count of requests that accessed a resource not found on the server.
Requests	Count of requests.
SSLErrors	Count of SSL Errors.

Hackers often issue requests that result in errors as a means of profiling the vulnerabilities of a server. Too many BadRequestErrors or NotFoundErrors may indicate hacking activity. Similarly, too many SSLErrors may be an indication that someone is probing the SSL implementation.

The Errors counter is an aggregation of all request errors regardless of classification.

Defenses

Monitors specify one or more Defenses to be invoked when the monitor is triggered. Defenses invoke remedies to mitigate the threat.

Defenses are user-define rules file that specify a remedy to invoke. The Defenses directive takes the form:

defenses: {
    "NAME": {
        remedy: "REMEDY",
        status: STATUS,
        message: "MESSAGE",
        period: "PERIOD",
    },
},

The defense Name is the name used by Monitors running the defense. Multiple Monitors may reference a single defense. Remedy arguments are provided as properties in the defense. For example:

defenses: {
    report: {
        remedy: "email",
        to: "admin@example.com",
        from: "admin",
        subject: "Network defense initiated",
    }
}

Remedies

ESP provides a suite of remedies for Defenses to invoke to mitigate security threats. The remedy names are used in Defense remedy property.

Name	Description
ban	Ban the client IP address.
cmd	Run an external command.
delay	Delay requests from a client IP address.
email	Send an email message.
http	Invoke a HTTP request.
log	Write to the ESP error log.
reboot	Reboot ESP

Ban Remedy

The ban remedy prevents requests from the banned client IP address for a period of time. The default action is to simply close the socket and ignore the request. This is most efficient and gives the offending client the minimum of information. Alternatively, you can specify a response STATUS and MESSAGE. For example, to ban without a response message:

defenses: {
    mydefense: {
        remedy: "ban",
        period: "10mins",
    }
}

To ban with a response message:

defenses: {
    mydefense: {
        remedy: "ban",
        status: 406
        message: "Client banned",
        period: "10mins",
    }
}

Cmd Remedy

The Cmd remedy invokes an external command. This is an open interface to permit you to run a program of your choosing. The command line is specified by the CMD keyword. For example:

defenses: {
    mydefense: {
        remedy: "cmd",
        cmd: "path/to/my/program args ..."
    }
}

To send data to the standard input of the command, use the pipe symbol: |. This will send a generic message to the sendmail program

defenses: {
    mydefense: {
        remedy: "cmd",
        cmd: "${MESSAGE} | sendmail user@example.com",
    }
}

The command is run by ESP in the background and will not block the server.

Delay Remedy

Sometimes a complete ban is too much, so the Delay remedy continues to serve the offending client, but with a delay to reduce their impact on the server. This remedy can be effective to prevent brute-force attacks the enumerate URLs or potential vulnerabilities. For example:

defenses: {
    mydefense: {
        remedy: "delay",
        delay: "1sec",
        period: "10mins",
    }
}

This will introduce a one second delay for each request from the offending client IP address. The delay units are by default milliseconds when used without a "sec|min|hr" suffix.

Email Remedy

The email remedy provides a interface to a local sendmail client program to send email notification. Your system must have the sendmail client installed and configured to use the email remedy.

The destination is specified by the TO keyword. ESP will provide a default subject and from address. These can be overridden via the SUBJECT and FROM keywords.

For example:

defenses: {
    mydefense: {
        remedy: "email",
        subject: "Alert",
        from: "root@server14",
        to: "info@example.com"
    }
}

HTTP Remedy

The http remedy allows you to issue a HTTP request as part of your defense. The default HTTP method is POST and this can be overridden via the method property. A default message is posted and this can be overridden by the message property.

The HTTP request will wait for a response for up to the configured request timeout.

Some examples:

defenses: {
    report: {
        remedy: "http",
        uri: "http://example.com/report",
    },
    escalate: {
        remedy: "http",
        method: "post"
        message": "Under Attack",
        uri: "http://example.com/report",
    }
}

Log Remedy

The log remedy simply writes a message to the error log file.

The message can be augmented or overridden via the message property.

defenses: {
    record: {
        remedy: "log",
        message: "Details: ${MESSAGE}",
    },
}

Reboot Remedy

The reboot remedy will immediately reboot ESP. This may be useful if the Memory utilization exceeds the defined limit. This should not be used in production as any memory growth should be diagnosed and eliminated.

Generate Notes

All remedies have access the following KEYWORD tokens. These are also used in the message:

Name	Description
COUNTER	Name of the triggering counter
DATE	Date when the monitor triggered
IP	IP address of the offending client if relevant
LIMIT	Limit configured by the Monitor
MESSAGE	Generic or overridden message
PERIOD	Sampling period defined by the Monitor
SUBJECT	Descriptive title for the event
VALUE	Counter value

More Examples

defenses: {
    block: {
        remedy: "ban",
        period: "30 mins",
    },
    alarm: {
        remedy: "cmd",
        cmd: "afplay klaxon.mp3",
    },
    goslow: {
        remedy: "delay",
        delay: "1sec",
        period: "10mins",
    },
    report: {
        remedy: "http",
        uri: "http://example.com/report",
    },
},
monitors: {
    probe: {
        "expression": "NotFoundErrors > 50",
        "period": "1minute",
        "defenses": [ "block" ],
    },
    probe2: {
        "expression": "Errors > 100",
        "period": "1minute",
        "defenses": [ "block" ],
    },
    load: {
        "expression": "NetworkIO > 50MB",
        "period": "1minute",
        "defenses": [ "block" ],
    },
    manyConnections: {
        "expression": "ActiveConnections > 10",
        "period": "1minute",
        "defenses": [ "block" ],
    },
    manyRequests: {
        "expression": "ActiveRequests > 10",
        "period": "1minute",
        "defenses": [ "block" ],
    }
}

APIs

You can define Monitors and Defenses via the esp.json file, or via the ESP APIs: httpAddDefense and httpAddMonitor. Via the API, you can also define remedies using httpAddRemedy.