Monitoring and Defending

Appweb includes a monitoring and defensive response framework to mitigate denial-of-service (DOS) and vulnerability survey threats. This framework monitors key server metrics and triggers defenses should any metric be outside anticipated norms. Monitored metrics include the number of requests that result in an error, the number of SSL errors, the total number of requests and more.

Defensive responses invoke remedies that can be implemented to ward off the threat. Remedies include delaying or banning the offending client, notifying network administration or running an external program to implement a custom defense or recovery action.

Embedded Systems

Embedded devices typically have a well defined, understood and anticipated work load. Unlike their enterprise cousins, which have highly variable work loads, embedded systems usually serve a specific purpose with known clients that follow a more predictable access pattern. This allows an embedded system to define a range of operation that is considered "normal" and to trigger alerts and defenses if access is outside of this "normal" operation.

For example: an embedded system may expect to be accessed by at most 1 or 2 clients at a time. Appweb can then be configured to regard it as threat if the system is accessed by more than 2 clients.

Monitors

Monitors are defined in the appweb.conf configuration file via the Monitor directive. The monitor directive takes the form:

Monitor Expression Period List-of-Defenses...

The monitor directive defines an expression to be regularly tested. The Period defines how often it will be tested. If the expression is true, the list of defenses will be invoked to address the threat. For example:

Monitor "Errors > 20" 1min report

This will test each minute if any client has more than 20 errors in the preceding period. If any are found, the "report" defense will be invoked.

Monitor Expressions

Monitor expressions are simple comparisons between a server Counter and a limit wrapped in quotes. They take the form:

"Counter [<|>] Limit"

Although a monitor can only access one counter, multiple monitors can be created.

Counters

Counters are metrics defined and maintained by Appweb for monitoring purposes.

Appweb provides three classes of of counters:

Server-Wide counters pertain to the whole server and are not maintained per client. Per-Client counters are maintained per client IP address (not per TCP/IP socket connection). Custom counters are defined by the user developer using the httpAddCounter API.

Server-Wide Counters

The server-wide counters are accumulated for the entire Appweb server and not per-client IP address. The value examined by Monitors is the current value of the counter.

NameDescriptionNotes
ActiveClients Count of client IP addresses recently accessing the server. The ActiveClients counter is pruned of old clients whenever a monitor runs to examine its value or every 5 minutes whichever is more.
ActiveProcesses Count of CGI processes actively running.
Memory Amount of memory used by Appweb. The Memory counter is not representative of an outside security threat, but is provided for as it does impact the ability of the server to provide service.

Per-Client Counters

All the per-client counters are accumulated for each client IP address. For all counters except ActiveConnections and ActiveRequests, the value examined by Monitors is the change in the value over the monitor period. ActiveConnections and ActiveRequests are instantaneous values.

NameDescription
ActiveConnections Count of active sockets.
ActiveRequests Count of active requests.
BadRequestErrors Count of requests that were badly formatted.
Errors Count of requests that resulted in an error of some kind.
LimitErrors Count of requests that violated the defined limits in appweb.conf.
NetworkIO Total network I/O in bytes performed.
NotFoundErrors Count of requests that accessed a resource not found on the server.
Requests Count of requests.
SSLErrors Count of SSL Errors.

Hackers often issue requests that result in errors as a means of profiling the vulnerabilities of a server. Too many BadRequestErrors or NotFoundErrors may indicate hacking activity. Similarly, too many SSLErrors may be an indication that someone is probing the SSL implementation.

The Errors counter is an aggregation of all request errors regardless of classification.

Defenses

Monitors specify one or more Defenses to be invoked when the monitor is triggered. Defenses invoke remedies to mitigate the threat.

Defenses are user-define rules in the appweb.conf file that specify which remedy to invoke. Defenses must be defined before any monitor that uses the defense. The Defenses directive takes the form:

Defense Name RemedyArgs...

The defense Name is the name specified by Monitors using the defense. Multiple Monitors may reference a single defense. The remedy is specified via the RemedyArgs which are of the form:

KEYWORD=VALUE

The VALUE string may embed other keyword values by using the form: "${KEYWORD}". This will substitute the value of the keyword. For example:

REMDY=ban PERIOD=10mins

Remedies

Appweb provides a suite of remedies for Defenses to invoke to mitigate security threats. The remedy names are used in Defense directives.

NameDescription
banBan the client IP address.
cmdRun an external command.
delayDelay requests from a client IP address.
emailSend an email message.
httpInvoke a HTTP request.
logWrite to the Appweb error log.
rebootReboot Appweb

Ban Remedy

The ban remedy prevents requests from the banned client IP address for a period of time. The default action is to simply close the socket and ignore the request. This is most efficient and gives the offending client the minimum of information. Alternatively, you can specify a response STATUS and MESSAGE. For example, to ban without a response message:

Defense mydefense REMEDY=ban PERIOD=10mins

To ban with a response:

Defense mydefense REMDY=ban STATUS=406 MESSAGE="Client banned" PERIOD=10mins

Cmd Remedy

The Cmd remedy invokes an external command. This is an open interface to permit you to run a program of your choosing. The command line is specified by the CMD keyword. For example:

Defense mydefense REMEDY=cmd CMD="/path/to/my/program args ..."

To send data to the standard input of the command, use the pipe symbol: |. This will send a generic message to the sendmail program

Defense mydefense REMEDY=cmd CMD="${MESSAGE} | sendmail user@example.com"

The command is run by Appweb in the background and will not block the server. You may sleep in the command for the designated ${PERIOD} and then reverse the ban.

Delay Remedy

Sometimes a complete ban is too much, so the Delay remedy continues to serve the offending client, but with a delay to reduce their impact on the server. This remedy can be effective to prevent brute-force attacks the enumerate URLs or potential vulnerabilities. For example:

Defense mydefense REMEDY=delay DELAY=1sec PERIOD=10mins

This will introduce a one second delay for each request from the offending client IP address. The delay units are by default milliseconds when used without a "sec|min|hr" suffix.

Email Remedy

The email remedy provides a interface to a local sendmail client program to send email notification. Your system must have the sendmail client installed and configured to use the email remedy.

The destination is specified by the TO keyword. Appweb will provide a default subject and from address. These can be overridden via the SUBJECT and FROM keywords.

For example:

Defense mydefense REMEDY=email TO=info@example.com
Defense notify REMEDY=email SUBJECT="Alert" FROM=root@server14 TO=info@example.com

HTTP Remedy

The http remedy allows you to issue a HTTP request as part of your defense. The default HTTP method is POST and this can be overridden via the METHOD keyword. A default message is posted and this can be overridden by the MESSAGE keyword.

The HTTP request will wait for a response up to the configured RequestTimeout in the appweb.conf file.

Some examples:

Defense report REMEDY=http URI=http://example.com/report
Defense report REMEDY=http METHOD=GET URI=http://example.com/alert
Defense report REMEDY=http METHOD=POST MESSAGE="Under Attack" URI=http://example.com/report

Log Remedy

The log remedy simply writes a message to the log file.

The message can be augmented or overridden via the MESSAGE keyword.

Defense report REMEDY=log
Defense report REMEDY=log MESSAGE="Report ${MESSAGE}"

Reboot Remedy

The reboot remedy will immediately reboot Appweb. This may be useful if the Memory utilization exceeds the defined limit. This should not be used in production as any memory growth should be diagnosed and eliminated.

Generate Notes

All remedies have access the following KEYWORDS. These are also used in the message:

NameDescription
COUNTERName of the triggering counter
DATEDate when the monitor triggered
IPIP address of the offending client if relevant
LIMITLimit configured by the Monitor
MESSAGEGeneric or overridden message
PERIODSampling period defined by the Monitor
SUBJECTDescriptive title for the event
VALUECounter value

Examples

Defense block REMEDY=ban PERIOD=30mins
Defense alarm REMEDY=cmd CMD="afplay klaxon.mp3"
Defense goslow REMEDY=delay PERIOD=10mins DELAY=1sec
Defense report REMEDY=http URI=http://example.com/report
Defense notify REMEDY=email TO=info@example.com
Monitor "NotFoundErrors > 20" 1min block
Monitor "Errors > 100" 1min block
Monitor "NetworkIO > 50MB" 1min block
Monitor "ActiveConnections > 10" 1min block
Monitor "ActiveRequests > 10" 1min block

APIs

You can define Monitors and Defenses via the appweb.conf file, or via the Appweb APIs: httpAddDefense and httpAddMonitor. Via the API, you can also define remedies using httpAddRemedy.

© Embedthis Software, 2003-2015. All rights reserved.