Monitoring and Defending
Appweb includes a monitoring and defensive response framework to mitigate denial-of-service (DOS) and vulnerability survey threats. This framework monitors key server metrics and triggers defenses should any metric be outside anticipated norms. Monitored metrics include the number of requests that result in an error, the number of SSL errors, the total number of requests and more.
Defensive responses invoke remedies that can be implemented to ward off the threat. Remedies include delaying or banning the offending client, notifying network administration or running an external program to implement a custom defense or recovery action.
Embedded Systems
Embedded devices typically have a well defined, understood and anticipated work load. Unlike their enterprise cousins, which have highly variable work loads, embedded systems usually serve a specific purpose with known clients that follow a more predictable access pattern. This allows an embedded system to define a range of operation that is considered "normal" and to trigger alerts and defenses if access is outside of this "normal" operation.
For example: an embedded system may expect to be accessed by at most 1 or 2 clients at a time. Appweb can then be configured to regard it as threat if the system is accessed by more than 2 clients.
Monitors
Monitors are defined in the appweb.conf configuration file via the Monitor directive. The monitor directive takes the form:
Monitor Expression Period List-of-Defenses...
The monitor directive defines an expression to be regularly tested. The Period defines how often it will be tested. If the expression is true, the list of defenses will be invoked to address the threat. For example:
Monitor "Errors > 20" 1min report
This will test each minute if any client has more than 20 errors in the preceding period. If any are found, the "report" defense will be invoked.
Monitor Expressions
Monitor expressions are simple comparisons between a server Counter and a limit wrapped in quotes. They take the form:
"Counter [<|>] Limit"
Although a monitor can only access one counter, multiple monitors can be created.
Counters
Counters are metrics defined and maintained by Appweb for monitoring purposes.
Appweb provides three classes of of counters:
- Server-Wide Counters
- Per-Client Counters
- Custom Counters
Server-Wide counters pertain to the whole server and are not maintained per client. Per-Client counters are maintained per client IP address (not per TCP/IP socket connection). Custom counters are defined by the user developer using the httpAddCounter API.
Server-Wide Counters
The server-wide counters are accumulated for the entire Appweb server and not per-client IP address. The value examined by Monitors is the current value of the counter.
| Name | Description | Notes |
|---|---|---|
| ActiveClients | Count of client IP addresses recently accessing the server. | The ActiveClients counter is pruned of old clients whenever a monitor runs to examine its value or every 5 minutes whichever is more. |
| ActiveProcesses | Count of CGI processes actively running. | |
| Memory | Amount of memory used by Appweb. | The Memory counter is not representative of an outside security threat, but is provided for as it does impact the ability of the server to provide service. |
Per-Client Counters
All the per-client counters are accumulated for each client IP address. For all counters except ActiveConnections and ActiveRequests, the value examined by Monitors is the change in the value over the monitor period. ActiveConnections and ActiveRequests are instantaneous values.
| Name | Description |
|---|---|
| ActiveConnections | Count of active sockets. |
| ActiveRequests | Count of active requests. |
| BadRequestErrors | Count of requests that were badly formatted. |
| Errors | Count of requests that resulted in an error of some kind. |
| LimitErrors | Count of requests that violated the defined limits in appweb.conf. |
| NetworkIO | Total network I/O in bytes performed. |
| NotFoundErrors | Count of requests that accessed a resource not found on the server. |
| Requests | Count of requests. |
| SSLErrors | Count of SSL Errors. |
Hackers often issue requests that result in errors as a means of profiling the vulnerabilities of a server. Too many BadRequestErrors or NotFoundErrors may indicate hacking activity. Similarly, too many SSLErrors may be an indication that someone is probing the SSL implementation.
The Errors counter is an aggregation of all request errors regardless of classification.
Defenses
Monitors specify one or more Defenses to be invoked when the monitor is triggered. Defenses invoke remedies to mitigate the threat.
Defenses are user-define rules in the appweb.conf file that specify which remedy to invoke. Defenses must be defined before any monitor that uses the defense. The Defenses directive takes the form:
Defense Name RemedyArgs...
The defense Name is the name specified by Monitors using the defense. Multiple Monitors may reference a single defense. The remedy is specified via the RemedyArgs which are of the form:
KEYWORD=VALUE
The VALUE string may embed other keyword values by using the form: "${KEYWORD}". This will substitute the value of the keyword. For example:
REMDY=ban PERIOD=10mins
Remedies
Appweb provides a suite of remedies for Defenses to invoke to mitigate security threats. The remedy names are used in Defense directives.
| Name | Description |
|---|---|
| ban | Ban the client IP address. |
| cmd | Run an external command. |
| delay | Delay requests from a client IP address. |
| Send an email message. | |
| http | Invoke a HTTP request. |
| log | Write to the Appweb error log. |
| reboot | Reboot Appweb |
Ban Remedy
The ban remedy prevents requests from the banned client IP address for a period of time. The default action is to simply close the socket and ignore the request. This is most efficient and gives the offending client the minimum of information. Alternatively, you can specify a response STATUS and MESSAGE. For example, to ban without a response message:
Defense mydefense REMEDY=ban PERIOD=10mins
To ban with a response:
Defense mydefense REMDY=ban STATUS=406 MESSAGE="Client banned" PERIOD=10mins
Cmd Remedy
The Cmd remedy invokes an external command. This is an open interface to permit you to run a program of your choosing. The command line is specified by the CMD keyword. For example:
Defense mydefense REMEDY=cmd CMD="/path/to/my/program args ..."
To send data to the standard input of the command, use the pipe symbol: |. This will send a generic message to the sendmail program
Defense mydefense REMEDY=cmd CMD="${MESSAGE} | sendmail user@example.com"
The command is run by Appweb in the background and will not block the server. You may sleep in the command for the designated ${PERIOD} and then reverse the ban.
Delay Remedy
Sometimes a complete ban is too much, so the Delay remedy continues to serve the offending client, but with a delay to reduce their impact on the server. This remedy can be effective to prevent brute-force attacks the enumerate URLs or potential vulnerabilities. For example:
Defense mydefense REMEDY=delay DELAY=1sec PERIOD=10mins
This will introduce a one second delay for each request from the offending client IP address. The delay units are by default milliseconds when used without a "sec|min|hr" suffix.
Email Remedy
The email remedy provides a interface to a local sendmail client program to send email notification. Your system must have the sendmail client installed and configured to use the email remedy.
The destination is specified by the TO keyword. Appweb will provide a default subject and from address. These can be overridden via the SUBJECT and FROM keywords.
For example:
Defense mydefense REMEDY=email TO=info@example.com
Defense notify REMEDY=email SUBJECT="Alert" FROM=root@server14 TO=info@example.com
HTTP Remedy
The http remedy allows you to issue a HTTP request as part of your defense. The default HTTP method is POST and this can be overridden via the METHOD keyword. A default message is posted and this can be overridden by the MESSAGE keyword.
The HTTP request will wait for a response up to the configured RequestTimeout in the appweb.conf file.
Some examples:
Defense report REMEDY=http URI=http://example.com/report
Defense report REMEDY=http METHOD=GET URI=http://example.com/alert
Defense report REMEDY=http METHOD=POST MESSAGE="Under Attack" URI=http://example.com/report
Log Remedy
The log remedy simply writes a message to the log file.
The message can be augmented or overridden via the MESSAGE keyword.
Defense report REMEDY=log
Defense report REMEDY=log MESSAGE="Report ${MESSAGE}"
Reboot Remedy
The reboot remedy will immediately reboot Appweb. This may be useful if the Memory utilization exceeds the defined limit. This should not be used in production as any memory growth should be diagnosed and eliminated.
Generate Notes
All remedies have access the following KEYWORDS. These are also used in the message:
| Name | Description |
|---|---|
| COUNTER | Name of the triggering counter |
| DATE | Date when the monitor triggered |
| IP | IP address of the offending client if relevant |
| LIMIT | Limit configured by the Monitor |
| MESSAGE | Generic or overridden message |
| PERIOD | Sampling period defined by the Monitor |
| SUBJECT | Descriptive title for the event |
| VALUE | Counter value |
Examples
Defense block REMEDY=ban PERIOD=30mins Defense alarm REMEDY=cmd CMD="afplay klaxon.mp3" Defense goslow REMEDY=delay PERIOD=10mins DELAY=1sec Defense report REMEDY=http URI=http://example.com/report Defense notify REMEDY=email TO=info@example.com Monitor "NotFoundErrors > 20" 1min block Monitor "Errors > 100" 1min block Monitor "NetworkIO > 50MB" 1min block Monitor "ActiveConnections > 10" 1min block Monitor "ActiveRequests > 10" 1min block
APIs
You can define Monitors and Defenses via the appweb.conf file, or via the Appweb APIs: httpAddDefense and httpAddMonitor. Via the API, you can also define remedies using httpAddRemedy.