Multithreaded Programming

Overview

Programming in a multithreaded environment can be difficult. Sometimes programming errors are timing related as multiple threads interact. Multithread locks can become tangled and bugs can be difficult to reproduce. To alleviate these problems and enable the benefits of a multi-threaded core, ESP provides a suite of facilities to make multithreaded programming easier, more reliable, and more efficient.

The Problem of Multithreaded I/O Events

A particularly thorny issue in multithreaded servers is how to handle I/O without consuming a thread for each request. It is not practical to dedicate a thread to each HTTP request. A single browser will often send 6-10 simultaneous requests. If each request consumed a thread for the duration, the web server would quickly consume too many threads and system performance would greatly suffer.

One solution is to use a thread pool. This allows a request to borrow a thread from a thread pool while the request is active and return the thread when it cannot immediately continue with servicing the request. A thread may not be able to continue because it is waiting for I/O from the client, or waiting for I/O to the client to drain over the network. Returning the thread to the pool allows other requests to use the thread while the first request is waiting for network I/O. When the network is ready, a thread can be obtained from the pool and the original request can continue. For this to work, an efficient network event service is essential. ESP uses such a thread pool and event service to efficiently manage thread resources.

However, this use of such an event service and thread pool raises another problem: races between the foreground request thread and the background async I/O event thread. It is easy for these two threads to simultaneously interact and corrupt critical data structures. A typical solution is to use multithread locks to serialize access to such data, but this is a crude solution and often leads to brittle applications. ESP has a better solution that effectively serializes all activity for a request: per-request event dispatchers.

Event Dispatchers

The Multithreaded Portable Runtime (MPR) used by ESP has a facility called Event Dispatchers. These are event queues on which all I/O and other event activity for a request can be queued and serviced. Each request has its own dispatcher and so events for a request are serialized. When a network I/O event is received by ESP for a request, an event is queued on the request's dispatcher. If the request is currently active (using a thread from the thread pool), the event is queued and no immediate action is taken. When the request has finished its current activity, it will service events on its dispatcher queue and eventually service the I/O event. If the request is currently idle, a thread is assigned from the thread pool for the request, and the thread is resumed to service the request's dispatcher queue. This greatly simplifies ESP as all activity for a request is thus serialized via the dispatcher queue.

By using event dispatchers, multithread locking is not required on every API — because activity is serialized by the dispatcher. This significantly boosts performance and reduces the risk of lock contention.

Multithreaded ESP

By using request dispatchers, ESP serializes all request activity, yet it can support many simultaneous requests due to its multithreaded core. ESP efficiently utilizes thread resources by using a thread pool and not permanently dedicating threads to requests. Threads are temporarily assigned only as required by active requests.

Modules

ESP can be extended via ESP applications. These are loadable ESP web pages, and ESP Model-View-Controller applications. These can rely on the fact that request activity is serialized and locking primitives are typically not required. ESP code can thus be simple, single-threaded application code.

Locking

If you have a requirement for a data structure that will be accessed and manipulated simultaneously by multiple threads, the MPR provides a suite of locking primitives. See MprSync for the MPR Multithreaded Synchronization Services.