URL API
The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.
The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.
The library has configurable request/response timeouts and retries.
Features:
- Supports transfer-chunk encoding for dynamic request bodies.
- Control over headers
- Full-duplex streaming support for minimal memory usage.
- Full support for TLS including certificate verification.
- Configurable retries and timeouts.
- Parallelism via fiber coroutines.
The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.
Function Index
Url * | urlAlloc(int flags) |
Allocate a URL object. | |
void | urlClose(Url *up) |
Close the underlying network socket associated with the URL object. | |
int | urlError(Url *up, cchar *message, ...) |
Set the URL internal error message. | |
int | urlFetch(Url *up, cchar *method, cchar *url, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL. | |
int | urlFinalize(Url *up) |
Finalize the request. | |
void | urlFree(Url *up) |
Free a URL object and all resources. | |
char * | urlGet(cchar *url, cchar *headers, ...) |
Get a URL using a HTTP GET request. | |
cchar * | urlGetError(Url *up) |
Get the URL internal error message. | |
cchar * | urlGetHeader(Url *up, cchar *header) |
Get a response HTTP header from the parsed response headers. | |
Json * | urlGetJson(cchar *url, cchar *headers, ...) |
Issue a HTTP GET request and return parsed JSON. | |
Json * | urlGetJsonResponse(Url *up) |
Get the response to a URL request as a JSON object tree. | |
cchar * | urlGetResponse(Url *up) |
Get the response to a URL request as a string. | |
RBuf * | urlGetResponseBuf(Url *up) |
Get the response to a URL request in a buffer. | |
int | urlGetStatus(Url *up) |
Get the HTTP response status code for a request. | |
WebSocket * | urlGetWebSocket(Url *up) |
Get the WebSocket object for a URL request. | |
Json * | urlJson(Url *up, cchar *method, cchar *url, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL and return a JSON response if the HTTP request is successful. | |
int | urlParse(Url *up, cchar *url) |
Parse a URL into its constituent components in the Url structure. | |
char * | urlPost(cchar *url, cvoid *data, ssize size, cchar *headers, ...) |
Issue a HTTP POST request. | |
Json * | urlPostJson(cchar *url, cvoid *data, ssize len, cchar *headers, ...) |
Issue a HTTP POST request and return parsed JSON. | |
ssize | urlRead(Url *up, char *buf, ssize bufsize) |
Low level read routine to read response data for a request. | |
void | urlSetBufLimit(Url *up, ssize limit) |
Set the maximum number of bytes to buffer from the response. | |
void | urlSetCerts(Url *up, cchar *ca, cchar *key, cchar *cert, cchar *revoke) |
Define the certificates to use with TLS. | |
void | urlSetCiphers(Url *up, cchar *ciphers) |
Set the list of available ciphers to use. | |
void | urlSetDefaultTimeout(Ticks timeout) |
Set the default request timeout to use for future URL instances. | |
void | urlSetFlags(Url *up, int flags) |
Set the URL flags. | |
void | urlSetMaxRetries(Url *up, int maxRetries) |
Set the maximum number of retries for SSE requests. | |
void | urlSetProtocol(Url *up, int protocol) |
Set the HTTP protocol to use. | |
void | urlSetStatus(Url *up, int status) |
Set the HTTP response status. | |
void | urlSetTimeout(Url *up, Ticks timeout) |
Set the request timeout to use for the specific URL object. | |
void | urlSetVerify(Url *up, int verifyPeer, int verifyIssuer) |
Control verification of TLS connections. | |
int | urlStart(Url *up, cchar *method, cchar *url) |
Start a Url request. | |
int | urlUpload(Url *up, RList *files, RHash *forms, cchar *headers, ...) |
Upload files in a request. | |
int | urlWait(Url *up) |
Wait for the connection to be closed. | |
int | urlWebSocket(cchar *url, WebSocketProc callback, void *arg, cchar *headers) |
Issue a simple WebSocket request. | |
void | urlWebSocketAsync(Url *up, WebSocketProc callback, void *arg) |
Define the async callbacks. | |
ssize | urlWrite(Url *up, cvoid *data, ssize size) |
Write body data for a request. | |
ssize | urlWriteFile(Url *up, cchar *path) |
Write a file contents for a request. | |
ssize | urlWriteFmt(Url *up, cchar *fmt, ...) |
Write formatted body data for a request. | |
int | urlWriteHeaders(Url *up, cchar *headers) |
Write request headers using a printf style formatted pattern. |
Typedef Index
Url | Url request object. |
UrlSseProc | URL SSE callback. |
Defines
#define | URL_CODE_ACCEPTED 202 |
The request has been accepted and processing is continuing. | |
#define | URL_CODE_BAD_GATEWAY 502 |
The server cannot act as a gateway for the given request. | |
#define | URL_CODE_BAD_METHOD 405 |
The request HTTP method was not supported by the resource. | |
#define | URL_CODE_BAD_REQUEST 400 |
The request is malformed. | |
#define | URL_CODE_BAD_VERSION 505 |
The server does not support the HTTP protocol version. | |
#define | URL_CODE_CONFLICT 409 |
The request had a conflict in the request headers and URI. | |
#define | URL_CODE_CONTINUE 100 |
Continue with request, only parial content transmitted. | |
#define | URL_CODE_CREATED 201 |
The request has completed and a new resource was created. | |
#define | URL_CODE_EXPECTATION_FAILED 417 |
The server cannot satisfy the Expect header requirements. | |
#define | URL_CODE_FORBIDDEN 403 |
The request was legal, but the server refuses to process. | |
#define | URL_CODE_GATEWAY_TIMEOUT 504 |
The server gateway timed out waiting for the upstream server. | |
#define | URL_CODE_GONE 410 |
The requested resource is no longer available. | |
#define | URL_CODE_IM_A_TEAPOT 418 |
Short and stout error code (RFC 2324). | |
#define | URL_CODE_INSUFFICIENT_STORAGE 507 |
The server has insufficient storage to complete the request. | |
#define | URL_CODE_INTERNAL_SERVER_ERROR 500 |
Server processing or configuration error. | |
#define | URL_CODE_LENGTH_REQUIRED 411 |
The request did not specify a required content length. | |
#define | URL_CODE_MOVED_PERMANENTLY 301 |
The requested URI has moved permanently to a new location. | |
#define | URL_CODE_MOVED_TEMPORARILY 302 |
The URI has moved temporarily to a new location. | |
#define | URL_CODE_NO_CONTENT 204 |
The request has completed and there is no response to send. | |
#define | URL_CODE_NO_RESPONSE 444 |
The connection was closed with no response to the client. | |
#define | URL_CODE_NOT_ACCEPTABLE 406 |
The requested resource cannot generate the required content. | |
#define | URL_CODE_NOT_AUTHORITATIVE 203 |
The request has completed but content may be from another source. | |
#define | URL_CODE_NOT_FOUND 404 |
The requested resource was not found. | |
#define | URL_CODE_NOT_IMPLEMENTED 501 |
The server does not recognize the request or method. | |
#define | URL_CODE_NOT_MODIFIED 304 |
The requested resource has changed since the last request. | |
#define | URL_CODE_OK 200 |
The request completed successfully. | |
#define | URL_CODE_PARTIAL 206 |
The request has completed and is returning parial content. | |
#define | URL_CODE_PAYMENT_REQUIRED 402 |
Reserved for future use. | |
#define | URL_CODE_PERMANENT_REDIRECT 308 |
The request has been permanently redirected to a new location. | |
#define | URL_CODE_PRECOND_FAILED 412 |
The server cannot satisfy one of the request preconditions. | |
#define | URL_CODE_RANGE_NOT_SATISFIABLE 416 |
The request content range does not exist for the resource. | |
#define | URL_CODE_REQUEST_TIMEOUT 408 |
The server timed out waiting for the request to complete. | |
#define | URL_CODE_REQUEST_TOO_LARGE 413 |
The request is too large for the server to process. | |
#define | URL_CODE_REQUEST_URL_TOO_LARGE 414 |
The request URI is too long for the server to process. | |
#define | URL_CODE_RESET 205 |
The request has completed with no content. | |
#define | URL_CODE_SEE_OTHER 303 |
The requested URI can be found at another URI location. | |
#define | URL_CODE_SERVICE_UNAVAILABLE 503 |
The server is currently unavailable or overloaded. | |
#define | URL_CODE_SWITCHING 101 |
Switching protocols. | |
#define | URL_CODE_TEMPORARY_REDIRECT 307 |
The request should be repeated at another URI location. | |
#define | URL_CODE_UNAUTHORIZED 401 |
Authentication for the request has failed. | |
#define | URL_CODE_UNSUPPORTED_MEDIA_TYPE 415 |
The request media type is not supported by the server or resource. | |
#define | URL_CODE_USE_PROXY 305 |
The requested resource must be accessed via the location proxy. | |
#define | URL_HTTP_0 0x20 |
Use HTTP/1.0. | |
#define | URL_SHOW_NONE 0x1 |
Trace nothing. | |
#define | URL_SHOW_REQ_BODY 0x2 |
Trace request body. | |
#define | URL_SHOW_REQ_HEADERS 0x4 |
Trace request headers. | |
#define | URL_SHOW_RESP_BODY 0x8 |
Trace response body. | |
#define | URL_SHOW_RESP_HEADERS 0x10 |
Trace response headers. |
Typedefs
URL SSE callback.
- Parameters:
up URL object. id Event ID. event Type of event. data Associated data. arg User argument.
- API Stability:
- Evolving.
Url request object.
- Description:
- The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.
- Fields:
REvent abortEvent Abort event. char * boundary HTTP request method. Multipart mime upload file boundary. ssize bufLimit Maximum number of bytes to buffer from the response. uint certsDefined Certificates have been defined. uint chunked Request is using transfer chunk encoding. uint close Connection should be closed on completion of the current request. Ticks deadline Request time limit expiry deadline. char * error Error message for internal errors, not HTTP errors. RFiber * fiber Fiber. uint finalized The request has been finalized. uint flags Alloc flags. uint gotResponse Response has been read. cchar * hash Request hash portion. cchar * host Request host. uint inCallback SSE callback is in progress. ssize lastEventId Last event ID (SSE). uint maxRetries Maximum of retries for SSE. uint needFree Free the URL object. cchar * path Request path without leading "/" and query/ref. int port Request port. uint protocol Use HTTP/1.0 without keep-alive. Defaults to HTTP/1.1. cchar * query Request query portion. cchar * redirect Redirect location. char * response Response as a string. RBuf * responseBuf Buffer to hold complete response. uint retries Number of retries for SSE. RBuf * rx Buffer for progressive reading response data. RBuf * rxHeaders Buffer for Rx headers. ssize rxLen Length of rx body. ssize rxRemaining Remaining rx data to read from the socket. cchar * scheme Request scheme. RSocket * sock Network socket. uint sse SSE request. void * sseArg SSE callback argument. UrlSseProc sseProc SSE callback. uint status Response (rx) status. Ticks timeout Request timeout. RBuf * txHeaders Buffer for Tx headers. ssize txLen Length of tx body. uint upgraded WebSocket upgrade has been completed. char * url Request URL. char * urlbuf Parsed and tokenized URL. WebSocket * webSocket WebSocket object. uint wroteHeaders Tx headers have been written.
- API Stability:
- Evolving.
Functions
Allocate a URL object.
- Description:
- A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.
- Parameters:
flags Set flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY. Defaults to 0. Can also call urlSetFlags to set flags.
- Returns:
- The url object.
- API Stability:
- Evolving.
Close the underlying network socket associated with the URL object.
- Description:
- This is not normally necessary to invoke unless you want to force the socket connection to be recreated before issuing a subsequent request.
- Parameters:
up URL object.
- API Stability:
- Evolving.
Set the URL internal error message.
- Description:
- This routine is used to set the URL internal error message. Use $urlGetError to get the error message.
- Parameters:
up URL object. message Printf style message format string. ... Message arguments.
- Returns:
- Zero if successful, otherwise a negative status code.
- API Stability:
- Evolving.
Fetch a URL.
- Description:
- This routine issues a HTTP request with optional body data and returns the HTTP status code. This routine will return after the response status and headers have been read and before the response body data has been read. Use $urlRead or $urlGetResponse to read the response body data. This routine will block the current fiber while waiting for the request to respond. Other fibers continue to run.
- Parameters:
up URL object. method HTTP method verb. url HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Response HTTP status code. Use urlGetResponse or urlRead to read the response.
- API Stability:
- Evolving.
Finalize the request.
- Description:
- This routine finalizes the request and signifies that the entire request including any request body data has been sent to the server. This routine MUST be called at the end of writing the request body (if any). It will read the response status and headers before returning. If using WebSockets, this should be called before calling urlWebSocketAsync to verify the WebSocket handshake. Internally, this function is implemented by calling urlWrite with a NULL buffer. This call is idempotent.
- Parameters:
up Url object.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Free a URL object and all resources.
- Parameters:
up URL object.
- API Stability:
- Evolving.
Get a URL using a HTTP GET request.
- Description:
- This will issue a HTTP GET request to the specified URL. If the HTTP status is 200, this will return the response body. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Parameters:
url HTTP URL to fetch. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
Get the URL internal error message.
- Description:
- Errors are defined for unexpected errors. HTTP requests that respond with a non-200 status do not set the error message.
- Parameters:
up URL object.
- Returns:
- The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.
- API Stability:
- Evolving.
Get a response HTTP header from the parsed response headers.
- Parameters:
up URL object. header HTTP header name. This can be any case. For example: "Authorization".
- Returns:
- The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.
- API Stability:
- Evolving.
Issue a HTTP GET request and return parsed JSON.
- Description:
- This will issue a HTTP GET request to the specified URL and if successful, will return the parsed response body.
- Parameters:
url HTTP URL to fetch. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
- API Stability:
- Evolving.
Get the response to a URL request as a JSON object tree.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Parameters:
up URL object.
- Returns:
- The response body as parsed JSON object. Caller must free the result via jsonFree.
- API Stability:
- Evolving.
Get the response to a URL request as a string.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
If receiving a binary response, use urlGetResponseBuf instead.
- Parameters:
up URL object.
- Returns:
- The response body as a string. Caller must NOT free. Will return an empty string on errors.
- API Stability:
- Evolving.
Get the response to a URL request in a buffer.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Parameters:
up URL object.
- Returns:
- The response body as runtime buffer. Caller must NOT free.
- API Stability:
- Evolving.
Get the HTTP response status code for a request.
- Parameters:
up URL object.
- Returns:
- The HTTP status code for the most recently completed request.
- API Stability:
- Evolving.
Get the WebSocket object for a URL request.
- Parameters:
up URL object.
- Returns:
- The WebSocket object for the URL request.
- API Stability:
- Evolving.
Fetch a URL and return a JSON response if the HTTP request is successful.
- Description:
- This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Parameters:
up URL object. method HTTP method verb. url HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used. ... Optional header arguments.
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().
- API Stability:
- Evolving.
Issue a HTTP POST request.
- Description:
- This will issue a HTTP POST request to the specified URL with the optional body data. Regardless of the HTTP status, this will return the response body.
- Parameters:
url HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Headers arguments.
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
Issue a HTTP POST request and return parsed JSON.
- Description:
- This will issue a HTTP POST request to the specified URL with the optional body data. If successful, it will return the parsed response body.
- Parameters:
url HTTP URL to fetch. data Body data for request. Set to NULL if none. len Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
- API Stability:
- Evolving.
Low level read routine to read response data for a request.
- Description:
- This routine may be called to progressively read response data. It should not be called if using $urlGetResponse directly or indirectly via urlGet, urlPost, urlPostJson or urlJson. This routine will block the current fiber if necessary. Other fibers continue to run.
- Parameters:
up URL object. buf Buffer to read into. bufsize Size of the buffer.
- Returns:
- The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.
- API Stability:
- Evolving.
Set the maximum number of bytes to buffer from the response.
- Parameters:
up URL object. limit Maximum number of bytes.
- API Stability:
- Evolving.
Define the certificates to use with TLS.
- Parameters:
up URL object. ca Certificate authority to verify client certificates. key Certificate private key. cert Certificate text. revoke Certificates to revoke.
- API Stability:
- Evolving.
Set the list of available ciphers to use.
- Parameters:
up URL object. ciphers String list of available ciphers.
- API Stability:
- Evolving.
Set the default request timeout to use for future URL instances.
- Description:
- This does not change the timeout for existing Url objects.
- Parameters:
timeout Timeout in milliseconds.
- API Stability:
- Evolving.
Set the URL flags.
- Parameters:
up URL object. flags Set flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY.
- API Stability:
- Evolving.
Set the maximum number of retries for SSE requests.
- Parameters:
up URL object. maxRetries Maximum number of retries.
- API Stability:
- Evolving.
Set the HTTP protocol to use.
- Parameters:
up URL object. protocol Set to 0 for HTTP/1.0 or 1 for HTTP/1.1. Defaults to 1.
- API Stability:
- Evolving.
Set the HTTP response status.
- Parameters:
up URL object. status HTTP status code.
- API Stability:
- Evolving.
Set the request timeout to use for the specific URL object.
- Parameters:
up URL object. timeout Timeout in milliseconds.
- API Stability:
- Evolving.
Control verification of TLS connections.
- Parameters:
up URL object. verifyPeer Set to true to verify the certificate of the remote peer. verifyIssuer Set to true to verify the issuer of the peer certificate.
- API Stability:
- Evolving.
Start a Url request.
- Description:
- This is a low level API that initiates a connection to a remote HTTP resource. Use $urlWriteHeaders to write headers and $urlWrite to write any request body. Must call $urlFinalize to signify the end of the request body.
- Parameters:
up URL object. method HTTP method verb. url HTTP URL to fetch.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Upload files in a request.
- Description:
- This constructs and writes the HTTP request headers for a multipart/form-data request. Use this routine instead of $urlWriteHeaders to write headers. Use $urlStart to initiate the request before calling $urlUpload.
- Parameters:
up URL object. files List of filenames to upload. forms Hash of key/value form values to add to the request. headers Printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Wait for the connection to be closed.
- Description:
- Used for SSE and WebSocket connections.
- Parameters:
up URL object.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Issue a simple WebSocket request.
- Parameters:
url HTTP URL. callback Callback function to invoke for read data. arg Argument to pass to the callback function. headers Optional request headers. Individual header lines must be terminated with "\r\n".
- Returns:
- Zero when closed, otherwise a negative status code.
Define the async callbacks.
- Description:
- This will invoke the callback function with an OPEN event.
- Parameters:
up URL object. callback Callback function to invoke for read data. arg Argument to pass to the callback function.
- API Stability:
- Evolving.
Write body data for a request.
- Description:
- This routine will block the current fiber. Other fibers continue to run.
- Parameters:
up URL object. data Buffer of data to write. Set to NULL to finalize the request body. size Length of data to write. Set to -1 to calculate the length of data as a null terminated string.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write a file contents for a request.
- Description:
- This routine will read the file contents and write to the client. It will block the current fiber. Other fibers continue to run.
- Parameters:
up URL object. path File pathname.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write formatted body data for a request.
- Description:
- This routine will block the current fiber. Other fibers continue to run.
- Parameters:
up URL object. fmt Printf style format string. ... Format arguments.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write request headers using a printf style formatted pattern.
- Description:
- This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will add a Transfer-Encoding: chunked header if the Content-Length is not supplied and the request method is not a GET. If the HTTP scheme is ws or wss, this routine will add the WebSocket upgrade headers. Request headers will be traced if the flags have URL_SHOW_REQ_HEADERS set. This routine will block the current fiber. Other fibers continue to run.
- Parameters:
up URL object. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.