Evented APIs

Authors

Sam Curren, <sxc@kynetx.com>

Phillip J. Windley, <pjw@kynetx.com>

Status

Version: 1.0

Released: Oct 4, 2011

Overview

Events indicate something has happened. In this they differ from the request-response interaction style popular on the Web. Event-based systems are declarative whereas request-response systems are interrogatory. The difference between events (“this happened”) and requests (“will you do this?”) offers benefits in looser coupling of components as well as semantic encapsulation (see On Hierarchies and Networks for more detail).

APIs have become an economic imperative for many companies. But APIs based solely on request-response style interactions limit integrations to those where one system always knows what it wants from the other. The calling service must script the interaction and the APIs simply follow along.

We envision a world where applications integrate multiple products and services as equals based on event-driven interactions. Evented APIs, following the form described in this document, enable building such applications.

General Concepts

The following concepts are useful in understanding the specification that follows.

Components and General Operation

Event-driven systems comprise

Events are pushed from an event generator to event consumers when the event generators sees a relevant state change. Event consumers indicate their desire to see specific events through subscription. In the following diagram events from generators A-D are received by event consumers. Note that some events (e.g. A) are received by multiple consumers, some consumers received more than one event, and some events (e.g. C) are received by no one.

Using an Evented API, any program can act as as either an event generator or event consumer so long as it meets certain, minimal requirements (described below). The Internet--specifically the HTTP protocol--acts as the event distribution network. Subscription occurs when an event generator records the event signal URL for an event consumer.

Events are Not Remote Procedure Calls

Events are not Remote Procedure Calls (RPC) or requests. Events are a notification that something significant occurred. Events do not carry any instructions or directives. Events have attributes whereas an RPC has parameters.

Years ago, computer designers discovered that having some components interrupt the CPU when they needed service was more efficient and easier to design and program. Before interrupts, the CPU polled components to determine if they needed service. Without events, an application must poll an API which is difficult to manage and expensive from a computational and communications standpoint. Event-driven systems gain the same benefits that interrupts gave hardware designers years ago.

Events are State Changes

An event is a notification of a state change in an Evented API. The temporal granularity of an event is assumed to be small, although unspecified.  Event generators are free to determine what “state change” means. Event generators may specify a timestamp indicating when the event . occurred. If the timestamp is missing in the event signal, the timestamp of the event is determined by the timestamp of its delivery.

Language and Platform Agnostic

The Evented API specification is designed to be language and platform agnostic. The only language requirement is that event generators must be able to perform an HTTP POST or GET and event consumers must be able to receive them (they are associated with or act as an HTTP server). Occasionally-connected event consumers must run behind an event proxy service that stores and forwards events.

Event Generator Bias

The Evented API specification has been designed so that signalling events by event generators is as easy as possible. Where work or complexity could be shifted from the event generator to the event consumer, we have done so.

How it Works

Event Schema

An event has the following components:

Event Dictionary

Event generators are responsible for publishing a dictionary of events by domain and name. The dictionary should list possible attributes along with the syntax and semantics. Event consumers are responsible for understanding the event dictionary.

Event Signal URL

An event consumer subscribes to an event generator by recording the consumer’s event signal URL. The semantics of the event signal URL are opaque to the event generator although its structure should have meaning to the event consumer.  The event signal URL encodes the entity identifier and thus represents an event consumer for a particular entity.

The event consumer creates the event signal URL, as follows:

The event signal URL must not contain a query string.

Event Signaling

The event generator signals an event by encoding the event as a query string, appending it to the event signal URLs that it has recorded, and using HTTP to signal each URL.

The event generator should signal events using HTTP POST using the event signal URL. The event domain, event name, and attributes must be sent as key-value pairs in the body of the POST. Keys with a leading underscore character are reserved.

There are two required key-value pairs that must be sent:

The values sent for _domain and _name must be a string made of alphanumeric, underscore, dot, and dash characters [a-zA-Z0-9_.-].  

In addition to the required key-value pairs given above, the generator may include any other attributes that the generator wishes to send with the event.

Generators may include a timestamp indicating when the event occurred (which can be different than when it was signalled) using the _timestamp key. The value of this field is an HTTP-date. Consumers may use the value in the _timestamp field in lieu of the time when the signal was received.

When sent as the body of an HTTP POST these parameters must be encoded using one of the methods shown in the following table. Note that the Content-Type header must be set to the appropriate mime-type for the event consumer to understand the message.

encoding

mime-type

form

application/x-www-form-urlencoded

JSON

application/json

If the body is JSON-encoded, the encoding should be a JSON object containing the keys and values as follows:

{“_domain” : “web”,

 “_name” : “pageview”,

 “urls” : [“http://www.exampley.com/foo/bar.html”,

           “http://www.google.com/search”]

}

Using HTTP GET

An event generator may use an HTTP GET instead of POST in circumstances where a POST is difficult. In this case, the key-value pairs representing the event name, domain, and attributes are encoded as a query string as follows:

<event-signal-url>?field1=value1&field2=value2&field3=value3...

Multiple values can also be associated with a single field:

<event-signal-url>?field1=value1&field1=value2&field1=value3...

The keys and values must be URL encoded to encode reserved URL characters.

Event consumers must accept both POST and GET signals.

Success

Event consumers should return an HTTP response of 2xx to indicate that the event has been successfully signaled. Event consumers should not use response code 206 (partial content).

The response content is unspecified in this version of the API. Event consumers should specify as part of their event dictionary what responses they expect, if any.

Error Handling

An HTTP status code of 4xx (client error) or 5xx (server error) represents that the event consumer has failed to receive the event signalled.

Failed event signals with a status code of 500 (internal service error), 503 (service unavailable), or 504 (gateway timeout) may be retried by the event generator.  Event consumers may indicate a willingness to entertain retries using the HTTP Retry-After header on a 503 status. Event generators should respect the Retry-After header.

Event generators must not retry event signals for error status codes other than 500, 503, or 504.

Service Termination

Event consumers can indicate that they no longer wish to receive event signals by returning the HTTP status code 410 (gone). Event generators must respect the 410 code and must not continue signalling events to that consumer after receiving a 410 response.

Redirection

Event generators must respond correctly to redirection (HTTP status codes 3xx) responses from the consumer.

Event Subscription

An event consumer subscribes to events from a particular event generator by providing an event signal URL structured as described above. The URL might be registered via an API that the event generator provides or via a user interface into which a human copies the event signal URL. The event consumer must provide an interface where users can generate correctly formatted URLs with an appropriate, embedded entity identifier.  

Users generally control event consumers (whether stand-alone or multi-tenanted). Users configure event consumers by subscribing to event generators of interest. Event consumers must be designed with the events for particular event generators in mind.

The flow of a user manually subscribing an event consumer to an event generator manually is shown in the figure below.

The steps are:

  1. User logs into the event consumer
  2. User uses the supplied user interface to generate a event signal URL (<esl>).
  3. User copies the event signal URL (<esl>).
  4. User logs into the event generator.
  5. User stores the event signal URL at the generator using an interface.
  6. Event generator uses the event signal URL to signal event X in domain A (<esl>?_domain=A&_name=X).

The event generator now has a entity-specific URL that it can use to signal events to the event consumer. This process can be automated in various ways. For example, Web-annotation technologies can be used to allow users to configure a consumer installation without directly wrangling URLs.

Proposed Changes

Require X-EventedAPI header in consumer response

The presence of this header would indicate that the consumer URL does understand the Evented API. A consumer response of a 2xx series without this header indicates to the generator that this is not a valid consumer URL.

This, combined with the 410 error response, will serve to prevent attackers from using generators as an unwanted source of traffic to any URL.

Change _timestamp argument to XSD:Datetime

This would simplify parsing of timestamps. (Docs for XSD:Datetime)

Adopt Salmon Magic Signatures as an optional event signature

For systems processing events needing more security then SSL, allow for the use of Salmon Magic Signatures to sign event data.

Support Activity Stream Payloads

Activity streams give a semantic schema for activities (events). Activity streams support JSON encoding, so they make a fine payload for an event. Support for them would entail specifying that Activity stream payloads require using the application/activitystream+json mime-type so that event consumers know they’re being sent an activity stream encoded event.

When mapping activity streams structures, the _domain should be considered to be activitystreams, and the _name should be equal to the verb specified within the activity streams structure.

Change Event Name to Event Type, and _name to _type

Using Type would represent the real value of the name/type attribute of an event more clearly then Name.

The Future of Evented APIs

The first version of the Evented API specification is simple by design. The following are features that may be added in the future:

Salience Filters

Event generators may produce some events at a rate that is beyond the ability of the event consumer to process. Often information about specific events that the consumer cares about can allow the generator to filter the event stream before events are signaled. This is called event salience. Future versions of this document may specify a salience API.  

Event Subscription

While cutting and pasting URLs is the simplest way to support event subscription, future versions of this specification may include a standard subscription interface.

Event Dictionaries

There will likely be a need to specify a common format for event dictionaries.

Evented API Responses

The response content of the event consumer is currently unspecified. Future versions of this document may specify the format and encoding of the response.

Event Batching

Some event generators may find it useful to batch events and signal them all at once. Future versions of this document may specify the syntax and semantics of batched event signaling.

FAQ

The following questions and answers explain some of the nuances of Evented APIs. Please send additional questions to the authors listed at the top of this document.

How does an evented API compare with a streaming API?

Streaming APIs typically open a long-lived Web socket to transfer data more or less continuously. Streaming APIs, such as maintained by Twitter, are an efficient way to transfer lots of data. For sites with less volume, and particularly for consuming apps, a streaming API is not very efficient. Evented APIs are efficient and scale in a well-known, efficient manner. This makes evented API’s easier to implement, both for the generator and the consumer.

How does an evented API compare with Atom and PubSubHubub (PuSH)?

PuSH is designed for broadcast subscriptions, not one-to-one or one-to-a-few subscriptions such as we’re envisioning in the Evented API specification.

How does an evented API compare with Pushed Data?

Pushed Data, most popularly used by Flickr, is really a simplified form of PuSH. There isn’t anything wrong with this, but there is value in standardizing the approach.

How does an evented API compare to webhooks?

Webhooks are used for both events and RPCs, and (intentionally) lack constraint on how they are used. Evented APIs are only used for transferring events, and the API allows for a generalized way of transferring events with a common format.

Because of the similarities between webhooks and Evented APIs, you can support a limited form of an Evented API with a webhook by locking the webhook to a single event type.

Why use HTTP instead of XMPP or some other notification protocol?

There are several reasons:

  1. HTTP is available everywhere online. Very few firewalls block port 80.
  2. HTTP is available in almost every programming language, making the use of event-driven APIs over HTTP accessible.

How much data should be sent as attributes?

It is a good idea to send enough information as event attributes to prevent common API calls to retrieve additional data. Data that is particularly large in size, and not always of interest to the receiving party should be made available through an API. If event consumers must always make an API call to retrieve additional information, then that information should be included as an event attribute.

When should the event be sent?

The event should be sent immediately, but there is room for using background systems to send the events. Simpler systems can simply send the event in the same thread handling the original request. Most evented systems will operate fine if the event is sent within a minute of occurring, though faster transmission might be required for some systems. The exact timing is up to the generator, who has the best idea of what timing makes sense.

Copyright

© Copyright 2011 by Sam Curren and Phillip J. Windley.

Change History

Who

What

When

Phil Windley

Added activity stream proposal

10/19/2011