Core concepts
A rundown of Sequin from top to bottom:
- Every API produces one or more collections that Sequin can consume.
- To add an API’s data to Sequin, you select which collections you want to sync.
- Sequin provisions a sync cluster which first extracts all data from the API, then monitors it for changes.
- As changes occur, Sequin propagates those changes to your account’s record stream.
- You can consume these streams directly via Sequin’s Stream APIs (HTTP).
- Or you can use one of Sequin’s consumers to forward the data to a destination like Postgres or Kafka.
- And to write back to the API, you can use the Record Mutation API.
All this and more can be managed in the console or via the Management API.
Collections
A collection is a class of records in the API. In RESTful APIs, a collection usually corresponds to one API endpoint. For example, Stripe has a subscription
collection located at /v1/subscriptions
and Salesforce has a contact
collection that you can access at /services/data/v51.0/sobjects/Contact
.
Sequin uses unique IDs to refer to each collection. The ID is a combination of the API provider’s name and the collection name. Some examples:
- Stripe subscriptions ->
stripe:subscription
- Salesforce contacts ->
salesforce:contact
- HubSpot pipelines ->
hubspot:pipeline
- Shopify products ->
shopify:product
- GitHub repositories ->
github:repository
Syncs
To generate a stream of API data, you create a sync. When you create a sync, Sequin first extracts all data from the API. Then, Sequin monitors the API for changes and emits events when changes occur.
Data consistency is a huge priority for Sequin, so the backbone of most syncs is polling. Sequin augments polling with other strategies (like listening to API webhooks) where applicable to offer the fastest possible sync.
Backfills
When you first setup your sync, Sequin will backfill your event stream with all historical data from the API. Sequin will also kickoff a backfill process whenever you add a new collection to your sync.
The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you’ve allocated for Sequin.
Backfills involving millions of records can take several hours. You can view the status of your backfill at any time on the Sequin console:
Speed
Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of an API’s total quota. The amount of your quota we use is configurable in the Sequin console or via the Management API.
The time it takes Sequin to detect changes in the API and push to your streams is called the “max lag time” of any given change. Max lag time is subject to a few factors:
- the API you’re syncing from
- the collections you’re syncing from that API
- the rate limit you set Sequin to consume
- whether the collection supports webhooks or other fast means of detecting changes
For most syncs and most collections, the max lag time is rarely above one minute.
Streams
Sequin provisions two streams for your account. All data from your syncs is pushed to these two streams.
The record stream contains a stream of all records across syncs, while the event stream contains all the recent create, update, and delete events across syncs.
Record stream
The record stream is a stream of all API records across all APIs you are syncing. A record contains the current state of an object in the API.
The record stream contains each record only once:
- When Sequin detects a record insert in the API, it appends the record to the stream.
- When Sequin detects a record update in the API, it pops the old record and appends the updated record to the stream.
- When Sequin detects a record delete in the API, it pops the old record and appends the updated record to the stream with the
deleted
property set totrue
.
You’ll want to use records whenever a consumer just needs the most recent state of records. For example, a sync to a data store like Redis or Postgres.
Records
Records have the following shape:
Record{
sync_id: string;
collection_id: string;
upstream_id: string;
upstream_created_at: string;
upstream_updated_at: string;
data: object;
deleted: boolean;
}
Here’s an example of a Record
:
{
"sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
"collection_id" : "salesforce:task",
"upstream_id": "0018b000021BldZAAS",
"upstream_created_at": "2024-11-10T18:39:00.070453Z",
"upstream_updated_at": "2024-11-10T18:39:00.070453Z",
"data": {
"activity_date": "2023-09-12",
"description" : "task description [ … ] ",
// …
},
"deleted": false
}
A Record
is a JSON object that contains the following properties:
sync_id
: The ID of the sync that generated this event.collection_id
: The ID of the collection that generated this event (e.g.stripe:subscription
orsalesforce:contact
).upstream_id
: The ID of the record in the API.upstream_created_at
: The time the record was created in the API, in ISO 8601.upstream_updated_at
: The time the record was last updated in the API, in ISO 8601.data
: All the fields of the API object.deleted
: Whether the record has been deleted in the API.
Sequin may add fields to records at any time. If you’re using a JSON validator to parse records, you should allow for unknown fields.
Event stream
An event represents a single change to a record. Every time Sequin detects that a record was changed in the API, it creates an event and appends it to the event stream. The event stream is a log of all changes to all records across all syncs that happened in the last 7 days.
You’ll want to use the event stream when you need to know how a record has changed over time. For example, you might want to trigger a side effect when a specific field has changed or kick off a workflow when a record is created.
Both events and records share the same data
property, which contains the fields of the API object. Here’s an example of an inserted
event:
{
"sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
"collection_id" : "salesforce:task",
"upstream_id": "0018b000021BldZAAS",
"upstream_updated_at": "2024-11-10T18:39:00.070453Z",
"data": {
"activity_date": "2023-09-12",
"description" : "task description [ … ] ",
// …
},
"event": "inserted",
"event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
"event_inserted_at": "2024-11-10T18:39:00.070453Z"
}
For any given record, Sequin’s event stream is strictly ordered. So, if a record was updated twice in the API, the event stream will contain two updated
events in the order they occurred.
Note that while Sequin will always emit an event for the final state of a record after an update, Sequin can “miss” ephemeral state changes. For example, if a Salesforce opportunity switches from Prospecting
to Closed Won
but then quickly back to Prospecting
before Sequin picks it up, Sequin may not emit an event for the Closed Won
state.
Sequin’s ability to detect ephemeral state changes depends on the API. For example, it’s easier to pick up ephemeral state changes in APIs that support webhooks.
We think this is rarely a problem, but if you have a use case where this is an issue, please let us know.
Events
Event{
sync_id: string;
collection_id: string;
upstream_id: string;
upstream_updated_at: string;
data: object;
event: "inserted" | "updated" | "deleted";
event_id: string;
event_inserted_at: string;
// Only present on "updated" events
previous_data?: object;
}
The event is a JSON object that contains the following properties:
sync_id
: The ID of the sync that generated this event.collection_id
: The ID of the collection that generated this event (e.g.stripe:subscription
orsalesforce:contact
).upstream_id
: The ID of the record in the API.upstream_updated_at
: The time the record was last updated in the API, in ISO 8601.data
: All the fields of the API object. Fordeleted
events, this is the entire record before it was deleted. This matches thedata
attribute found on records.event
: Eithercreated
,updated
, ordeleted
.event_id
: A Sequin-generated ID for this event. This ID is universally unique.event_inserted_at
: The time the event was created (and inserted into the event stream), in ISO 8601.previous_data
(updated
events only): contains the prior values of all columns that were changed. (See below.)
Here’s an example of an created
event:
{
"sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
"collection_id" : "salesforce:task",
"upstream_id": "0018b000021BldZAAS",
"upstream_updated_at": "2024-11-10T18:39:00.070453Z",
"data": {
"activity_date": "2023-09-12",
"description" : "task description [ … ] ",
// …
},
"event": "created",
"event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
"event_inserted_at": "2024-11-10T18:39:00.070453Z"
}
And an example of a deleted
event:
{
"sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
"collection_id" : "salesforce:task",
"upstream_id": "0018b000021BldZAAS",
"upstream_updated_at": "2024-11-10T18:39:00.070453Z",
"data": {
"activity_date": "2023-09-12",
"description" : "task description [ … ] ",
// …
},
"event": "deleted",
"event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
"event_inserted_at": "2024-11-10T18:39:00.070453Z"
}
And finally an updated
event:
{
"sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
"collection_id" : "salesforce:task",
"upstream_id": "0018b000021BldZAAS",
"upstream_updated_at": "2024-11-10T18:39:00.070453Z",
"data": {
"activity_date": "2023-09-12",
"description" : "Some description of the task [...]",
"completed_at": "2023-09-12T18:39:00.070453Z",
// …
},
"previous_data": {
"activity_date": "2023-08-30",
"description": "Some description of the task [...]",
"completed_at": null
},
"event": "updated",
"event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
"event_inserted_at": "2024-11-10T18:39:00.070453Z"
}
Unlike created and deleted events, updated events have a previous_data
property. previous_data
contains the prior values of all columns that were changed. In the example above, activity_date
was changed from "2023-08-30"
to "2023-09-12"
. Likewise, completed_at
was previously null
but now has a value. Because description
was not changed, it does not appear in the previous_data
object.
Stream APIs
You can use Sequin’s Stream APIs to consume from your record or event streams. The Stream APIs are a set of HTTP interfaces to your streams. You can paginate through all records across all your syncs, or you can filter to just the collections or events you need.
Learn more about the Stream APIs.
Consumers
Sequin offers a number of consumers that you can use to forward your data to a target. For example, you can use the Postgres consumer to forward your data to a Postgres database or the Kafka consumer to forward your data to Kafka.
Consumers make it easy to build on APIs using the data platforms you already know and use.
Targets
A target is infrastructure in your application environment, such as a Postgres database or a Kafka cluster, that you register with Sequin.
Sequin’s consumers forward data from either the event stream or the record stream to a target.
Mutations
Sequin makes it easy to write changes back to upstream APIs. Most collections that Sequin syncs are mutable. To create a record in that collection or mutate an existing one, you can use Sequin’s Record Mutation API.
Management API
You can use Sequin’s Management API to develop applications that create and manage Sequin syncs. You can automate everything that you can do in the Sequin console. This includes creating, updating, and deleting syncs on behalf of your users using their provided credentials, as well as turning on and off consumers.
Was this page helpful?