How Sequin works

Continuous sync

When you connect a source like Salesforce to your database, Sequin provisions a continuous sync. Sequin's sync is always running and checking the source API for changes.

Speed

Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of your total quota. The amount of your quota we use is configurable in the Sequin console.

The speed of your sync from the API to your database is the "max lag time" of any given change. That is: when your data changes in the API, what is the maximum amount of time you'd expect to wait before seeing that change reflected in your database?

Max lag time is subject to a few factors:

• the API you're syncing with
• the collections you're syncing from that API
• the rate limit you set Sequin to consume

For most syncs and most collections, the max lag time is rarely above one minute. For example, by default we make one request every three seconds for Salesforce syncs. Assume you kept this default and you're syncing five Salesforce collections. It will take Sequin about 15 seconds to check each collection endpoint for changes. So, the max lag time for this hypothetical sync is 15 seconds.

There are two notable exceptions.

The first is Airtable. For Airtable, the lag time of the sync depends on (1) how big your base is and (2) how much of your base's API quota we're using. The Sequin console shows the lag time of your Airtable database. By default, we use 3 out of 5 of your allotted requests per second, but you can configure this in your console. (More about how our sync works for Airtable on our blog.)

Similarly, for HubSpot, all objects and custom objects sync near-immediately. The exception is HubSpot associations. The max lag time for associations is roughly dependent on the number of associations you're syncing.

Learn more about rate limits.

Consistency

Sequin's sync relies primarily on polling the API. We use this strategy because it gives us the strongest consistency guarantee.

We're rolling out support for webhooks for the fastest propagation where supported. But all syncs are built on polling to ensure we never miss a create, update, or delete.

Backfills

When you first setup your sync, Sequin will backfill your tables with all historical data from the API. Sequin will also kickoff a backfill process whenever you add a new table or column or change a column type (see Changing tables or columns).

The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you've allocated for Sequin.

Backfills involving millions of records can take several hours. You can view the status of your backfill at any time on the Sequin UI:

Additionally, when your initial backfill completes, Sequin will email you letting you know that your database is ready to go.

Sequin-owned tables

Before Sequin begins syncing API data, we generate the destination tables and columns in your database. You can select which tables and columns to sync, as well as rename columns to better suit the needs of your application.

Indexes

You're encouraged to add indexes to your Sequin tables according to your app's needs. Just make sure you don't add unique indexes (see the note about constraints below).

Avoid constraints or modifications

It's important that you never modify your Sequin tables or columns directly. Any modifications to your Sequin tables or columns should happen via the Sequin console.

Additionally, avoid adding any constraints to your tables. These include things like foreign keys, non-null constraints, or unique indexes. Constraints added outside of Sequin can break Sequin's upsert and delete process, and break your sync.

Changing tables or columns

You can change the tables and columns you're syncing from the API at any time in the Sequin console. When you add a new table or column, Sequin will kickoff a backfill process to populate your new schema with data.

Additionally, for some sources, schema changes can originate from the API. For example, Airtable, HubSpot, and Salesforce all offer the ability to customize your data model. In Airtable, you can add and remove tables and modify the columns in those tables. In HubSpot and Salesforce, you can add, remove, and modify custom objects.

Sequin polls the source API for schema changes. When we detect a schema change that impacts your sync, we'll automatically propagate that migration to your database. For example, if you drop a field from a custom object, we'll drop the corresponding column in your table.

Proxy

Sequin provides a Proxy for writes. Writes through the Proxy will commit changes to both the API and your database at the same time. This enables workflows that perform reads-after-writes. A read-after-write is a situation where you want to query your database after performing a write and need the changes from your write to be there.

Read more about writes through our Proxy.