Managing the rate limit or quota of an API is one of the most tedious parts of any integration. Rate limits are often the reason a team decides to sync API data in the first place, because rate limits mean your read/write access to your data is throttled.
The rate limit of an API specifies how many requests you can make to the API in a given interval. Most APIs specify limits in intervals of one second or one minute, e.g. "you can make five requests per second." Some have daily request maximums, which adds complexity.
Fortunately, Sequin has controls to manage your rate limit for you. Sequin's rate limit consumption is predictable and configurable.
By default, Sequin uses a fraction of your total quota (what we consider a "conservative" starting point.) The amount of your quota we use is configurable on the Sequin console. This is Sequin's allotment of your quota.
When editing or creating a sync, you can change your allotment in the Rate limit section by dragging the slider left or right.
For some syncs, like Stripe and Close, that we sync using an
/events endpoint you can only configure the backfill rate limit. The allotment used during the sync is fixed at 2 requests per second.
By default, Sequin will use the following allotments for each platform:
|Stripe||Backfill: 50 req/s|
Sync: 2 req/s
|Close||Backfill: 20 req/s|
Sync: 2 req/s
How to determine Sequin's allotment
There's a tradeoff to consider when determining Sequin's allotment. The tradeoff is between your sync speed and leaving overhead in your quota for other API consumers.
The speed of your sync from the API to your database is assessed as the "max lag time" of any given change. That is: when a change is made in the API, what is the maximum amount of time you'd expect to wait before seeing that change reflected in your database?
Sequin's allotment will impact this max lag time. For example, let's say you're syncing five Salesforce objects with Postgres. Sequin will round-robin between these five objects, sending an API request to check for creates or updates for each object kind. If you allot Sequin one request every three seconds, it will take 15 seconds to round-robin through each object. So your max lag time here is 15 seconds. If you allotted Sequin half of this quota – one request every six seconds – your max lag time would double to 30 seconds.
So, why allot Sequin only half that quota? You might have other software you've written or other services you use that interface with the same API. If Sequin is sharing your API quota with other consumers, you don't want Sequin to use your full quota. You need to ensure that there's enough quota "left over" for the other consumers to use.
If Sequin will be the only consumer of the API you're syncing, you can safely allot Sequin your entire quota.
Sequin syncs data from APIs to your database and changes in your database back to the API. Changes in each direction share the same overall quota allotment.
If your application is write-heavy, consider configuring batching for your writes. This means Sequin will more effectively use your quota. You can tell Sequin to wait for a batch of
Y records for up to
X seconds. Without batching, if you make 100 changes in your database over the span of a minute, there's a chance this could trigger dozens of requests to the API. This would also slow down changes from flowing from the API to your database for a bit. With batching enabled, changes will be collected into batches that will be sent to the API together, meaning much fewer requests.
Only Salesforce and HubSpot sync bidirectionally. For our other platforms, data flows one-way from the API to your database. Bidirectional sync is coming to all platforms soon.
Questions and support
Our docs won't be able to cover everything, so if you have any issues, don't hesitate to reach out through Intercom on the bottom left. We'd love to help you get your rate limits configured just right.