I am attempting to pull a list subscriptions from stripe, convert the dates from UTC to date time, and then add them to my retool database. This all works fine, so long as the auto-pagination limit is 130 or less. When I pass a larger number of records (pagination set to 131-999 etc.), the upsert fails.
What's also strange is at 131, the records count is 140, which is unexpected. Bug? Am I doing something wrong?
Hey @Aaron_Hogan!
It looks like the default page size for the Stripe resource is 10, since Auto-paginate is pulling pages until it reaches the limit the result will be in multiples of 10. You can make this more granular by manually setting the limit query parameter but for large datasets this could create too many APIs called (and might be best set higher instead of lower).
I'm not sure why the bulk upsert is failing, would you mind sending over the complete error message? If it does have to do with the dataset being too large you might try batching your requests.
