Goal
I'm trying to use ReTool to fetch data from Okta. Querying Okta's API returns the first 200 results, after which the remaining records are accessed by paginating through the endpoint using a cursor.
Let's assume I am trying to get a list of all Okta users:
- An API query, getOktaUsers, is made to
https://company.okta.com/api/v1/users
with a response of:
{
"response": {
"data": [...],
"headers": {
"link": [
"<https://company.okta.com/api/v1/users?limit=200>; rel=\"self\", <https://company.okta.com/api/v1/users?after=s0m3un1queUU1D&limit=200>; rel=\"next\""
],
"status": 200,
"statusText": "OK"
}
}
- A JS query, getAllOktaUsers, is made to recursively call getOktaUsers, using
after={cursor}
as the cursor, per Returning all results for a cursor-based paginated API
Issue
Unlike most API endpoints that use URL queries which accept a null
cursor, including the after
key into the request with a null
or undefined value results in Okta returning an error of Invalid cursor
Troubleshooting steps
I have attempted to set both the key and value as undefined variables, or override the URL with a variable, and setting them in additionalScope
, but this doesn't appear to work, and I'm unable to debug to step through where the issue is.
getOktaUsers
getAllOktaUsers
const fetchAll = (cursor, data, hasMorePages) => {
// Base case: there are no more pages
if (!hasMorePages) return data
// Wrap the query result in a promise
return new Promise(resolve => {
return getOktaUsers.trigger({
additionalScope: {
cursorKey : cursor ? "after" : null,
cursorValue : cursor || null
},
onSuccess: queryResult => {
const newResults = data.concat(queryResult.data);
const link = queryResult.metadata?.headers?.link[0] || '';
const nextCursor = link.match(/after=([^&>]+)/)?.[1] || null;
const hasNextPage = link.includes('rel="next"');
return resolve(fetchAll(nextCursor, newResults, hasNextPage))
}
})
})
}
return fetchAll(null, [], true)
Any guidance on how I can circumvent this issue, or pointers to a better way to achieve this, would be much appreciated!