Paginating through Okta API where null URL query value throws invalid cursor error

Goal
I'm trying to use ReTool to fetch data from Okta. Querying Okta's API returns the first 200 results, after which the remaining records are accessed by paginating through the endpoint using a cursor.

Let's assume I am trying to get a list of all Okta users:

  1. An API query, getOktaUsers, is made to https://company.okta.com/api/v1/users with a response of:
{
  "response": {
    "data": [...],
    "headers": {
      "link": [
        "<https://company.okta.com/api/v1/users?limit=200>; rel=\"self\", <https://company.okta.com/api/v1/users?after=s0m3un1queUU1D&limit=200>; rel=\"next\""
      ],
    "status": 200,
    "statusText": "OK"
  }
}
  1. A JS query, getAllOktaUsers, is made to recursively call getOktaUsers, using after={cursor} as the cursor, per Returning all results for a cursor-based paginated API

Issue
Unlike most API endpoints that use URL queries which accept a null cursor, including the after key into the request with a null or undefined value results in Okta returning an error of Invalid cursor

Troubleshooting steps
I have attempted to set both the key and value as undefined variables, or override the URL with a variable, and setting them in additionalScope, but this doesn't appear to work, and I'm unable to debug to step through where the issue is.

getOktaUsers

getAllOktaUsers

const fetchAll = (cursor, data, hasMorePages) => {

  // Base case: there are no more pages
  if (!hasMorePages) return data
  
  // Wrap the query result in a promise
  return new Promise(resolve => {
    return getOktaUsers.trigger({
      additionalScope: {
        cursorKey : cursor ? "after" : null,
        cursorValue : cursor || null
      },
      onSuccess: queryResult => {
        const newResults = data.concat(queryResult.data);
        const link = queryResult.metadata?.headers?.link[0] || '';
        const nextCursor = link.match(/after=([^&>]+)/)?.[1] || null;
        const hasNextPage = link.includes('rel="next"');
        return resolve(fetchAll(nextCursor, newResults, hasNextPage))
      }
    })
  })
}

return fetchAll(null, [], true)

Any guidance on how I can circumvent this issue, or pointers to a better way to achieve this, would be much appreciated!

I figured out the issue.

getOktaUsers by default had its Transform Results set to return data, therefore making the metadata unavailable for getAllOktaUsers to access.

Explicitly writing return {data, metadata} seems to have resolved this.

const fetchAll = (cursor, data = []) => {
  if (cursor === null && data.length > 0) return data;
  
  return new Promise(resolve => {
    getOktaUsers.trigger({
      additionalScope: { 
        cursorKey : cursor ? "after" : null,
        cursorValue : cursor || null
      },      
      onSuccess: queryResult => {
        const newResults = data.concat(queryResult.data.map(({ id, profile }) => ({ id, ...profile })));
        const link = queryResult.metadata?.headers?.link.toString();
        console.log(newResults);
        const nextCursor = link.includes("next") ? link.match(/after=([^&>]+)/)?.[1] : null;

        return resolve(fetchAll(nextCursor, newResults))
      }
    })
  })
}

return fetchAll();
1 Like

Glad to hear it. :slightly_smiling_face: Thanks for sharing this update, @ThomasLu_EarthDaily!