API pagination; variable call in API

Hello!
Just a heads up, I am still very new to javascript and retool.
My goal is to use the Scopus API to retreive data. The issue being that I could only retrieve 25 results at once. I used the cursor parameter to get through the pagination. I found the following on Stackoverflow

The basic idea behind this pagination (that's what you're after) is:

  1. Run a query with unlimited number of results with cursor set to "*"
  2. Set start to 0 and get the first count results
  3. Set start to start+count+1 and get the next count results
  4. Repeat step 3 until all results are fetched

I found some code in this forum and tried to change it to fit the description above as follows:

const fetchAll = (start, records, count) => {
  // Base case: if no more records are returned
  if (records.length < count) return records; // Or some other stopping condition
  
  // Wrap the query result in a promise
  return new Promise(resolve => {
    return searchScopus.trigger({
      additionalScope: {
        cursor: "*",    // Use "*" as the cursor to start from the beginning
        start: start,   // Start from the current position (start)
        count: count    // How many results to fetch
      },
      onSuccess: queryResult => {
        // Add the records from this query to the accumulated results
        const newResults = records.concat(queryResult.records);
        
        // Update start for the next API call: start + count + 1
        const newStart = start + count + 1;
        
        // Recursively call fetchAll to continue fetching the next batch of records
        return resolve(fetchAll(newStart, newResults, count));
      }
    });
  });
};

// Start the process with start = 0 and specify count (e.g., 100 results per page)
return fetchAll(0, [], 25);  // Adjust count as needed

My initial thought was that Ill just put the cursor variable as input for the header:
image

yet I get an error. I assume I am misunderstanding something here. Can anyone help me understand?

Thank you in advance!

Hi @Katunga,

I've not used cursor based pagination before, but looking quickly at the Scopus docs, it looks like you should use cursor OR start/count. If you start with cursor: * then you should get back a cursor property in your results, which you use for each subsequent query until there is no more cursor property (maybe it's null, or false, docs aren't clear).

This parameter is used when a user wants to execute deep pagination searching (i.e. iterate to the end of a search result set).

Under normal circumstances, when using the 'start' parameter (results offset), access to the total result set is limited to a predefined maximum number of results. By using the cursor in place of the 'start' the user can iterate to the very end of the result set, with the restriction that results can only be accessed by iterating forward sequentially (there will be no 'prev' or 'last' links available).

This capability is initially accessed by sending a "*" in the first search request. Subsequent requests should submit the 'cursor/@next' value from each corresponding response as the 'cursor' value. The 'cursor/@next' value must be URL encoded by the client application. The navigation links ('next') can also be used to navigate to each succeeding search result entry and are URL encoded by default.

So I think you need to

  1. Run your first query with cursor as *
  2. Add the results to an array to persist
  3. Check for a cursor property in the response, if it exists
  4. If it does, repeat with new cursor value
  5. If not, return the results
let results = []
let cursor = '*'

const getData = async (cursor) => {
  // I prefer async/await, easier to follow IMO
  let data = await searchScopus.trigger( { additionalScope: { cursor } } )
  // Pull out the records, for clarity
  let newResults = data.records
  // Maybe there could be no results? Double check
  if(newResults.length >0) {
    return
  }
  results.push(...newResults)
  // Double check the path the where the @next prop is.
  let cursor = newResults.cursor['@next']
  // If there is a cursor, call self with new cursor value
  if(cursor) {
     return await getData(cursor)
  } else {
    return
  }
}
// This will call it the first time with the * for cursor (set above)
await getData(cursor)
// When the above finishes, return all the results
return results

This is roughly how it should work. Not sure of the paths to all the data in your responses, but it should keep calling the query with the next cursor until it's done, you don't need to worry about manually paginating your data.

Hello and thank you for your reply. I tried your code and unfortunately it throws some errors:



After I tinkered a bit but my lack of js skills hindered me to get anywhere.

I also then tried to do it manually for just the first two pages and after changing it a bit I didn't get any errors but the output is rather lacking:

let results = []
let cursor = '*'

let data = await searchScopus.trigger( { additionalScope: { cursor } } )

let newResults = data.records

results.push(newResults)
  cursor = '@next'
//cursor = newResults.cursor['@next']

data = await searchScopus.trigger( { additionalScope: { cursor } } )

newResults = data.records
//  
results.push(newResults)


// When the above finishes, return all the results
return results

image

Just in case it might help I also give yu a snippet of my json:

 "prism:url": "https://api.elsevier.com/content/abstract/scopus_id/85215119954",
        "dc:identifier": "S4",
        "eid": "19954",
        "dc:title": "Construction of an Inquiry Letter Sentiment Dictionary Using SO-PMI and Word2Vec for Sentiment Analysis",
        "dc:creator": "Wang W.",
        "prism:publicationName": "Tehnicki Vjesnik",
        "prism:issn": "13303651",
        "prism:eIssn": "18486339",
        "prism:volume": "32",
        "prism:issueIdentifier": "1",
        "prism:pageRange": "54-65",
        "prism:coverDate": "2025-12-31",
        "prism:coverDisplayDate": "31 December 2025",
        "prism:doi": "10.17559/TV-20230629000774",
        "dc:description": "S",
        "citedby-count": "0",
        "affiliation": [
          {
            "@_fa": "true",
            "affiliation-url": "https://api.elsevier.com/content/affiliation/affiliation_id/60018273",
            "afid": "60018273",
            "affilname": "University of Science and Technology Beijing",
            "affiliation-city": "Beijing",
            "affiliation-country": "China"
          }
        ],
        "prism:aggregationType": "Journal",
        "subtype": "ar",
        "subtypeDescription": "Article",
        "author-count": {
          "@limit": "100",
          "@total": "4",
          "$": "4"
        },
        "author": [
          {
            "@_fa": "true",
            "@seq": "1",
            "author-url": "https://api.elsevier.com/content/author/author_id/57221427227",
            "authid": "57221427227",
            "authname": "Wang W.",
            "surname": "Wang",
            "given-name": "Wei",
            "initials": "W.",
            "afid": [
              {
                "@_fa": "true",
                "$": "60018273"
              }
            ]
          },
          {
            "@_fa": "true",
            "@seq": "2",
            "author-url": "https://api.elsevier.com/content/author/author_id/36095296100",
            "authid": "36095296100",
            "authname": "Wei G.",
            "surname": "Wei",
            "given-name": "Guiying",
            "initials": "G.",
            "afid": [
              {
                "@_fa": "true",
                "$": "60018273"
              }
            ]
          },
          {
            "@_fa": "true",
            "@seq": "3",
            "author-url": "https://api.elsevier.com/content/author/author_id/7407182341",
            "authid": "7407182341",
            "authname": "Wu S.",
            "surname": "Wu",
            "given-name": "Sen",
            "initials": "S.",
            "afid": [
              {
                "@_fa": "true",
                "$": "60018273"
              }
            ]
          },
          {
            "@_fa": "true",
            "@seq": "4",
            "author-url": "",
            "authid": "57221124316",
            "authname": "He H.",
            "surname": "He",
            "given-name": "Huixia",
            "initials": "H.",
            "afid": [
              {
                "@_fa": "true",
                "$": "60018273"
              }
            ]
          }
        ],
        "authkeywords": "",
        "source-id": "14569",
        "fund-acr": "NSFC",
        "fund-no": "22 &amp; ZD153.",
        "fund-sponsor": "",
        "openaccess": "0",
        "openaccessFlag": false
      },
      {
        "@_fa": "true",
        "link": [
          {
            "@_fa": "true",
            "@ref": "self",
            "@href": "https://api.elsevier.com/content/abstract/scopus_id/85215112502"
          },
          {
            "@_fa": "true",
            "@ref": "author-affiliation",
            "@href": "https://api.elsevier.com/content/abstract/scopus_id/85215112502?field=author,affiliation"
          },
          {
            "@_fa": "true",
            "@ref": "scopus",
            "@href": "https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85215112502&origin=inward"
          },
          {
            "@_fa": "true",
            "@ref": "scopus-citedby",
            "@href": "https://www.scopus.com/inward/citedby.uri?partnerID=HzOxMe3b&scp=85215112502&origin=inward"
          }
        ],

I tried cleaning the json of any potential sensitive data.

I also noticed at the beginning of the json it says:

"search-results": {
    "opensearch:totalResults": "3365020",
    "opensearch:itemsPerPage": "25",

Is it rather retool which doesnt let me display more than 25 items?
I did assume it was the pagination.

Once again thank you in advance!

Hi @Katunga,

Thanks for sharing the JSON, that's helpful. Can you look through it and see if there is any mention of cursor or @next? You'll need to know the path to those values in order to pull it out and use it in your function calls.

Also, my code wasn't set up to be a drop in, you'll need to adjust it a bit for both the path to the cursor/@next property as well as the root array of your records. Oftentimes the data comes back something like this:

{ 
  page: 1,
  cursor: {
    @next: 4234234
  },
  records: [
    {
      id: 1,
      name: "One"
    },
    {
      id: 2,
      name: "two"
    }
  ],
  totalResults: 213412341
}

So from each API call, you'd get data like that back. Assuming you got the result back in a variable called data (as in my example), then cursor would be data.cursor['@next'], and since you wouldn't want to save the cursor and page and totalResults count for each count, you'd only want the array called records to be saved into the results array we created at the start, which is why let newResults = data.records would just pull that section of data out of the everything returned, and then results.push(...newResults) would take those records and add them to the results array, so they would keep getting added until the cursor is exhausted. Then at the end, it would return the results array, which should be all the records, as if they all came in from one API call.

Ahh okay thank you for the explanation!
I did find the @next part:

{
  "search-results": {
    "opensearch:totalResults": "3371055",
    "opensearch:itemsPerPage": "25",
    "opensearch:Query": {
      "@role": "request",
      "@searchTerms": "machine learning ",
      "@startPage": ""
    },
    "cursor": {
      "@current": "*",
      "@next": "..."
    },
    "link": [
      {

I replaced that part with "..." just in case, Im not sure whether its sensitive or not.
So what I need to do now is do another API call and this time set the cursor-header to whatever is returned here, right?

Thank you again for your help!

I used chatgpt to help me with the code and troubleshooting.

let results = [];  // Array to hold all the results
let cursor = '*';  // Start with the first cursor

// Assuming 'searchTerm' is the variable tied to your search input field in Retool
let searchTerm =  rt_publ_searchfield.value ;  // Replace with actual reference to your search field

while (cursor) {  // Loop until there's no next cursor
  try {
    // Fetch data from the API using the current cursor and dynamic search term
    let data = await searchScopus.trigger({
      additionalScope: {
        cursor: cursor,       // Start with the initial cursor value
        query: searchTerm,    // Pass the search term to the query
        count: 25             // Number of results per page
      }
    });

    // Log the entire response object for debugging
    console.log('Raw API Response:', data);

    // If the response is undefined or empty, check why it's happening
    if (!data) {
      console.error('API returned undefined data.');
      break;  // Exit if no data is returned
    }

    // Validate that 'search-results' exists in the response
    if (data['search-results']) {
      console.log('search-results:', data['search-results']);
    } else {
      console.error('search-results not found in the response.');
      break;
    }

    let searchResults = data['search-results'];

    // Check if the 'entry' field is present and contains results
    if (searchResults.entry) {
      let newResults = searchResults.entry;

      // Add the new results to the results array
      results.push(...newResults);

      // Check if there's a next cursor
      if (searchResults.cursor && searchResults.cursor['@next']) {
        // Update cursor to continue pagination (if available)
        cursor = searchResults.cursor['@next'];
        console.log('Next cursor:', cursor);  // Log the next cursor to debug
      } else {
        console.log('No more results. Pagination completed.');
        cursor = null;  // Exit the loop when no next cursor is found
      }
    } else {
      console.error("No entries found in 'search-results'.");
      cursor = null;  // Exit if no entries were found
    }
  } catch (error) {
    console.error('Error fetching data from the API:', error);
    cursor = null;  // Exit on error
  }
}

// Return all the results
return results;

And I get the msg "API returned undefined data".
I did replace with the actual reference in the searchTerm variable.

When I used

let data = await searchScopus.trigger({ additionalScope: { cursor } });

console.log(data);

it returned an object.
I am a bit confused about why it doesn't return an object now.

Once again, thank you in advance!

Can you see the results of this part in your console when the function runs?

    // Log the entire response object for debugging
    console.log('Raw API Response:', data);

What does it log?

It logs undefined.
from what I have gatehered there is an issue when submittig the cursor via

      additionalScope: {
        cursor: cursor,       // Start with the initial cursor value
        query: searchTerm,    // Pass the search term to the query
        count: 25             // Number of results per page
      }```
when its not set to be a global variable. 

it did return something when I set cursor to a global one but the setValue function I need for assigning a value only updates the variable at the end, so I run into an infinite loop.

I have tinkered a bit and found a way to get data with the js code.
The issue I am now having is that setValue works asynchronously.
I tried to use await before and I found a thread which states I should use .then. Unfortunately I cannot seem to figure out how to make it work.
I also tried now to use await query.trigger() but this also doesn't seem to update my cursor state.

Here is my code:

let results = [];
let searchTerm = rt_publ_searchfield.value;  // Your search term
let cursorValue = "*"; // Start with the initial cursor
cursor.setValue(cursorValue); // Set initial cursor

let pageCount = 0;

const fetchPage = async () => {
  // First API request with the initial cursor value
  let data = await searchScopus.trigger({
    additionalScope: {
      cursor: cursor,   // Use the cursor Retool variable here
      query: searchTerm,      // Pass the search term
      count: 25               // Set items per page
    }
  });
 console.log(cursor.value)
  //console.log('First API Response:', JSON.stringify(data, null, 2));

  // Extract the results
  let newResults = data['search-results'];
  results.push(newResults);

  // Get the next cursor value from the response and update the cursor Retool variable
  let nextCursor = newResults.cursor['@next'];
  
  if (nextCursor != cursor.value) {
    // Update the cursor Retool variable
    //cursor.setValue(nextCursor); // Set the updated cursor value for the next iteration
     await query9.trigger()
    // Set a delay before triggering the next request
    setTimeout(fetchPage, 1000); // Delay next request by 1 second (or adjust as needed)

    
  }
  else {
    // No more pages, so return the results
    console.log('All pages fetched');
    return results;
  }
};

// Start the recursive pagination fetch
fetchPage();

And query9 is just:

let searchTerm = rt_publ_searchfield.value

let data = await searchScopus.trigger({
    additionalScope: {
      cursor: cursor,   // Use the cursor Retool variable here
      query: searchTerm,      // Pass the search term
      count: 25               // Set items per page
    }
  });

let newResults = data['search-results'];
let nextCursor = newResults.cursor['@next'];
await cursor.setValue(nextCursor)```

I am stuggling a lot here, so thank you for any tips and tricks you can give me!