API pagination, setValue for global variable

Hello again!
I have labeled this as a bug although I might have misunderstood something.
I am still trying to get data with the scopus API. I can only access 25 entries at a time so what I can use is pagination to get the rest of the entries. What I need to do for that is send the cursor parameter set to "*" retrieve the @next and submit this again until theres no pages left.

So far so good.
The problem I'm now facing is that when I use let cursor = "*" the API call doesnt recognize this as valid and I get an error:

let cursorValue = "*";  // Initial cursor

// Trigger the first query to fetch the data
await searchScopus.trigger({
  additionalScope: {
    cursor: cursorValue,     // Use the initial cursor
    query: 'machine learning',  // Your search term
    count: 25          // Number of results per page
  }
});


So I set a global variable called cursor:
image

this I cannot update by simply using cursor = "*" so I used the setValue function:

let results = [];
let searchTerm = '';  // Your search term
let cursorValue = "*"; // Start with the initial cursor
cursor.setValue(cursorValue); // Set initial cursor
console.log(cursor.value)

let pageCount = 0;

const fetchPage = async () => {
  // First API request with the initial cursor value
  let data = await searchScopus.trigger({
    additionalScope: {
      cursor: cursor,   // Use the cursor Retool variable here
      query: searchTerm,      // Pass the search term
      count: 25               // Set items per page
    }
  });

  //console.log('First API Response:', JSON.stringify(data, null, 2));

  // Extract the results
  let newResults = data['search-results'];
  results.push(newResults);

  // Get the next cursor value from the response and update the cursor Retool variable
  let nextCursor = newResults.cursor['@next'];
  
  if (nextCursor) {
    // Update the cursor Retool variable
    cursor.setValue(nextCursor); // Set the updated cursor value for the next iteration
    
    // Set a delay before triggering the next request
    setTimeout(fetchPage, 1000); // Delay next request by 1 second (or adjust as needed)
  }
  else {
    // No more pages, so return the results
    console.log('All pages fetched');
    return results;
  }
};

// Start the recursive pagination fetch
fetchPage();

here is my whole code.
As you can see, I tried to use the function somewhat recursively. I also the setTimeout function to try to update the cursor variable. Here is my issue, I want to update the cursor rather frequently to get to the next page unfortunately setValue seemingly only updates after the code has been executed, therefore this recursive code is caught in an infinite loop. As I said Im rather new to both retool as well as javascript hence I am not sure if this is intended or not.
Edit: I have found this but it doesn't work for me or at least I am not sure how to use it properly:

let cursorValue = "*"; // Start with the initial cursor
cursor.setValue(cursorValue).then((d)=>console.log(cursorValue)); // Set initial cursor
console.log("Cursor", cursor.value);

I'd appreciate clarification and/or workarounds!

Thank you in advance!

hi,

Here's my code for the same idea. I have an API with limit and i want to have the complete data. I have just one parameter "page" to go to the next instance.

Hope that can help :

let allUsers = []
let returnUsers = []
const toto = await getUsers.trigger({
      additionalScope: {
        page:0
      }})

const nbpages = Math.round(toto.total/50)
allUsers.push(toto.users)



for (let i=1;i<=nbpages;i++) {
  const tata = await getUsers.trigger({
        additionalScope: {
          page:i
        }})
  allUsers.push(tata.users)  

}

returnUsers = allUsers.flat()

return returnUsers

Thank you for the reply!
I managed to work it out: I just had to use the await command as well as check the box in the advanced section saying "Keep variable references inside the query in sync with your app". I have only done so separately until this point.

1 Like