Loop Unknown times in Workflow

So there's a few other posts about looping to call a Rest API but none of them seem to capture my use case where I only know I've reached the end of the pages to call, when I get less than 100 results.

I'm sure I could do this in a self calling iterative workflow but:

  1. I shouldn't have to
  2. It will have a huge impact on the billable runs for zero value add.
  3. The below should work, but isn't...

I currently have the following working in an App:

var cursor = 1
const fetchAll = (page, records) => {
  if (page == null) return records
  return new Promise(resolve => {
    return Invoices.trigger({
      additionalScope: {
        page 
      },
      onSuccess: queryResult => {
        var newRecords = queryResult.Invoices
        var num = newRecords.length
        //console.log(queryResult)
        console.log(num)
        if (num == 100) {
        //  console.log
          cursor++
        } else {cursor = null}
       // console.log(cursor)
        const newResults = records.concat(newRecords)
        return resolve(fetchAll(cursor, newResults))
      }
    })
  })
}

return fetchAll(cursor,[])

Where Invoices is a Rest API resource call configured with the following endpoint Invoices?ContactIDs={{ContactID.data.Contacts[0].ContactID}}&summaryOnly=true&createdByMyApp=true&page={{page}}

So I have tried to replicate this in a Workflow using a loop block with the "same" code in a loop block:

var cursor = 1
const fetchAll = (page, records) => {
  if (page == null) return records
  return new Promise(resolve => {
    return Invoices_lambda.trigger({
      additionalScope: {
        page 
      },
      onSuccess: queryResult => {
        var newRecords = queryResult.Invoices
        var num = newRecords.length
        //console.log(queryResult)
        //console.log(num)
        if (num == 100) {
          //console.log
          cursor++
        } else {cursor = null}
        //console.log(cursor)
        const newResults = records.concat(newRecords)
        return resolve(fetchAll(cursor, newResults))
      }
    })
  })
}

return fetchAll(cursor,[])

where the loop block is called Invoices and the Loop Lambda is configured as the same rest api call with same params etc as the App was

The code block just errors with {"data":null,"metadata":{},"error":"Internal Error running a block: Error: An internal server error occurred"}

Any ideas please? :pray:

Have you tried just running the loop a few times? It could be a rate-limit you're hitting by blasting out a ton of requests to the same endpoint all at once

1 Like

I like your thinking, but unfortunately not. I swapped out cursor++ for cursor=null to force just a single loop but no joy.

I suspect this is some undocumented restriction in the Retool Loop block...

I randomly came across this post that might be related to your problem.