Loop Unknown times in Workflow

Workflows
, ,
1

So there's a few other posts about looping to call a Rest API but none of them seem to capture my use case where I only know I've reached the end of the pages to call, when I get less than 100 results.

I'm sure I could do this in a self calling iterative workflow but:

  1. I shouldn't have to
  2. It will have a huge impact on the billable runs for zero value add.
  3. The below should work, but isn't...

I currently have the following working in an App:

var cursor = 1
const fetchAll = (page, records) => {
  if (page == null) return records
  return new Promise(resolve => {
    return Invoices.trigger({
      additionalScope: {
        page 
      },
      onSuccess: queryResult => {
        var newRecords = queryResult.Invoices
        var num = newRecords.length
        //console.log(queryResult)
        console.log(num)
        if (num == 100) {
        //  console.log
          cursor++
        } else {cursor = null}
       // console.log(cursor)
        const newResults = records.concat(newRecords)
        return resolve(fetchAll(cursor, newResults))
      }
    })
  })
}

return fetchAll(cursor,[])

Where Invoices is a Rest API resource call configured with the following endpoint Invoices?ContactIDs={{ContactID.data.Contacts[0].ContactID}}&summaryOnly=true&createdByMyApp=true&page={{page}}

So I have tried to replicate this in a Workflow using a loop block with the "same" code in a loop block:

var cursor = 1
const fetchAll = (page, records) => {
  if (page == null) return records
  return new Promise(resolve => {
    return Invoices_lambda.trigger({
      additionalScope: {
        page 
      },
      onSuccess: queryResult => {
        var newRecords = queryResult.Invoices
        var num = newRecords.length
        //console.log(queryResult)
        //console.log(num)
        if (num == 100) {
          //console.log
          cursor++
        } else {cursor = null}
        //console.log(cursor)
        const newResults = records.concat(newRecords)
        return resolve(fetchAll(cursor, newResults))
      }
    })
  })
}

return fetchAll(cursor,[])

where the loop block is called Invoices and the Loop Lambda is configured as the same rest api call with same params etc as the App was

The code block just errors with {"data":null,"metadata":{},"error":"Internal Error running a block: Error: An internal server error occurred"}

Any ideas please? :pray:

2

Have you tried just running the loop a few times? It could be a rate-limit you're hitting by blasting out a ton of requests to the same endpoint all at once

1 Like
3

I like your thinking, but unfortunately not. I swapped out cursor++ for cursor=null to force just a single loop but no joy.

I suspect this is some undocumented restriction in the Retool Loop block...

4

I randomly came across this post that might be related to your problem.

5

So I think this is specific to the retool Loop block. I have worked around it by using a Function for the REST API Lambda and the loop just in a standard JS block

var contact = ContactID.data.Contacts[0].ContactID
var since = XeroDate.data


var cursor = 1
var retry = 0

 async function fetchAll(page, records) {
  if (page == null) return records
  var res = await InvoicesFunction(page,contact,since)

 if (res.metadata.status == "200") {
    var newRecords = res.data.Invoices
    var num = newRecords.length
    if (num == 100) {
          console.log
          cursor++
        } else {cursor = null}
    const newResults = records.concat(newRecords)
    return fetchAll(cursor, newResults)
  }  else if (retry < 10) {
    retry++
    return fetchAll(cursor, records)
  }
}
return await fetchAll(cursor,[]) //.filter(x=> x.InvoiceNumber.substring(0,2)=="S-")return ContactID.data```

Seems to work from the IDE, will see how it gets on tomorrow running on its schedule.  If I don't report back then all good.
1 Like