Awaiting query to finish

Queries and Resources
1

I am struggling with a piece of code trying to load data from an external datasource via API-call to a Retool-database. The code looks as follows:

queryExpensesFullSync.trigger({    
  onSuccess: function (data) {
    data.expenses.forEach(row => {
      updateExpensesFullSync.trigger({
      additionalScope : {
        expense_id: row.id,
        spent_date: row.spent_date
      }});
      //console.log("Row.id = " + row.id);
    }
  );
  },
  onFailure: function (error) {
    console.log(error);
  },
});
  • queryExpensesFullSync: External API-Call getting a substantial amount of data
  • updateExpensesFullSync. SQL Query inserting/updating new/existing records in a Retool database

The operation seems to fail because there are too many parallel calls of updateExpensesFullSync.trigger (if I reduce the number of records to be processed by changing the API call, it works just fine).

Now I wonder how I can make sure that only one call is made at a time. I fiddled around with Javascript Promises (also looking at this: Await for a query to finish) but don't get it to work. Any help is highly appreciated.

2

Hey mbrobst!

I can share some of my experience and things that I found that helped me.

  1. Batching request into smaller size requests (especially when dealing with large sets of data i.e. 100k)

  2. Using Workflows instead of apps/dashboards in sync what I mentioned in 1.

  3. Modifying forEach query to include await (using promises) - example of the syntax used in Workflows

const results = [];
for (const [index, value] of queryExpensesFullSynct.data.entries()) {
  const result = await updateExpensesFullSync.trigger();
  results.push(result)
}
return await Promise.all(results)

Hope this helps!

1 Like
3

Hi Stefan,
thanks for your valuable input!
What do you mean by using Workflows?

4

Hey mprobst - there is a beta for Retool Workflows you can check out the details.

1 Like