Issue with Batch Processing on GoogleMap API in Loop Queries

[Paul_Merz]

Dear community,

I have spent a full day on this, I'm just about to

Any help would be greatly appreciated !

• Goal:

  • I am trying to process a list of locations in batches using JavaScript within Retool. My goal is to make batched API calls to an external service (e.g., Google Maps API) for each group of locations in order to reduce the number of goals and the related API fees. I expect each batch to process a subset of the total locations and then aggregate the results. Do you know if this makes any sense or if making one call per row would result in the same cost structure ?

• Steps:

  • I have created a JavaScript function that divides the locations from the array into batches and then triggers an asynchronous API call for each batch. I've used Promise.all to handle the asynchronous nature of these calls. However, I am encountering an issue where only the last item in the locations array is processed and returned, instead of the entire batch.

• Details:

  • Here's the relevant part of my JavaScript code in Retool:

async function processInBatches(batchSize) {
    const locations = getLocations(); //returns an array ["address1","...","address220"]
    const results = [];
    let batch = [];

    for (const [index, value] of locations.entries()) {
        batch.push(value);

        if (batch.length === batchSize || index === locations.length - 1) {
            const promise = query10_lambda.trigger({ additionalScope: { locations: batch } });
            results.push(promise);
            batch = [];
        }
    }

    const allResults = await Promise.all(results);
    return allResults;
}

• • Each time I run this script, it only outputs the result for the last location of the array instead of all the batched locations. I suspect there might be an issue with how the batches are being processed or how the asynchronous calls are being handled.

• Screenshots:
  

  

Any idea ?

[ekeith]

Hi Paul,

Happy to help out here.

Something that I've used in the past that might work here is the following:

const promises = locations.entries.map((value, index)=>
      return query10_lambda.trigger({additionalScope: {locations: value} })
); 

return Promise.all(promises)

This is just a stripped version that you can iterate on but I've always received all the calls when using this.

For your setup, it might be worth also checking if the query is being run multiple times and it's just that your JS code block isn't returning each one.

I hope this helps and let me know if you have any questions!

Best,
Evan

[Paul_Merz]

Thanks this worked for me !