I'm getting Runtime error: exited workflow with code 139 error.
It appears to be happening in this loop block.
I've seen posts about the same error but have not found a resolution yet.
Instead of using a loop block, using a code block and implementing batch processing with delay solved the issue.
I am confused why this works without an issue and the loop block results in the error. And when should we be using loop blocks vs code blocks when you are iterating over a long array (500+ length) and calls api to get data etc.
async function processInBatchesRecursively(dataArray, batchSize, delayMs, results = [], index = 0) {
if (index >= dataArray.length) {
return results;
}
let batch = dataArray.slice(index, index + batchSize);
let batchResults = await Promise.all(batch.map(item => myFunction(
item.start,
item.stop,
item.id
/*rest of the arguments*/
)));
results = results.concat(batchResults);
await new Promise(resolve => setTimeout(resolve, delayMs));
return processInBatchesRecursively(dataArray, batchSize, delayMs, results, index + batchSize);
}
async function run(data, batchSize, delayMs) {
let finalResults = await processInBatchesRecursively(inputArray, batchSize, delayMs);
let cleanedResults = []
finalResults.forEach(item => {
if(item.data){
cleanedResults.push(item.data)
}
})
return cleanedResults.flat()
}
let inputArray = previousBlock.data;
let batchSize = 30;
let delayMs = 500;
return run(inputArray, batchSize, delayMs);
myFunction is a multi-step function that fetches timeseries data using id, start and stop and calculates metrics.
I think I can help you out here and explain what's going on.
In javascript, error code 139 is a 'Segmentation Fault' (SIGSEGV). Don't worry if that means absolutely nothing, it has to do with memory, because this is one of those things lots of coders now-a-days don't know about as newer languages are able to completely abstract away memory management.
In general, Seg Faults occur from trying to access memory outside of what's been allocated for your code. This can be from things like trying to read from restricted sections of memory or trying to write to a read-only area.... in code, this could be from something like trying to read the 100th item of a list with only 2 items (ive oversimplified this) or creating too many objects overflowing the stack. A loop can have both of these problems.
Since retool manages the loop index for us, we can assume they correctly navigate problems like trying to access an array index that doesn't exist. Loops don't get any more basic here so we can safely move else where to find the source of the error.
Retool also manages a stack (memory) for us but unlike loops, 'custom memory management' is still reliant on more basic functions for finding free memory and allocating it for your use. This means that Retools 'stack' has built-in limits from the OS/server, so while Retools 'stack' says "according to my known limits I can allocate you more memory", when it tries to do so ask JS it gets a response saying "hold on, let me ask the OS for more memory" and the OS replies "nope, you don't get any more" which JS passes on to Retool who passes it on to you.
So why does Retools loop cause the error but yours doesn't? This is because of all the extra code Retool needs to manage ANY kind of loop with ANY kind of data creates a lot of overhead, whereas your loop is designed for 1 specific reason using data whos structure you know.
TLDR; This is a result of 'good at everything' vs 'master of 1 thing'.... the Retool loop block is designed to work 3 different ways using ANY a list of ANY kind of data. So it's 'good at everything', where your code is designed for your specific use making it a 'mast of 1 thing'.
3 Likes