I have a data intensive application - which essentially does the follow:
Reads transactions and file data from netsuite (files tranferred as base64 encoded string) via custom API endpoint
PUT’s that file data into company sharepoint site
POSTS a batch to the file adding custom netsuite transaction metadata
this is done for thousands of records.
I was able to get this app to run continuously for > 12 hrs, but ONLY by occasionally going into the page console.log and clearing all of the trigger logs that retool adds.
In the app, i dsiabled every notification on query succes, internal logs, etc etc. Why does retool insist on forcing all of those logs to the console? In my case, with such large payloads both sending and rxing, eventually crashes the app.
Is there a chance this could be a system parameter to disable?
As always, retool is awesome, but always room for improvement
In the end, my app successfully migrated 22,000 files from netsuite into Sharepoint with all the netsuite record details! A function that would have cost us 15k from a third party, as well as a recurring S3 bucket fee for the data storage!
Great to hear you were able to migrate such a large number of files!
I am curious about these console logs that you are seeing. Can you share a screenshot of what these look like? Are the coming from one or all of the queries on run success?
And in my case, since i was transferring alot of very large datablocks, it eventually would drive these numbers very high, unless i kept manually clearing the browser (collecting garbage) – and manually clearing browser console.logs
Thank you for specifying and sharing the screen shot!
Yes Retool does log to the browsers console the data from query runs. I just talked to an engineer on our team and he thought that the more likely cause of the browser crashing is the volume of query data being processed via Retool and the browser and less so the console logs.
The recommended batching the calls to reduce how much data is in browser memory.
They also explained to me that retool-edge is mostly loading client size javascript.
I can make a feature request for toggling off the console logs to the browser but I don't think that fix would be made soon and may not have much of an impact.
Have you considered using Retool Workflows for handling large number of API calls for data migrations?
