Workflow: Large document handling - best practices?

Workflows
1

Hey there,

Situation:

  • I let users upload documents in a retool app
  • Some uploading 100 MB documents and sometimes 100 documents in one upload
  • I chunk these documents in batches of 5 documents and send the base64 string over to workflows.
  • the workflow uploads to S3 and processes the document

Problem:

  • Handing over larger base64 strings to workflows is crazy. Workflows become slow and timeout eventually.
  • Also, I can't trigger the same workflow multiple time since the workflow trigger in the app waits for the workflow to complete.

Questions:

  • Is there a way to trigger the same workflow several times at one form a retool app without waiting for the response of the workflow (e.g. hand over 50 documents at once?)
  • If I upload the documents to S3 in the frontend (retoool app) I need to wait for each document to upload. Is the intended workaround to just have two or more triggers / Rest API resources and split the documents between those triggers to parallelize?

Thank you!

2

Hey @jpmin! You're able to trigger the same workflow several times from a Retool app but please note these constraints:

Webhooks
Workflows support up to 200 requests every 10 seconds. If this limit is exceeded, subsequent runs may not execute.

Webhook-triggered workflows also support concurrent execution, allowing up to 50 runs at the same time. If a workflow reaches this concurrency limit, one of the runs must complete (whether successful or not) before another can start. [Workflows Rate Limits]

For best practices we also have a Workflow performance best practices docs page.

Regarding your specific use case, handling large amounts of base64 encoded strings is difficult as each workflow has 258MB of available memory. It may be helpful to:

  • Utilize promises within loop blocks in your workflow
  • Use S3 Multipart upload
  • Build separate workflows to perform batched tasks