Workflow: Large document handling - best practices?

Hey there,

Situation:

  • I let users upload documents in a retool app
  • Some uploading 100 MB documents and sometimes 100 documents in one upload
  • I chunk these documents in batches of 5 documents and send the base64 string over to workflows.
  • the workflow uploads to S3 and processes the document

Problem:

  • Handing over larger base64 strings to workflows is crazy. Workflows become slow and timeout eventually.
  • Also, I can't trigger the same workflow multiple time since the workflow trigger in the app waits for the workflow to complete.

Questions:

  • Is there a way to trigger the same workflow several times at one form a retool app without waiting for the response of the workflow (e.g. hand over 50 documents at once?)
  • If I upload the documents to S3 in the frontend (retoool app) I need to wait for each document to upload. Is the intended workaround to just have two or more triggers / Rest API resources and split the documents between those triggers to parallelize?

Thank you!