I have a workflow that processes prescription information.
Every 1 in 100 times it appears to crash at the start trigger block. I do not know why. Since it's in the start trigger, I'm not even sure how I would go about debugging this.
The webhook is called on submission of a Cognito form.
If I rerun the failed call, it continues to fail, implying there is something wrong with the JSON payload.
Hi @Tom_Burns, and welcome to the Community Forum!
On a failed run, can you check the Input tab and see what that payload looks like? That could help us debug what the issue is. Something else you could do is check the logs for a failed run and DM me the workflowrun ID as well as the workflow ID, and we can find some logs and see whats happening.
Starting run for workflow: dd91263c-4766-4167-a77d-5fde4d891328 in environment production
Mon 2025-06-02 11:53:47
Workflow Run Id: 01972e58-471a-7059-b15b-73bc39a98762
(Attachment failed_workflow.txt is missing)
{"value":{"Form":{"Id":"9","InternalName":"MedicationOrderPrescriptionDetailsCollection","Name":"Medication Order - Prescription Details Collection"},"YourDetails":{"Name":{"First":"Sylvia","FirstAndLast":"Sylvia Vaartjes","Last":"Vaartjes","Middle":null,"MiddleInitial":null,"Prefix":null,"Suffix":null},"OrderNumber":1000006,"PrescriptionUpload":[{"Id":"2Kdy5a","UploadPrescription":[{"ContentType":"image/heic","Id":"F-1Sc88btD$Yyr5xFA5iil5E","IsEncrypted":true,"Name":"IMG_3637.HEIC","Size":3517426,"StorageUrl":null,"File":"https://www.cognitoforms.com/fa/Fxbx7xmyuEaChkNBzbdYNg?token=ccFK2yLnLbp1JjMRHRYLV8uDi6iqhNm1l8D-EZLQKWp1eiXp54eCWgfKzL_M8-GdhHHWdPdtnHiUbPQJDu-Cmg$$"}],"PetName":"Fifi","ItemNumber":1}],"PrescriptionPostageDetails":{"HaveYouPostedYourOriginalPrescriptionsToPetChemistYet":true,"PleaseSelectOneOfTheFollowingOptionsThatBestApplies":null,"DidYouPostYourPrescriptionByRegisteredTrackedOrExpressPostrecommended":false,"TrackingNumberForPostedOriginalPrescription":null,"PostalTrackingNumber":null,"PhotoOfTrackingNumber":[],"HaveYouPostedYourOriginalPrescriptionsToPetChemistYet2":"Yes","PleaseSelectOneOfTheFollowingOptionsThatBestApplies_IsRequired":false},"HaveYouOrderedOneOfTheFollowingRestrictedMedications":"No","OrderNumber_Minimum":18,"OrderNumber_IncrementBy":1},"Entry":{"AdminLink":"https://www.cognitoforms.com/MadPaws1/9/entries/114818","DateCreated":"2025-06-02T01:53:44.146Z","DateSubmitted":"2025-06-02T01:53:44.146Z","DateUpdated":"2025-06-02T01:53:44.146Z","Number":114818,"Order":null,"Origin":{"City":null,"CountryCode":null,"IpAddress":"122.150.145.21","IsImported":false,"Region":null,"Timezone":null,"UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"},"Timestamp":"2025-06-02T01:53:44.146Z","User":null,"Version":1,"Action":"Submit","Role":"Public","Status":"Submitted","Document1":"https://www.cognitoforms.com/d/Fxbx7xmyuEaChkNBzbdYNg?code=ez0KE34sQNZJdcPqoXP5EiDQN5Zc4jwBzzwExXUW1O4$","Document2":"https://www.cognitoforms.com/d/Fxbx7xmyuEaChkNBzbdYNg?code=lRl3X53BUmIzQ11V240cqDKFUL2KAsVhhBz11vmKV7s$"},"Id":"9-114818"},"metadata":null,"data":{"Form":{"Id":"9","InternalName":"MedicationOrderPrescriptionDetailsCollection","Name":"Medication Order - Prescription Details Collection"},"YourDetails":{"Name":{"First":"Sylvia","FirstAndLast":"Sylvia Vaartjes","Last":"Vaartjes","Middle":null,"MiddleInitial":null,"Prefix":null,"Suffix":null},"OrderNumber":1000006,"PrescriptionUpload":[{"Id":"2Kdy5a","UploadPrescription":[{"ContentType":"image/heic","Id":"F-1Sc88btD$Yyr5xFA5iil5E","IsEncrypted":true,"Name":"IMG_3637.HEIC","Size":3517426,"StorageUrl":null,"File":"https://www.cognitoforms.com/fa/Fxbx7xmyuEaChkNBzbdYNg?token=ccFK2yLnLbp1JjMRHRYLV8uDi6iqhNm1l8D-EZLQKWp1eiXp54eCWgfKzL_M8-GdhHHWdPdtnHiUbPQJDu-Cmg$$"}],"PetName":"Fifi","ItemNumber":1}],"PrescriptionPostageDetails":{"HaveYouPostedYourOriginalPrescriptionsToPetChemistYet":true,"PleaseSelectOneOfTheFollowingOptionsThatBestApplies":null,"DidYouPostYourPrescriptionByRegisteredTrackedOrExpressPostrecommended":false,"TrackingNumberForPostedOriginalPrescription":null,"PostalTrackingNumber":null,"PhotoOfTrackingNumber":[],"HaveYouPostedYourOriginalPrescriptionsToPetChemistYet2":"Yes","PleaseSelectOneOfTheFollowingOptionsThatBestApplies_IsRequired":false},"HaveYouOrderedOneOfTheFollowingRestrictedMedications":"No","OrderNumber_Minimum":18,"OrderNumber_IncrementBy":1},"Entry":{"AdminLink":"https://www.cognitoforms.com/MadPaws1/9/entries/114818","DateCreated":"2025-06-02T01:53:44.146Z","DateSubmitted":"2025-06-02T01:53:44.146Z","DateUpdated":"2025-06-02T01:53:44.146Z","Number":114818,"Order":null,"Origin":{"City":null,"CountryCode":null,"IpAddress":"122.150.145.21","IsImported":false,"Region":null,"Timezone":null,"UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"},"Timestamp":"2025-06-02T01:53:44.146Z","User":null,"Version":1,"Action":"Submit","Role":"Public","Status":"Submitted","Document1":"https://www.cognitoforms.com/d/Fxbx7xmyuEaChkNBzbdYNg?code=ez0KE34sQNZJdcPqoXP5EiDQN5Zc4jwBzzwExXUW1O4$","Document2":"https://www.cognitoforms.com/d/Fxbx7xmyuEaChkNBzbdYNg?code=lRl3X53BUmIzQ11V240cqDKFUL2KAsVhhBz11vmKV7s$"},"Id":"9-114818"},"headers":{"host":"api.retool.com","x-request-id":"803877dea68ab42ff8de878d1283f9bb","x-real-ip":"4.156.104.40","x-forwarded-for":"4.156.104.40","x-forwarded-host":"api.retool.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"2145","x-requested-with":"XMLHttpRequest","user-agent":"CognitoForms","content-type":"application/json","request-id":"|dc270172-4a31-4885-b4f5-ce324f314868.22bde98a6da7c306.","traceparent":"00-1b2977c872e6d140259463cd1f6c5ed9-22bde98a6da7c306-00"},"urlParams":{},"pathParams":null}
Thanks for the payload and the IDs! I tracked down the workflow run and found an error saying "The workflow run exceeded the memory limit of around 1GB". Is it possible that your users are able to submit forms with too much data? Maybe you can look into your failed runs and see if there's a pattern with them in that regard, especially since you mentioned that it fails on reruns of that payload.
If you find anything I'd be curious to know but I'll keep looking into it on my end as well.
This is confusing.
This is a workflow that failed recently.
Starting run for workflow: dd91263c-4766-4167-a77d-5fde4d891328 in environment production
Mon 2025-06-02 15:32:36
Workflow Run Id: 01972f20-9ba1-7045-83ef-5c36759d0986
Mon 2025-06-02 15:32:36
Workflow Trigger Id: 5ce16f64-e9d4-4073-82ee-7a7ed08eb3c6
I assume it also exceeded the 1gb memory limit. There are 3 prescriptions attached to the form. If I run them individually they succeed, and state ~11mb data usage.
I've attached an image of what I see in the logs. Is it crashing on start up or further in the workflow? Do you see more diagnostic info than I've got here?
Assuming no, why would it exceed 1gb of memory on startTrigger, before my code has even started to run?
Script #1
Starting run for workflow: dd91263c-4766-4167-a77d-5fde4d891328 in environment production
Tue 2025-06-03 08:53:34
Workflow Run Id: 019732d9-a2f3-73ff-b04f-097bc3e407fa
Script #2
Starting run for workflow: dd91263c-4766-4167-a77d-5fde4d891328 in environment production
Tue 2025-06-03 08:54:12
Workflow Run Id: 019732da-3a74-767c-8e4c-fc790c72b681
Script #3
Starting run for workflow: dd91263c-4766-4167-a77d-5fde4d891328 in environment production
Tue 2025-06-03 08:54:34
Workflow Run Id: 019732da-904a-70cb-bee6-280860b8313f
For that workflow run, I'm seeing this error: "FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory", so it definitely tracks with memory issues. I also see the runtime error: exited workflow with code 139, and here is a thread on that. I've asked internally and there is a 100MB payload limit for any workflow block, so a lot lower than 1GB. While I get that ~33MB of data is lower than 100MB, I would still look into ways to limit the size of the incoming payloads.
Hi @Tom_Burns, how did it go? Were you able to smooth things out with the memory issues?
Hi Mike,
I've had a little success, but it's still happening. For other reasons I had to split my Retool account and I migrated this workflow to the new account.
The errors I'm seeing are a little different, is this data for the most recent failure.Is this still a memory issue?
Starting run for workflow: d92a6065-3e99-4703-b211-97719d49616a in environment production
Tue 2025-07-08 16:09:25
Workflow Run Id: 0197e8a7-41f7-76e8-9b4a-91151f2c51ff
Tue 2025-07-08 16:09:25
Workflow Trigger Id: a59967f9-9860-43cd-857c-f24509a69ee5
If it is crashing here, presumably in the checkFormData code block, I'm not sure how it would consume all memory so quickly. Later in the workflow converts PDFs to images, before running AI analysis. I would assume any of those steps would consume more memory. I think it's difficult for me to fix without understanding more of the inner workings of the system.
Is there any chance we could set up a call to discuss this and what options might be available?
I checked the logs and I'm seeing Runtime error: exited workflow with code 139 again, as well as FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory deeper into the stack trace.
When I've looked up the 139 error internally it seems like looping over large amounts of data is a common culprit, so this could be caused by the step you mentioned where it converts PDFs to images. For these types of errors we recommend either limiting the data that comes in or split the work up into separate workflows.
I've also heard that in this situation, having a webhook response block attached will cause logs not to show up. If you want to see the logs with a more descriptive error you can try to intentionally send a large amount of data through the workflow without the response block attached, and your global error handler should be able to pick it up.
We are hosting Office Hours tomorrow on Discord from 2pm - 3pm EDT and that is the best way to get 1 on 1 help from us. I'll be there as well as other knowledgeable folks on the team!