How to handle Polling in workflows

💬 Workflows
,
1

  1. My goal:
    I am using a workflow that grabs data from the powerbi api. One of the endpoints requires me to poll to check if the “status” field gets updated from “running” to “succeeded”. I am trying to find a way handle this gracefully in the Retool Workflow UI.

  2. Issue:
    I am trying to follow this pattern to recursively run my workflow so that it polls for data every 3 seconds or so. However, it seems like all my code blocks work, but my workflow never gets triggered again. All it does it just calls the workflow again and it shows this default “code1” block in the logs.

    • Call the API via REST block
    • Branch Block based on status
    • Wait Block (~3s)
    • Trigger workflow again recursively (pdf-workflow)

  3. Steps I've taken to troubleshoot:

    • Looked in the workflow logs for any information for failures
    • Looked into refactoring to use a loop coding block, but I don’t think this will work, there is no option to run the loop (i.e. a while loop). The input requires an array.

  4. Additional info: (Cloud or Self-hosted, Screenshots)
    We are Cloud Hosted.
    screenshots:

    image
    image437×667 22.6 KB

    image
    image1083×459 77.4 KB

image
image941×405 36.1 KB

2

Hi @brpark, have you seen this topic on recursive workflows? I believe a difference maker could be including a Response block in the workflow.

I'd be remiss not to mention the potential for a recursive workflow to add up quickly in cost, since the price is based on the number of workflow runs. Here is another topic discussing the tradeoffs there.

3

Thanks @Mike_M ! that’s exactly what I’m trying to do. I have the same kind of structure as the OP, but like Sharon_Alexander, in the thread, I see the same issue as her. It seems like she also has the Response block in the workflow too.

4

Hi there! Just thought I’d point out that if you know the timeout on whatever your polling (usually if something requires polling it has a limit for how long it will be labeled ‘running’ before it automatically switches to ‘failed’ to prevent stuck requests and stuff), but if this time is less than 10 mins, then you can guarantee your polling loop will always either ‘succeed’ or ‘fail’ before the workflow will timeout so you can move your polling to inside the workflow. You could actually be accidently ‘queuing’ up a bunch of requests and not know it…. workflow calls are actually REST requests, so if the overhead of making the request + the time for the server to respond + the time to validate and process response headers >= 3 seconds then the next polling iteration, and possibly every one after it, will attempt to send a request before the server could respond to the previous request creating a traffic jam. Personally, I’d consider either moving the polling to inside the workflow OR moving the workflow to the UI and using a REST or API Resource (or javascript/python), as long as your not polling 10 different services at once your UI responsiveness shouldn’t take a hit (I use polling for a chat app which is always running and haven’t had any issues)

1 Like
5

I think we found a work around for now, but thanks @bobthebear , that is helpful information!

1 Like
6

are you able to share what workaround you went with? I’m mostly just curious, but I also wouldn’t mind seeing a post that can be marked as the solution for others that come across this issue. :beers: