Workflow Won't Terminate - RUNNING SINCE 1:15 PM Thursday - Retool?

I've passed the incident along to our engineering team, and they suspect it may be a frontend bug causing the Terminate Run button not to respond. This could explain why the workflow couldn't be exited as expected.

We'll keep you posted as we learn more, thanks for your patience in the meantime! :pray:

2 Likes

Shawn, (others),

I never really figured this out and I am convinced that to get this workflow successful and stable, I need your brain.

I have not succeeded in calling the pollFunction from a code block. It needs the parameters passed in. It can't see them to refer to them otherwise. My various attempts failed.

Also, the retry period for the existing startExport block is limited to 120 seconds, but I MUST wait 5 minutes between calls as per Microsoft (and as per the error message it generates if you try again to soon). That means I need to launch the API block or do some sort of javascript API call from a looping code block to successfully retry a startExport. (And of course, I need to guarantee that successive calls are at least 5 minutes apart.)

1 Like

Hey @rlhane.2_19635, we have an upcoming Office Hours session tomorrow (July 31), where you can get help live from the Retool team and other developers. We host these Office Hours twice a week!

Hope to see you there. Maybe we can help you solve this mystery! :beetle:

2 Likes

@rlhane.2_19635,

Don't give up! I feel you are very close.

I have a short day today and am away most of the day tomorrow. Office hours is a great place to solve issues live like this. There are always a few Retool team members and some smart developers to help.
Feel free to dm me if you haven't found a solution by Friday and I should be able to set aside some time.

Good luck!

3 Likes

Thanks, ChiEn, Please count me in. Are the screenshots and/or some of the dialog in this thread sufficient, or what should I have ready to share?

For my PBI calls to work, I needed a minimum of 5 minutes before each export. I found that errors caused some of my loops to process too quickly causing more errors that generated more errors. I changed the error handling to Continue, where ever an error condition was possible and added wait before throwing an error on the error path, so the loop could succeed the next time through. Now failed PBI calls aren't cascading failures and the exportFunction is retrying properly. So far, so good.

1 Like

Hey @rlhane.2_19635, if you don’t have a Discord account yet, please follow the instructions here to sign up. We’re starting in less than an hour.

Also, please have your workflows ready in your app, that would be super helpful!

Hello @rlhane.2_19635 Just checking in to see how you're doing. I heard you were able to get help during our office hour! :sunglasses:
You should now be able to run the workflows to completion. Check if your workflows can run for more than 15 minutes, are they doing what they're supposed to do?

Nope. Fails after 15 minutes and 3 reports spaced 5 minutes apart. Will get on the discussion again and check for more advice. Thx -Ron

We are on now @rlhane.2_19635 until 3pm EST if you want to jump on!

Hi ChiEn, I am running out of time. I may have to completely abandon Retool if I can't get to the bottom of this and get out from under this 15-minute workflow time out issue. I can't find the cause. I also don't like what I read here: Workflow always fails (suspiciously at around 15 mins!) - :speech_balloon: Workflows - Retool Forum. Can we discuss this offline? I need to know my options to get this resolved and get the help I need. Thx -Ron

Hi Ron,

Would you be willing to go on a quick Zoom later today? We could do a quick walkthrough to see where you are at and brainstorm scenarios to try?

I'm a bit late here, so sorry if I've got something wrong but it seems like the issue is needing a long running workflow. Normally, they're limited to 15mins, as you've noticed, but there is a way to pause them. Tasks.... I'm fairly sure they weren't meant for this, but I think it will work. The idea is that the majority of the time in that 15mins is actually spent waiting, so we want to move that outside of the workflow. We need to poll every 5 mins, so we don't want to use a workflow specifically for that

  • although, you COULD create a workflow on a 5 min cron job and have it get a list of tasks, poll each one, if it's ready it returns the result otherwise null... I'd actually use like 4min or something cause there is a scenario where a task is posted while the cron job workflow is running causing a longer than 5min wait.

Ideally, in Retool, we'd want the frontend to poll since there are no limits there. So you would want to set up something like:

function pollPowerBI(reportId) {
    console.log("Polling API");
    // Add your polling code here
}

const PbiPollTimer = setInterval(function() {
  // Get all incomplete tasks

  // For each task
  //  - get the reportId
  //  - call pollPowerBI(reportId);
  //  - check response
  //    1) if export isn't finished, leave the task alone and check the next one
  //    2) if export is finished, complete the task sending the polling results
}, 5 * 60 * 1000);

technically, like 95% of the time this will "run" in the background.... while it's paused. when it isn't paused though, the pollPowerBI(repotId) function runs on the "main" thread (the UI). If you're worried about blocking, it shouldn't be an issue since Retool is built on React, but I recently learned you can start a Web Worker with a Data URI instead of using a .js file so you could try using this method to ensure work doesn't interrupt the UI:

let worker = new Worker(
    `data:text/javascript,
    function functionThatTakesLongTime(someArgument){
        //There are obviously faster ways to do this, I made this function slow on purpose just for the example.
        for(let i = 0; i < 1000000000; i++){
            someArgument++;
        }
        return someArgument;
    }
    onmessage = function(event){    //This will be called when worker.postMessage is called in the outside code.
        let foo = event.data;    //Get the argument that was passed from the outside code, in this case foo.
        let result = functionThatTakesLongTime(foo);    //Find the result. This will take long time but it doesn't matter since it's called in the worker.
        postMessage(result);    //Send the result to the outside code.
    };
    `
);

worker.onmessage = function(event){    //Get the result from the worker. This code will be called when postMessage is called in the worker.
    console.log("The result is " + event.data);
}
let foo=10;
worker.postMessage(foo); 
  • little note, foo up above, can be most things except an object with methods. The solution here is dumb:
  // ... all the same code as above here

  let foo = myFunctionWithMethods;
  foo = JSON.parse(JSON.stringify(foo));
  worker.postMessage(foo)

I tested it and it ran (printing 1000000010), you'll obviously need to change it for polling but I think this will get you started (if not just yell at me lol).

I think if you set it up this way your workflow can sit and wait days for the export to complete.... the only downside is that since you've delegated polling to the frontend, you now MUST be on that page for the polling to run. so, a bit of a downside, but maybe its a better place than where you're stuck right now? hopefully? :innocent:

btw, I use polling in one of my apps (chat app actually) it polls on 30sec intervals and the UI has no issues at all. so I really don't think you'll need the Web Worker, but that'll have to be up to you since we don't know what happens after those 3 blocks :beers:

A post was split to a new topic: 2 workflows calling agents stopped running