Getting specific data from API in a loop until returns "success"

Queries and Resources
,
1

I have an Airflow instance and am calling DAGs with a simple API query from Retool.

I can poll the DAG status from an API call and part of the response includes a "state" key that returns either "running" or "success." I need to poll that API until I get "success."

I've looked at what seems like every page out there on scripting and Retool, and cannot get out of an infinite loop here. Just how in the world do I repeat a query until I get specific data that I want out of it?

Here's the "running" response:

{
  "conf": {
    "message": "29607"
  },
  "dag_id": "zillow_scrape_sold",
  "dag_run_id": "manual__2022-09-10T23:53:01.446968+00:00",
  "end_date": null,
  "execution_date": "2022-09-10T23:53:01.446968+00:00",
  "external_trigger": true,
  "logical_date": "2022-09-10T23:53:01.446968+00:00",
  "start_date": "2022-09-10T23:53:01.763408+00:00",
  "state": "running"
}

And the "success" response:

{
  "conf": {
    "message": "29607"
  },
  "dag_id": "zillow_scrape_sold",
  "dag_run_id": "manual__2022-09-10T23:53:01.446968+00:00",
  "end_date": "2022-09-10T23:53:26.577536+00:00",
  "execution_date": "2022-09-10T23:53:01.446968+00:00",
  "external_trigger": true,
  "logical_date": "2022-09-10T23:53:01.446968+00:00",
  "start_date": "2022-09-10T23:53:01.763408+00:00",
  "state": "success"
}

I have been able to get the specific "state" result out of there but repeating the query until I get "success" is eluding me.

1 Like
2

Hello @fowlerjk,

We would like to see your query and configuration.

Screenshots would be very helpful.

3

This is the best I have, from some StackOverflow answers:

(async function fetchAirflowSuccess() {
  let response = await qryGetDagSuccess.trigger();

  while (response.state != "success") {
    setTimeout(async () => {
      response = await qryGetDagSuccess.trigger();  
    }, 5000);
  }
})();
4

Hey @fowlerjk!

You can set a failure condition on your query that checks the state property of your response:

818×372 42.5 KB

With that, it's also possible to set a failure event handler that retriggers the query with a particular delay:

Could that help?

5

@Kabirdas This won't work as the query simply tells Airflow to start a DAG.

I need a query that polls the DAG until "success" is returned.

Nothing I've tried so far works.

6

More details.

qryGetDagSuccess polls Airflow and returns the state: either running, failed, or success.

I need that query to run every 5 seconds until it returns a success or failed response, at which point I will handle either one.

None of the async/await/promise/etc solutions I've seen have done ANYTHING except run on an endless loop and make me reload my browser.

Latest JS query:

(async function fetchAirflowSuccess() {
  let response = await qryGetDagSuccess.trigger();

  while (response != "success") {
    setTimeout(async () => {
      response = await qryGetDagSuccess.trigger();  
    }, 5000);
  }
})();
7

Bump....this is maddening.

8

Hey @fowlerjk, sorry about the late reply!

If you're looking to run the loop from a JS query you might try something like:

(async function fetchAirflowSuccess() {
  let response = await qryGetDagSuccess.trigger(); 

  while (response != "success") {
    await new Promise(resolve => setTimeout(resolve, 5000));
    response = await qryGetDagSuccess.trigger();
  }
})();

JS functions are already async as well so, if you'd like, you can do away with the wrapper and just use:

let response = await qryGetDagSuccess.trigger(); 

while (response != "success") {
  await new Promise(resolve => setTimeout(resolve, 5000));
  response = await qryGetDagSuccess.trigger();
}

Using await new Promise(resolve => setTimeout(resolve, ms)) causes a synchronous delay as opposed to just using setTimeout by itself which runs async.

Let me know if that helps!