Calling code repeatedly and using states in Workflow

💬 Workflows
, , , ,
1

Hello!
I am currently trying to work out my first workflow.
I have been working on an app for a while and I wrote code which takes a very long time to execute. As I would like to trigger it regularly, I figured a retool-workflow would suit this perfectly.
As it stands I am having issues setting it up.
The code I would like to trigger regularly is the follwing:

const batchSize = 5;
const orcids = select_expertOrcid.data.orcid;
let allResults = [];
let seenIDs = new Set();
let dupl_counter = 0; 

//  Retry wrapper
//retries 3 times if response is empty 
async function fetchPageWithRetry(query, cursor, retries = 3) {
  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      await query_scopus_db.setValue(query);
      await cursor_scopus_db.setValue(cursor);

      const response = await searchScopus_db.trigger();

      //  Check if the response is undefined or empty
      if (!response || !response["search-results"]) {
        throw new Error("Received empty or undefined response from Scopus API.");
      }

      return response;
    } catch (err) {
      console.warn(` Attempt ${attempt}/${retries} failed: ${err.message || err}`);
      if (attempt === retries) {
        throw new Error(" Max retries reached — skipping batch.");
      }
      await new Promise(resolve => setTimeout(resolve, 3000));
    }
  }
}


for (let i = 0; i < orcids.length; i += batchSize) {
  const batchOrcids = orcids.slice(i, i + batchSize);
  const query = batchOrcids
    .filter(o => o && o.length > 3)
    .map(o => `ORCID(${o})`)
    .join(" OR ");

  const batchIndex = i / batchSize + 1;
  console.log(` Starting batch ${batchIndex}:`, query);

  let cursor = "*";
  let batchResults = [];

  try {
    while (true) {
      console.log(` Fetching page for batch ${batchIndex}, cursor: ${cursor}`);
      let response = await fetchPageWithRetry(query, cursor);

      const entries = response["search-results"]?.entry || [];
      console.log(` Entries this page: ${entries.length} | Total in batch: ${batchResults.length}`);

      for (const entry of entries) {
        const id = entry['dc:identifier'];
        if (id && !seenIDs.has(id)) {
          seenIDs.add(id);
          batchResults.push(entry);
        }
        else if(seenIDs.has(id)){
          dupl_counter++
            console.log("number of dublicate entries: " + dupl_counter)
        }
      }

      const nextCursor = response["search-results"].cursor?.["@next"];
      if (!nextCursor || nextCursor === cursor) {
        console.log(` Finished batch ${batchIndex}, collected ${batchResults.length} entries.`);
        break;
      }

      cursor = nextCursor;
      await new Promise(resolve => setTimeout(resolve, 3000)); // throttle between pages
    }
  } catch (err) {
    console.error(` Batch ${batchIndex} failed: ${err.message}`);
    // continue to next batch instead of halting
  }

  allResults.push(...batchResults);
  await new Promise(resolve => setTimeout(resolve, 3000)); // throttle between batches
}

await tableResults_db.setValue(allResults);
console.log(`🎉 All batches complete! Total unique results: ${allResults.length}`);

searchScopus is an API query and select_expertOrcid is an SQL query connected to my database.
as the code suggests, query_scopus_db , cursor_scopus_db and table_results are all states.

The code is meant to both send get requests for each ORCID in batches of 5 (because to many orcids at once lead to a timeout error) as well as pagination.

I would like to trigger this code once a month and collect the data (I would like to store it as a JSON string) and display it in a table-component in my main app.

I am struggling to execute the code in the workflow, because, as it seems, I cannot use states in workflows.

Thank you in advance !

2

You can't officially use states... but... you can recursively call the same workflow, feeding in "state" you need each call.

This thread has some good insights

And this one

3

I tried to do it similiarly but now I'm having issues with getting data from the API response.
To be clear I do get a result from the get request itself but


  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      const response = await searchScopus(query,cursor)
        console.log("test")
      console.log(response)
        if (!response || !response["search-results"]) {
        throw new Error("Empty or undefined response from Scopus.");
          
      }
console.log("test123")
      return response;
    } catch (err) {
      if (attempt === retries) {
        throw new Error("Max retries reached.");
      }
      await new Promise(resolve => setTimeout(resolve, 3000));
    }
  }

returns nothing and I constantly get stuck in the

Error("Max retries reached.");

Additionally, I need to split up the pagination so I would have to run two loops and I'm not entirely sure what the best practice is.

4

okay I noticed that I had to use response.data["search-results"].
So I made a function called "fetchPageWithRetry"
and used the following code to do what I did in "base" Retool:

//const batchSize = 5;
const allResults = [];
const seenIDs = new Set();
let dupl_counter = 0;

//return batchBuilder.data.join(" OR ")
//const result =  searchScopus(batchBuilder.data[0], "*")
//return result
for (let i = 0; i < batchBuilder.data.length - 1; i++ ) {
    console.log(i)
  const query = batchBuilder.data[i]

  let cursor = "*";
  let batchResults = [];

  
  while (true) {
    const response = await fetchPageWithRetry(query, cursor, 3);
    const entries = response.data["search-results"]?.entry || [];

    for (const entry of entries) {
      const id = entry['dc:identifier'];
      if (id && !seenIDs.has(id)) {
        seenIDs.add(id);
        batchResults.push(entry);
      } else if (seenIDs.has(id)) {
        dupl_counter++;
      }
    }

    const nextCursor = response.data["search-results"].cursor?.["@next"];
    if (!nextCursor || nextCursor === cursor) break;

    cursor = nextCursor;
    await new Promise(resolve => setTimeout(resolve, 3000));
  }
  allResults.push(...batchResults);
  await new Promise(resolve => setTimeout(resolve, 3000));
}

return allResults;

unfortunatly I get the folloing error:

{"data":null,"metadata":{},"error":"Internal Error running a block: TypeError: NetworkError when attempting to fetch resource."}

I am not entirely sure what that specifically means.

Once again, thanks for any help in advance!

5

I would recommend using the Loop block to manage the delay or batching instead of manually doing it.

I'm using Monday here, but you can have the loop executor be Javascript code.

image
image470×810 43.3 KB

Another example with Javascript calling workflow Functions

image
image570×703 45.6 KB

2 Likes
6

How does the return work in the loop block?
doesnt it return something each iteration?
I need to collect all the fetched data and display it at once!

7

The result of the loop block will be an array of all of the iterations in aggregate. The try/catch suggestion is used to format all of the responses accordingly.

1 Like
8

Imagine the executor block in my second screenshot as a map function.

Each return in the block would be for one iteration of the loop

image
image975×457 27.7 KB

I put a try/catch in mine to not crash the loop if I error out. I just return the error instead of the assembled object.

1 Like