Too many ways to skin the cat

🤝 Discussion
1

I just wanted to put out some of my musings while I was building a workflow. I was realizing how many different ways we can accomplish a task with a workflow.

What is the "best way"?

The task is to get many pages of data from a service with a cursor.
We get our first page and our cursor, so now what? We could...

  • Branches and Loop blocks to get the pages in sequence on one level
  • Have the workflow accept a cursor, so instead of looping we can recursively call the workflow
    • My question here, would a recursive style of calling a workflow inflate it's data usage? since we pass more and more records to it? The inverse being if it is all in one workflow with loops, is that any "cheaper"?
  • Make a dedicated GetNextPage function that accepts a cursor?
  • Multi-Step function to get all the pages

I don't know if it affects anyone else, but sometimes I get hung up on which is the "best way" to do it since there are so many options.

Chime in!

2 Likes
2

Well, I could chime in but to be fair I constantly make use of the example you provided here to implement it to similar use cases.

So, my prefer method is the function, but curious to see what other folks share

1 Like
3

Referring me to my own post :rofl: I love it.

Like many things, I forgot about that!

1 Like
4

I like when I search for some solution or design pattern on the forums, find one that seems promising, then learn that I wrote it as a shoddy POC until I could find something better :smiley:

If the server response provides me with a next link (cursor?) I just use a function to process things in a loop... but usually I stick to JS Code runners and decide on the right loop for the job. In this case, it's usually an initial call followed with a while(next) loop.

1 Like
5

There is currently no "standard" way to do cursor pagination, although it is a common feature request and we might implement something to help with this in the future!

I recommend using a code block to implement a while loop that calls a function (similar to this). Implementing a recursive multi-step function is also feasible and will not incur extra cost, but it might be harder to read (unless you prefer recursive style). If you need longer timeouts/data limits, you can also call workflow from workflow recursively, but each call will be counted as a separate run.

3 Likes
6

I think I was also trying to ask:

If a workflow needs to process, say, 1000 records, and its 10mb of data.... If I do a recursive style, where a workflow calls another, collecting the results... would retool charge the data transport over and over for each 0.1 MB per loop?

In other words, would processing all rows in one workflow save on data egress / ingress?

Or Is 10mb of data always 10mb of data?

7

I've recently asked about recursive workflow use that @kent used and he said

However, I also just realized you asked about the physical cost.... not the resource cost.

as for your other questions, in my opinion these boil down to 1 main question:

  • do you want to retrieve and process on the server side or the client (browser) side?

workflows would be server side, functions will be client (not counting functions in workflows lol).

you could split it, load pages from a workflow then return the data to the page and process it there.... or you could do it the other way around, normally you want the connections to databases on the server where it's safer, but since we're using Retool that's not too big of an issue here so you'll probably want the more process/time intensive on the server side.

2 Likes
8

The physical cost of workflows is currently based purely on the number of runs, so data volume not really a consideration in that sense. That said, I believe workflows triggered by other workflows do could as a separate run, meaning a recursive workflow has the potential to add up fairly quickly.

1 Like