Best architecture for a scheduled content audit across 78 public pages?

💬 Workflows
,
4

Hi @cagatay_aydin! Welcome to the Community forum :waving_hand:

Thanks for your patience while I went through your questions! I would also try out the recommendations made by @JB_ventures as well if you haven't done so already. Here are my personal recommendations below:

  1. Loop block vs. child workflow?

For 78 URLs, I would suggest using a Loop block that calls each child workflow in batches. Using the Loop block already looks at success/failure for each item, and you can independently test each child workflow without kicking off the whole entire run.

  1. Parallelism and batching

I recommend running requests sequentially in small batches (e.g. groups of 10) and run each group’s child workflow calls in parallel, and add a small delay between batches. This allows for predictability and will be faster than doing 100% sequential of your 78 requests + timeouts.

  1. Handling individual failures

Perhaps do both - you can configure retries inside the child workflow since they depend on the type of failure (e.g. if timeout/5xx then retry; if 404 then don’t retry).
For continue-on-error, I would configure this in the parent loop since you are running through all your batches, and can show the failure in the child workflow instead.

  1. Parsing HTML

Parsing directly in a JS code block within the child workflow can work since it sounds like it will be a simple extraction task. I believe you can also do an internal API approach here, but it will require more of a heavy lift (e.g. setting up auth). Internal API might make more sense if your parsing logic gets too complex.

  1. Storing current and historical results

I would start with a view pulling the latest result from the historical table (from content_audit_runs table) since it would be the simplest method to maintain, and reading from 78 rows should not be noticeably slow. A separate current_status table can be used if you notice your SQL query is getting slow with many more rows to read through.

  1. Detecting meaningful changes

You can calculate the diff in the workflow and store it as JSON to show a field-by-field comparison against the previous run. Your app can then display the JSON as a table with the before/after changes.

  1. Idempotent upserts

For PostgreSQL, you can use ON CONFLICT (page_id, audit_run_id) DO UPDATE which handles duplicate keys - it inserts a new row or updates based on pre-existing data. I would make sure the audit_run_id is created at the start of the parent workflow and passed to child workflows so all records from one run share the same audit_run_id.

  1. Manual reruns from the app

I would recommend calling a separate workflow that accepts an array of page IDs, and have the workflow call the same child workflow. Keeping this workflow separate would make things easier to track.

  1. Dashboard structure

I would keep all of the information in one app, so you can have all of your reviewing in one central place. Also, I would make use of various components such as the table, collapsable container, and drawer frame components to keep the UI organized and less cluttered.

Lastly, if you are able to join us for Office Hours for live video assistance on Tuesdays and Thursdays, we can take a look at your setup and help you troubleshoot directly :slight_smile:

Hope this helps!

-Jen