Best architecture for a scheduled content audit across 78 public pages?

[Cagatay_Aydin]

I am prototyping an internal editorial QA dashboard in Retool for a Shopify-based reference website.

The site contains 78 related content pages, one for each item in a fixed reference set. This is the overview page that links to the individual articles:

https://deckaura.com/blogs/guide/tarot-card-meanings

Disclosure: I maintain the linked website. I am sharing it only to show the real content structure that the Retool app would audit.

The goal is to run a scheduled check and identify pages with issues such as:

• Non-200 HTTP responses

• Missing or incorrect canonical URLs

• Duplicate page titles

• Missing meta descriptions

• More than one H1

• Missing structured data

• Broken internal links

• Unexpected redirects

• Very short content

• Incorrect card names or slugs

• Missing upright or reversed sections

• Pages that have changed since the previous audit

My proposed workflow is:

1. Run once every night.

2. Fetch the expected list of 78 records from a database or JSON endpoint.

3. Request each public page.

4. Extract the required metadata and content checks.

5. Compare the result with the expected values.

6. Upsert one audit record per URL.

7. Display failures in a Retool app for editorial review.

8. Allow an editor to rerun checks for selected URLs.

A simplified audit record might look like this:

{
  "url": "https://example.com/article",
  "slug": "the-world",
  "status_code": 200,
  "canonical_url": "https://example.com/article",
  "title": "The World Tarot Meaning",
  "meta_description_length": 154,
  "h1_count": 1,
  "has_structured_data": true,
  "broken_link_count": 0,
  "content_hash": "sha256-value",
  "checked_at": "2026-07-23T02:00:00Z",
  "audit_status": "passed"
}

I would appreciate advice on the following architecture questions.

1. Loop block or child workflow?

For 78 URLs, would you process everything in one Loop block, or call a child workflow once per page?

A single loop seems easier to understand, but a child workflow might make retries and individual failures easier to isolate.

2. Parallelism and batching

Would you run the requests sequentially, in parallel, or in small batches?

I do not need the audit to finish immediately. I would prefer predictable execution and low load on the website over maximum speed.

A batch size of five or ten pages seems reasonable, but I am not sure what pattern works best within Retool Workflows.

3. Handling individual page failures

If one page times out or returns a temporary error, the remaining pages should still be checked.

Would you configure retries and continue-on-error directly inside the loop, or have the child workflow return a normalized failure object such as:

{
  "url": "https://example.com/article",
  "audit_status": "request_failed",
  "error_message": "Request timed out"
}

I would also like temporary network errors to retry automatically without retrying permanent 404 responses.

4. Parsing HTML

Would you parse the HTML directly in a JavaScript Code block, or expose a small internal API that returns normalized page data?

The internal API approach would keep HTML parsing outside Retool and make the workflow simpler. The direct approach would require less infrastructure.

The fields I need are fairly limited:

• Title

• Meta description

• Canonical URL

• H1 and H2 headings

• JSON-LD blocks

• Internal links

• Selected section headings

• Visible word count

5. Storing current and historical results

I am considering two tables:

content_pages
content_audit_runs

content_pages would contain the expected editorial state.

content_audit_runs would contain one historical result for every URL and audit date.

Would you keep a separate current-status table for faster dashboard queries, or derive the latest result from the historical table?

6. Detecting meaningful changes

I plan to store a normalized content hash after removing navigation, footer and other repeated Shopify template elements.

If the hash changes, the editor could compare the current result with the previous run.

Is there a good Retool pattern for displaying field-level differences, or would you calculate the diff in the workflow and save it as structured JSON?

7. Idempotent upserts

If a workflow is retried, it should not create duplicate audit records.

Would a unique key such as this be sufficient?

page_id + audit_run_id

The workflow could then perform an upsert instead of an insert.

8. Manual reruns from the app

In the Retool app, editors should be able to select failed rows and click “Recheck selected pages.”

Would you trigger the same workflow through a webhook or call a separate workflow that accepts an array of page IDs?

I would prefer the scheduled and manual audits to reuse the same checking logic.

9. Dashboard structure

My initial app layout would include:

• Summary cards for passed, warning and failed pages

• A filterable table of all 78 pages

• Expandable rows containing the exact checks

• A side panel showing current and previous values

• A button to rerun selected pages

• A notes and review-status field for editors

• CSV export for the latest audit

Would you keep all of this in one app, or separate the audit overview from the individual page-review interface?

I am mainly looking for a maintainable Retool architecture rather than help with the tarot content itself.

[JB_Ventures]

For 78 pages, I would use a small coordinator workflow plus one reusable single-page child workflow, rather than one large Loop block. The coordinator should create one audit_run_id, snapshot the enabled page IDs and expected values for that run, split them into batches of about five, cap child concurrency at roughly 3–5, persist each result independently, and then finalize run-level counts and status. The child should have a strict input/output contract and always return a normalized result object, including on request failure, so one bad page cannot stop the remaining audit. Retry only transient cases—connection/timeout failures, 408, 429, and 5xx—with a small exponential backoff and Retry-After support; do not retry permanent 4xx responses such as 404. For the fields listed, parsing server-rendered HTML in a JavaScript Code block is a reasonable first version if it uses a real HTML parser rather than regex. Resolve relative URLs against the final response URL and record redirects/final URL. I would keep broken-link checking as a separate bounded phase that deduplicates links and caches one result per link per run, otherwise 78 pages can fan out into thousands of requests. For storage, use content_pages for expected editorial state, content_audit_runs for one row per scheduled or manual run, and content_audit_results for one page per run with request facts, extracted fields, check results, normalized hash, structured diff, parser/normalizer versions, and error details. At this scale, derive current status from a view selecting the latest completed result per page; add a separate current table only if measured dashboard latency justifies the extra dual-write risk. For meaningful change detection, hash deliberately normalized main/article content after removing script/style/noscript and known template chrome, Unicode-normalizing, collapsing whitespace, and normalizing URLs. Store parser_version and normalizer_version so a parser change is not mistaken for an editorial change, and calculate a compact field-level diff in the checker so Retool only needs to render it. A unique key on (audit_run_id, page_id) is sufficient if retries reuse the original audit_run_id; upsert on that key and retain attempt_count plus first/last attempt timestamps. Generate a new run ID for a genuinely new scheduled run or manual rerun, not for a retry of the same invocation. Manual rechecks should call the same coordinator/checker path with page IDs, requested_by, source=manual, and an idempotency key; re-read URLs and expected values from content_pages rather than trusting client-supplied values. I would start with one Retool app: summary cards and a filterable table, with a drawer for current-versus-previous values, checks, notes, review status, and rerun action. Split review into another app only if permissions or workflow become materially different. Monitor missing/stale nightly runs, coordinator failures, incomplete runs, failure rate, duration, and retry counts rather than alerting on every expected page warning. I ran a narrow read-only two-page proof against the linked overview and one article: both returned HTTP 200 and passed basic bounded-request, metadata-parsing, normalized-hash, explicit-failure-record, and stable run_id-plus-URL-key checks. That is parser-shape evidence only; I have not run all 78 pages, implemented Retool scheduling or storage, performed a broken-link crawl, encoded the editorial rules, deployed anything, or provided uptime or ongoing support.

[Jen]

Hi @cagatay_aydin! Welcome to the Community forum

Thanks for your patience while I went through your questions! I would also try out the recommendations made by @JB_ventures as well if you haven't done so already. Here are my personal recommendations below:

1. Loop block vs. child workflow?

For 78 URLs, I would suggest using a Loop block that calls each child workflow in batches. Using the Loop block already looks at success/failure for each item, and you can independently test each child workflow without kicking off the whole entire run.

2. Parallelism and batching

I recommend running requests sequentially in small batches (e.g. groups of 10) and run each group’s child workflow calls in parallel, and add a small delay between batches. This allows for predictability and will be faster than doing 100% sequential of your 78 requests + timeouts.

3. Handling individual failures

Perhaps do both - you can configure retries inside the child workflow since they depend on the type of failure (e.g. if timeout/5xx then retry; if 404 then don’t retry).
For continue-on-error, I would configure this in the parent loop since you are running through all your batches, and can show the failure in the child workflow instead.

4. Parsing HTML

Parsing directly in a JS code block within the child workflow can work since it sounds like it will be a simple extraction task. I believe you can also do an internal API approach here, but it will require more of a heavy lift (e.g. setting up auth). Internal API might make more sense if your parsing logic gets too complex.

5. Storing current and historical results

I would start with a view pulling the latest result from the historical table (from content_audit_runs table) since it would be the simplest method to maintain, and reading from 78 rows should not be noticeably slow. A separate current_status table can be used if you notice your SQL query is getting slow with many more rows to read through.

6. Detecting meaningful changes

You can calculate the diff in the workflow and store it as JSON to show a field-by-field comparison against the previous run. Your app can then display the JSON as a table with the before/after changes.

7. Idempotent upserts

For PostgreSQL, you can use ON CONFLICT (page_id, audit_run_id) DO UPDATE which handles duplicate keys - it inserts a new row or updates based on pre-existing data. I would make sure the audit_run_id is created at the start of the parent workflow and passed to child workflows so all records from one run share the same audit_run_id.

8. Manual reruns from the app

I would recommend calling a separate workflow that accepts an array of page IDs, and have the workflow call the same child workflow. Keeping this workflow separate would make things easier to track.

9. Dashboard structure

I would keep all of the information in one app, so you can have all of your reviewing in one central place. Also, I would make use of various components such as the table, collapsable container, and drawer frame components to keep the UI organized and less cluttered.

Lastly, if you are able to join us for Office Hours for live video assistance on Tuesdays and Thursdays, we can take a look at your setup and help you troubleshoot directly

Hope this helps!

-Jen