Best architecture for a scheduled content audit across 78 public pages?

💬 Workflows
,
1

I am prototyping an internal editorial QA dashboard in Retool for a Shopify-based reference website.

The site contains 78 related content pages, one for each item in a fixed reference set. This is the overview page that links to the individual articles:

https://deckaura.com/blogs/guide/tarot-card-meanings

Disclosure: I maintain the linked website. I am sharing it only to show the real content structure that the Retool app would audit.

The goal is to run a scheduled check and identify pages with issues such as:

  • Non-200 HTTP responses

  • Missing or incorrect canonical URLs

  • Duplicate page titles

  • Missing meta descriptions

  • More than one H1

  • Missing structured data

  • Broken internal links

  • Unexpected redirects

  • Very short content

  • Incorrect card names or slugs

  • Missing upright or reversed sections

  • Pages that have changed since the previous audit

My proposed workflow is:

  1. Run once every night.

  2. Fetch the expected list of 78 records from a database or JSON endpoint.

  3. Request each public page.

  4. Extract the required metadata and content checks.

  5. Compare the result with the expected values.

  6. Upsert one audit record per URL.

  7. Display failures in a Retool app for editorial review.

  8. Allow an editor to rerun checks for selected URLs.

A simplified audit record might look like this:

{
  "url": "https://example.com/article",
  "slug": "the-world",
  "status_code": 200,
  "canonical_url": "https://example.com/article",
  "title": "The World Tarot Meaning",
  "meta_description_length": 154,
  "h1_count": 1,
  "has_structured_data": true,
  "broken_link_count": 0,
  "content_hash": "sha256-value",
  "checked_at": "2026-07-23T02:00:00Z",
  "audit_status": "passed"
}

I would appreciate advice on the following architecture questions.

1. Loop block or child workflow?

For 78 URLs, would you process everything in one Loop block, or call a child workflow once per page?

A single loop seems easier to understand, but a child workflow might make retries and individual failures easier to isolate.

2. Parallelism and batching

Would you run the requests sequentially, in parallel, or in small batches?

I do not need the audit to finish immediately. I would prefer predictable execution and low load on the website over maximum speed.

A batch size of five or ten pages seems reasonable, but I am not sure what pattern works best within Retool Workflows.

3. Handling individual page failures

If one page times out or returns a temporary error, the remaining pages should still be checked.

Would you configure retries and continue-on-error directly inside the loop, or have the child workflow return a normalized failure object such as:

{
  "url": "https://example.com/article",
  "audit_status": "request_failed",
  "error_message": "Request timed out"
}

I would also like temporary network errors to retry automatically without retrying permanent 404 responses.

4. Parsing HTML

Would you parse the HTML directly in a JavaScript Code block, or expose a small internal API that returns normalized page data?

The internal API approach would keep HTML parsing outside Retool and make the workflow simpler. The direct approach would require less infrastructure.

The fields I need are fairly limited:

  • Title

  • Meta description

  • Canonical URL

  • H1 and H2 headings

  • JSON-LD blocks

  • Internal links

  • Selected section headings

  • Visible word count

5. Storing current and historical results

I am considering two tables:

content_pages
content_audit_runs

content_pages would contain the expected editorial state.

content_audit_runs would contain one historical result for every URL and audit date.

Would you keep a separate current-status table for faster dashboard queries, or derive the latest result from the historical table?

6. Detecting meaningful changes

I plan to store a normalized content hash after removing navigation, footer and other repeated Shopify template elements.

If the hash changes, the editor could compare the current result with the previous run.

Is there a good Retool pattern for displaying field-level differences, or would you calculate the diff in the workflow and save it as structured JSON?

7. Idempotent upserts

If a workflow is retried, it should not create duplicate audit records.

Would a unique key such as this be sufficient?

page_id + audit_run_id

The workflow could then perform an upsert instead of an insert.

8. Manual reruns from the app

In the Retool app, editors should be able to select failed rows and click “Recheck selected pages.”

Would you trigger the same workflow through a webhook or call a separate workflow that accepts an array of page IDs?

I would prefer the scheduled and manual audits to reuse the same checking logic.

9. Dashboard structure

My initial app layout would include:

  • Summary cards for passed, warning and failed pages

  • A filterable table of all 78 pages

  • Expandable rows containing the exact checks

  • A side panel showing current and previous values

  • A button to rerun selected pages

  • A notes and review-status field for editors

  • CSV export for the latest audit

Would you keep all of this in one app, or separate the audit overview from the individual page-review interface?

I am mainly looking for a maintainable Retool architecture rather than help with the tarot content itself.

3

For 78 pages, I would use a small coordinator workflow plus one reusable single-page child workflow, rather than one large Loop block. The coordinator should create one audit_run_id, snapshot the enabled page IDs and expected values for that run, split them into batches of about five, cap child concurrency at roughly 3–5, persist each result independently, and then finalize run-level counts and status. The child should have a strict input/output contract and always return a normalized result object, including on request failure, so one bad page cannot stop the remaining audit. Retry only transient cases—connection/timeout failures, 408, 429, and 5xx—with a small exponential backoff and Retry-After support; do not retry permanent 4xx responses such as 404. For the fields listed, parsing server-rendered HTML in a JavaScript Code block is a reasonable first version if it uses a real HTML parser rather than regex. Resolve relative URLs against the final response URL and record redirects/final URL. I would keep broken-link checking as a separate bounded phase that deduplicates links and caches one result per link per run, otherwise 78 pages can fan out into thousands of requests. For storage, use content_pages for expected editorial state, content_audit_runs for one row per scheduled or manual run, and content_audit_results for one page per run with request facts, extracted fields, check results, normalized hash, structured diff, parser/normalizer versions, and error details. At this scale, derive current status from a view selecting the latest completed result per page; add a separate current table only if measured dashboard latency justifies the extra dual-write risk. For meaningful change detection, hash deliberately normalized main/article content after removing script/style/noscript and known template chrome, Unicode-normalizing, collapsing whitespace, and normalizing URLs. Store parser_version and normalizer_version so a parser change is not mistaken for an editorial change, and calculate a compact field-level diff in the checker so Retool only needs to render it. A unique key on (audit_run_id, page_id) is sufficient if retries reuse the original audit_run_id; upsert on that key and retain attempt_count plus first/last attempt timestamps. Generate a new run ID for a genuinely new scheduled run or manual rerun, not for a retry of the same invocation. Manual rechecks should call the same coordinator/checker path with page IDs, requested_by, source=manual, and an idempotency key; re-read URLs and expected values from content_pages rather than trusting client-supplied values. I would start with one Retool app: summary cards and a filterable table, with a drawer for current-versus-previous values, checks, notes, review status, and rerun action. Split review into another app only if permissions or workflow become materially different. Monitor missing/stale nightly runs, coordinator failures, incomplete runs, failure rate, duration, and retry counts rather than alerting on every expected page warning. I ran a narrow read-only two-page proof against the linked overview and one article: both returned HTTP 200 and passed basic bounded-request, metadata-parsing, normalized-hash, explicit-failure-record, and stable run_id-plus-URL-key checks. That is parser-shape evidence only; I have not run all 78 pages, implemented Retool scheduling or storage, performed a broken-link crawl, encoded the editorial rules, deployed anything, or provided uptime or ongoing support.