Solved. Thank you to Retool Support and posts in this community.
We learned the issue was a 504 Gateway Timeout. This was not obvious in the Workflows Resource Query component before our upgrade to 3.26.4, but we may have overlooked it.
With that information, we turned up this community article and this public one.
We added the settings below to nginx.conf and did a docker-compose restart. After that, we upserted 10k records without issue.
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
No loops or batches were needed. It was a big time sink, but I learned a lot.
Thanks,
Andrew