Hi, our organization just started experiencing 404 HTML page responses to requests our backend service has been making to /api/embed-url/external-user. We don't believe any configuration has changed on our end and are curious if something has changed on Retool's side that would be preventing us from using that endpoint to create external app embed links.
This is blocking some of our customers from using a Retool application we've exposed for customer consumption.
Hi @dennis_nimblerx,
Thanks for reaching out! Apologies for the delay - we had limited coverage for the holidays. Are you still experiencing this issue?
Given the limited coverage, there shouldn't have been major changes to Retool Cloud & apis last week; it looks like the last version change for Cloud was December 17th
Is it impacting all user groups? Could you share screenshots of your configuration?
Hi Tess,
This issue was eventually resolved on 12/26 by our dev ops team removing the custom domain name we use and recreating it. It seems today that we're experiencing the same error again. We will likely try the same trick again to temporarily fix, but we need someone on Retool's end to investigate why this keeps happening.
Edit/Update:
My report of issues occurring today was a false alarm. I encountered a similar looking, but entirely separate (internal) issue with a test account today.
We still believe there was a legitimate issue on Retool's end on 12/26 that caused us to have to cycle our custom embed domain, but it is unclear if that was a one-time problem or something we may encounter periodically.
I am on the same team as Dennis.
Tess, my theory is that there was an issue with SSL generation for the nginx proxy server. The cert retool creates is from letsencrypt and we had an updated config that restricted cert creation from third parties. And whatever maintenance jobs are occurring on retool side failed and caused this issue.
If possible, can you check whatever automation you have for nginx proxy creation for custom retool domains and check for failures to support/contradict my theory.
Hi @dennis_nimblerx & @gdprx,
Thanks for all of this info & thanks for your patience here. I looked into this internally -
@gdprx Yes, your theory is correct. Your domain had a CAA record that prevented us from renewing your letsencrypt certificate.
It looks like this was caught by our internal monitoring, but our team saw that you had already registered the domain again.