Hey all,
since this morning I have time out errors with all my workflows / apps which are using a specific database connection.
If I run the same queries via another SQL client then everything works fine. Also the user exist and there was no other change on the DB level for a while.
When I checked the connections in Retool, I saw that all DB connections were set to 10 sec time out, even tho most of them had 60 sec before. I changed them to 120 sec and still they are failing.
Does anyone have an idea how I can fix this or if there is a issue on Retool side? Also the "logs" are not really helpful, as they show the following:
invoice_list failed (120.324s):Query timed out after 120000ms
from invoice_list response(invoice_list)
in invoice_list.trigger()(invoice_list)
environment: "production"
on load
We have setup another database connection yesterday which was pointed as outbound resource to eu-central-1 Frankfurt.
The problematic connection was set to the default (Oregon).
I have changed the Oregon one to Frankfurt as well, and now everything works fine.
TIL; Retool doesn't allow two different outbound connections...
Thanks for confirming. It should allow multiple resources connected to different regions, is it possible that your DB is only whitelisted for the EU IP addresses? (i.e the default Oregon IPs were being blocked by the DB server?)
this would also explain why SQL clients were working as you are probably already on the allowlist on a VPN or office wifi etc.
Hey Isaac,
the "broken" connection was running via Oregon for over a year without an issue (and the IPs were whitelisted as well).
Yesterday we have added the new one, routing via Frankfurt.
It was resolved with adding Frankfurt to the broken one and adding the Frankfurt related IP to the whitelist on the "broken" DB.
So it seems that multiple resources connected to different regions doesn't work, or at least not for us.
roger that, thanks. I'll take a look into this sounds like a regression with the lastest release (given we rolled out a new version yesterday). If I can reproduce, I'll file an incident to raise this with the team immediately. Appreciate you flagging it!
@shudayoDE are you using MySQL or MSSQL by any chance? I saw some latency in Datadog for these resources, and since had a report from another user who was experiencing latency for MySQL, also using Oregon, which seems to now have resolved itself.
I was unable to reproduce this, so I'm assuming it's potentially an issue with one of those resources in combination with the outbound region.
If you do have time to test this out real quick with a new resource, we'd appreciate it, as it would be helpful for our engineering team!
But no worries if not, I've raised an incident to investigate the root cause of this and some other latency we've been seeing this morning.