All Database Queries Time Out

Hey all,
since this morning I have time out errors with all my workflows / apps which are using a specific database connection.

If I run the same queries via another SQL client then everything works fine. Also the user exist and there was no other change on the DB level for a while.

When I checked the connections in Retool, I saw that all DB connections were set to 10 sec time out, even tho most of them had 60 sec before. I changed them to 120 sec and still they are failing.

Does anyone have an idea how I can fix this or if there is a issue on Retool side? Also the "logs" are not really helpful, as they show the following:

invoice_list failed (120.324s):Query timed out after 120000ms

from invoice_list response(invoice_list)

in invoice_list.trigger()(invoice_list)

environment: "production"

on load

Every help is appreciated.

Thanks,
Alex

Hey Alex!

Are you using config vars in your resource setup by any chance?
If so you need to make sure the setup is: {{ environment.variables.YOUR_VAR_NAME }}

We've raised an incident for this and engineering be releasing the fix with today's release to handle other syntax.

Otherwise, if not using config vars:

  • Could you please DM me screenshots of your full resource setup?
  • If you test connection in the resource, does it also time out?
  • Did you make any changes to the DB recently (having it require SSL etc) (assuming not but just want to make sure)
1 Like

Alright, I found the reason.

We have setup another database connection yesterday which was pointed as outbound resource to eu-central-1 Frankfurt.
The problematic connection was set to the default (Oregon).

I have changed the Oregon one to Frankfurt as well, and now everything works fine.

TIL; Retool doesn't allow two different outbound connections...

Thanks for confirming. It should allow multiple resources connected to different regions, is it possible that your DB is only whitelisted for the EU IP addresses? (i.e the default Oregon IPs were being blocked by the DB server?)

this would also explain why SQL clients were working as you are probably already on the allowlist on a VPN or office wifi etc.

For reference: Configure Retool outbound regions for Retool Cloud | Retool Docs (outbound IPs here i.e Europe)

Original (US IPs): Retool Cloud IP addresses | Retool Docs

Hey Isaac,
the "broken" connection was running via Oregon for over a year without an issue (and the IPs were whitelisted as well).

Yesterday we have added the new one, routing via Frankfurt.

It was resolved with adding Frankfurt to the broken one and adding the Frankfurt related IP to the whitelist on the "broken" DB.
So it seems that multiple resources connected to different regions doesn't work, or at least not for us. :smiley:

Best,
Alex

1 Like

roger that, thanks. I'll take a look into this sounds like a regression with the lastest release (given we rolled out a new version yesterday). If I can reproduce, I'll file an incident to raise this with the team immediately. Appreciate you flagging it!

1 Like

@shudayoDE are you using MySQL or MSSQL by any chance? I saw some latency in Datadog for these resources, and since had a report from another user who was experiencing latency for MySQL, also using Oregon, which seems to now have resolved itself.

I was unable to reproduce this, so I'm assuming it's potentially an issue with one of those resources in combination with the outbound region.

I'm curious if Oregon now works for you again?

Both DBs are Postgres.
I didn't change it back to Oregon as Frankfurt works well for us now.

Also Frankfurt should be a bit better in latency since we are in Germany anyways.

Cheers!

Gotcha, thanks for confirming.

If you do have time to test this out real quick with a new resource, we'd appreciate it, as it would be helpful for our engineering team!
But no worries if not, I've raised an incident to investigate the root cause of this and some other latency we've been seeing this morning.

Cheers,
Isaac