Intermittent connection issue (timeout) to an SSH-tunneled Postgres resource

💬 Self Hosted Retool
1
  1. My goal: Get our Retool apps to reliably load their queries without random 60-second timeouts.
  2. Issue: When an app fires multiple queries in parallel on page load, a random subset of them hangs and returns HTTP 504 at exactly 60 seconds. Number of failures per load is random. Sometimes 0, sometimes up to 4 out of 6. And the failing query names shuffle each time. Refresh sometimes fixes it, sometimes not. Only affects one Postgres resource, which is the only resource that uses the built-in SSH tunnel.
  3. Retool version & hosting setup (Docker, K8s, cloud provider, etc.): Self-hosted via docker-compose on a single Ubuntu EC2 host. Retool v3.284.0.
  4. Error message(s) or screenshots: The 504 response body is the bare nginx page, not a Retool error
  5. What I’ve tried so far:

  • Ran the exact same query via psql through the same SSH tunnel: returns in ~40 ms every time

  • During a failure, the SSH bastion sees no connection attempts for the failed queries. They never leave the Retool host

  • Other Postgres resources in the same VPC as the Retool instance (no SSH tunnel) never show this problem

  • Audited the VPC setup in both AWS accounts. It's a simple one-VPC/one-NAT-per-account layout, nothing per-AZ that would cause intermittent partial failures. Route tables, NACLs, and security groups all check out.

2 Likes
3

MAJOR UPDATE. Found something in our logs. During the failure, our Retool api container throws these two errors:

SSH Tunnel encountered an error
Error: bind EADDRINUSE 127.0.0.1:41747


SSH Tunnel encountered an error
Error: This socket has been ended by the other party

Looks like two SSH tunnels tried to grab the same local port at the same time, and one of them crashed before it could actually reach out to our bastion. That would also explain why our bastion sees no connection attempts for the failed queries. Retool never got that far.

Is this a known bug? Any setting we can flip to keep it from happening, or a version we should upgrade to?

4

Hi @reeeeetool ! Thanks for sharing these details with us, and sorry to hear that you are experiencing this connection issue.

I checked in with the engineering team, and it looks like there is a PR that introduced a new env var DBCONNECTOR_POSTGRES_SSH_POOL_MAX_SIZE that was shipped for cloud v3.385.0, but is unavailable for on-prem at the moment.

This env var should address SSH tunneling slowness and random timeouts on PostgreSQL resources connected via SSH tunnel.

There is an internal request to add this to the latest stable on-prem release, but I'm uncertain how long it takes for the build to get released. I'll be sure to circle back once I get updates!

In the meantime, I would recommend de-parallelizing queries against that resource that uses the SSH tunnel, or combine/consolidate your queries if possible. For example, you can try using the "Run after" dependencies, or trigger some queries on an event shortly after page load, instead of firing all 6 of your queries at the same time.

Hope this helps! Let me know if you need anything else at this time!

-Jen