Hello,
I had been running a local self-hosted retool instance, but was suddenly unable to connect starting today, even from the local machine itself. I am using a custom SSL certificate (following the instructions at Configure SSL and custom certificates | Retool Docs) with the nginx image instead of the https-portal image (as instructed). I had been able to connect to and use Retool yesterday but was unable to connect starting today. When I checked the status of the Docker containers and their logs, everything appeared to be running fine.
When I revert back to using https-portal and the default automatically generated SSL certificate it is possible to connect, but when using nginx with our certificate, I am unable to connect, both from remote and on the system running retool.
Log from the nginx container:
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/08/04 17:56:51 [notice] 1#1: using the "epoll" event method
2023/08/04 17:56:51 [notice] 1#1: nginx/1.25.1
2023/08/04 17:56:51 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2023/08/04 17:56:51 [notice] 1#1: OS: Linux 4.18.0-477.15.1.el8_8.x86_64
2023/08/04 17:56:51 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/08/04 17:56:51 [notice] 1#1: start worker processes
2023/08/04 17:56:51 [notice] 1#1: start worker process 20
2023/08/04 17:56:51 [notice] 1#1: start worker process 21
2023/08/04 17:56:51 [notice] 1#1: start worker process 22
2023/08/04 17:56:51 [notice] 1#1: start worker process 23
2023/08/04 17:56:51 [notice] 1#1: start worker process 24
2023/08/04 17:56:51 [notice] 1#1: start worker process 25
Output of sudo netstat -lntp indicates that the machine is listening on ports 80 and 443, as it should be:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 38737/conmon
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1365/sshd
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 18860/conmon
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 38737/conmon
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2631/conmon
tcp6 0 0 :::22 :::* LISTEN 1365/sshd
Contents of nginx/nginx.conf (server_name obfuscated to retool.mydomain.com):
server {
listen 80;
server_name retool.mydomain.com;
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
server_name retool.mydomain.com;
ssl_certificate /etc/nginx/certs/myretool_ssl.crt;
ssl_certificate_key /etc/nginx/certs/myretool_ssl.key;
location / {
proxy_pass http://api:3000;
}
}
When entering the nginx container and checking /etc/nginx/certs the relevant files are present.
nginx configuration in docker-compose.yml:
nginx:
image: nginx:latest
ports:
- "80:80"
- "443:443"
command: [nginx-debug, "-g", "daemon off;"] # Improve error logging in the container
volumes:
- ./nginx:/etc/nginx/conf.d
- ./certs:/etc/nginx/certs
links:
- api
depends_on:
- api
networks:
- frontend-network
When attempting to connect to port 80 or 443 form the machine itself (i.e., localhost:80, 127.0.0.1:80, localhost:443, 127.0.0.1:443, or retool.mydomain.com) using curl -v I get the following:
> curl -v localhost:80
* Rebuilt URL to: localhost:80/
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,mydomain.com'
* Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 80 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to localhost port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 80: Connection refused
> curl -v localhost:443
* Rebuilt URL to: localhost:443/
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,mydomain.com'
* Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 443 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 443 failed: Connection refused
* Failed to connect to localhost port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 443: Connection refused
However calling curl on localhost:3000 (the Retool API container's port) succeeds and pulls the homepage. I did notice that using curl on 0.0.0.0:80 or 0.0.0.0.443 produces an ERR_ACCESS_DENIED from my DNS/proxy (managed by the network the machine is on).
If it is relevant, the system is running RHEL 8 and using Podman instead of Docker.
Help would be greatly appreciated!