-
My goal: I need to migrate Postgres DB to an external Azure Database for Postgres
-
Issue: After updating docker.env with the credentials of the external DB all containers (except for postgres, retooldb-postgres and nginx) are in infinite restarting loop
-
Setup: Retool 3.196.11-stable self hosted on Azure VM
-
Error message(s) or screenshots:
-
What I’ve tried so far:
I have successfully migrated the data from postgres container to the external DB.
There is no connection issue, I am able to query the DB from the VM:
Also tried to addPOSTGRES_SSL_REJECT_UNAUTHORIZED=falseto docker.env and it didn't help. The application works when pointing to the container DB and it doesn't when pointing to the external one.
idk if it'll fix this, but that warning "psql major version 16, server major version 17." means the client (Retool, in this case) is using psql version 16.9 while your external database is running version 17.5. Normally the solution is to upgrade the client, but we can't do that so if you wanna make the warning go away you'll need to downgrade the server version of psql from 17.5 to 16.9.
for the infinite restarting loop though, I think it's cause by at least 1 of those containers trying to use some feature or syntax from version 16 that got changed in 17 causing that container and any others on down the pipeline to fail. The auto-restarting is on purpose to minimize any down time of the service (the alternative is to hope and pray you catch the container failing as soon as it happens to manually restart it 
)
1 Like
The issue is not with the server major version, but apparently api is not able to establish an SSL connection.
When Azure PostgreSQL server parameter require_secure_transport is OFF all containers but code-executor-1 stay alive and I can even open the application in browser, but as soon as I turn require_secure_transport ON everything breaks down.
I've downloaded Azure Certificates, bundled them and mounted to api container.
services:
api:
build:
context: .
env_file: docker.env
environment:
- SERVICE_TYPE=MAIN_BACKEND,DB_CONNECTOR,DB_SSH_CONNECTOR
ports:
- 3000:3000
networks:
- frontend
- backend
- code-executor
depends_on:
- postgres
volumes:
- ./ca_postgres/azure_postgres_bundle.crt:/retool_backend/ca/azure_postgres_bundle.crt
- ./ca_postgres/azure_postgres_bundle.crt:/etc/ssl/certs/azure_postgres_bundle.crt
# - ./ca_postgres/DigiCertGlobalRootCA.crt:/etc/ssl/certs/DigiCertGlobalRootCA.crt
# - ./ca_postgres/DigiCertGlobalRootG2.crt.pem:/etc/ssl/certs/DigiCertGlobalRootG2.crt.pem
# - ./ca_postgres/MicrosoftRSARootCertificateAuthority2017.crt:/etc/ssl/certs/MicrosoftRSARootCertificateAuthority2017.crt
restart: always
NODE_ENV=production
NODE_EXTRA_CA_CERTS=/retool_backend/ca/azure_postgres_bundle.crt
POSTGRES_SSL_REQUIRE=true
POSTGRES_SSL_CA_CERT=/etc/ssl/certs/azure_postgres_bundle.crt
POSTGRES_SSL_REJECT_UNAUTHORIZED=false
Unfortunately, no success. I get:
api-1 | Failing checking database migrations
api-1 | /retool_backend/node_modules/.pnpm/sequelize@6.31.0_patch_hash=dk6qy5gyzbjysuc4vxu23jnkrm_mysql2@2.3.3_oracledb@5.3.0_snowflake-_geltwdb3fqjepyd2eep3x2qxtu/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:146
api-1 | reject(new sequelizeErrors.ConnectionError(err));
api-1 | ^
api-1 |
api-1 | ConnectionError [SequelizeConnectionError]: no pg_hba.conf entry for host "10.1.106.212", user "pgadmin", database "retoolapp", no encryption
api-1 | at Client._connectionCallback (/retool_backend/node_modules/.pnpm/sequelize@6.31.0_patch_hash=dk6qy5gyzbjysuc4vxu23jnkrm_mysql2@2.3.3_oracledb@5.3.0_snowflake-_geltwdb3fqjepyd2eep3x2qxtu/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:146:24)
api-1 | at Client._handleErrorWhileConnecting (/retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/client.js:364:19)
api-1 | at Client._handleErrorMessage (/retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/client.js:384:19)
api-1 | at Connection.emit (node:events:518:28)
api-1 | at /retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/connection.js:117:12
api-1 | at Parser.parse (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:40:17)
api-1 | at Socket.<anonymous> (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/index.js:11:42)
api-1 | at Socket.emit (node:events:518:28)
api-1 | at addChunk (node:internal/streams/readable:561:12)
api-1 | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) {
api-1 | parent: error: no pg_hba.conf entry for host "10.1.106.212", user "pgadmin", database "retoolapp", no encryption
api-1 | at Parser.parseErrorMessage (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:294:98)
api-1 | at Parser.handlePacket (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:126:29)
api-1 | at Parser.parse (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:39:38)
api-1 | at Socket.<anonymous> (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/index.js:11:42)
api-1 | at Socket.emit (node:events:518:28)
api-1 | at addChunk (node:internal/streams/readable:561:12)
api-1 | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
api-1 | at Readable.push (node:internal/streams/readable:392:5)
api-1 | at TCP.onStreamRead (node:internal/stream_base_commons:191:23) {
api-1 | length: 160,
api-1 | severity: 'FATAL',
api-1 | code: '28000',
api-1 | detail: undefined,
api-1 | hint: undefined,
api-1 | position: undefined,
api-1 | internalPosition: undefined,
api-1 | internalQuery: undefined,
api-1 | where: undefined,
api-1 | schema: undefined,
api-1 | table: undefined,
api-1 | column: undefined,
api-1 | dataType: undefined,
api-1 | constraint: undefined,
api-1 | file: 'auth.c',
api-1 | line: '630',
api-1 | routine: 'ClientAuthentication'
api-1 | },
Additionally, I have checked whether node pg module from retool_backend is able to connect to the database. And it does connect both ways: when require_secure_transport is OFF and ON.
retool@Retool-VM:~/retool-onpremise$ sudo docker compose exec api env POSTGRES_HOST="$POSTGRES_HOST" POSTGRES_PORT="$POSTGRES_PORT" POSTGRES_USER="$POSTGRES_USER" POSTGRES_PASSWORD="$POSTGRES_PASSWORD" POSTGRES_DB="$POSTGRES_DB" POSTGRES_SSL_REQUIRE="$POSTGRES_SSL_REQUIRE" POSTGRES_SSL_CA_CERT="$POSTGRES_SSL_CA_CERT" POSTGRES_SSL_REJECT_UNAUTHORIZED="$POSTGRES_SSL_REJECT_UNAUTHORIZED" NODE_EXTRA_CA_CERTS="$NODE_EXTRA_CA_CERTS" node /retool_backend/test_db_connection.js
Attempting connection with config: {
host: '10.1.106.228',
port: 5432,
user: 'pgadmin',
password: '[REDACTED]',
database: 'retoolapp',
ssl: { rejectUnauthorized: false, ca: 'CA_CERT_PATH_PROVIDED' }
}
SUCCESS: Successfully connected to PostgreSQL database!
Query result: {
current_user: 'pgadmin',
current_database: 'retoolapp',
version: 'PostgreSQL 16.9 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.2.0, 64-bit'
}
Connection closed.
But api itself which is supposed to use the same module is never able to connect with SSL.
Some additional remarks here.
-
Not sure if
depends_on: - postgresis stil needed in compose.yaml. -
code-executor-1 exits with code 1 regardless of SSL but that seems to be another issue...
code-executor-1 | {"level":"info","message":"Configuring StatsD for development...","timestamp":"2025-07-17T17:31:54.135Z"}
code-executor-1 | {"level":"info","message":"Installing request logging middleware...","timestamp":"2025-07-17T17:31:54.142Z"}
code-executor-1 | {"level":"info","message":"Installing request logging middleware...","timestamp":"2025-07-17T17:31:54.143Z"}
code-executor-1 | {"level":"info","message":"Checking if nsjail can be used","timestamp":"2025-07-17T17:31:54.145Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][1] initCloneNs():382 mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[F][2025-07-17T17:31:54+0000][1] runChild():483 Launching child process failed","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][24] standaloneMode():275 Couldn't launch the child process","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"info","message":"cleaning up job - /tmp/jobs/1a9dae09-d8ec-47bd-813d-0de57770043b","timestamp":"2025-07-17T17:31:54.182Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.187Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.187Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.195Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][1] initCloneNs():382 mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied\n[F][2025-07-17T17:31:54+0000][1] runChild():483 Launching child process failed\n[E][2025-07-17T17:31:54+0000][24] standaloneMode():275 Couldn't launch the child process","timestamp":"2025-07-17T17:31:54.196Z"}
code-executor-1 | {"level":"info","message":"can use nsjail: false","timestamp":"2025-07-17T17:31:54.196Z"}
code-executor-1 | {"level":"error","message":"Error when executing code in sandbox. Is the container configured with privileged and cloneUserNs permissions? If you do not need code sandboxing, set ALLOW_UNSAFE_CODE_EXECUTION='true' environment variable.","timestamp":"2025-07-17T17:31:54.196Z"}
workflows-backend-1 | {"level":"info","message":"Not configuring StatsD...","timestamp":"2025-07-17T17:31:54.851Z"}
workflows-worker-1 | {"level":"info","message":"Not configuring StatsD...","timestamp":"2025-07-17T17:31:54.944Z"}
code-executor-1 exited with code 1
After long long chats with Gemini, I have finally found the solution:
To enable SSL connection from API to your Postgres you need to add POSTGRES_SSL_ENABLED=true variable!
Alternatively you can add ?sslmode=require to your DATABASE_URL.
POSTGRES_SSL_REQUIRE=true alone has no effect in Retool environment.
I'd have saved days of research if this one line had been in the How-To documentation. Nevertheless, I found at least the description of environment variables here Storage database environment variables | Retool Docs (although too late).
There is no need to provide Microsoft certificates for Azure Database for Postgres, those are already available in container's /etc/ssl/certs/. You also cannot connect to Postgres instance by private IP with SSL, i.e. you must have a private DNS name for it and Vnet configured to use Azure provided DNS service.
Whether depends_on: - postgres in compose.yaml and postgres container itself are still needed, stays unclear. I removed them and the app seems to work.
But you need to adjust temporal.yaml, otherwise workflows-worker container won't work. I've configured it like this:
environment:
# To enable TLS between temporal and external postgres, set both below variables to true
- SQL_TLS_ENABLED=true
- SQL_TLS=true
# Defined twice because temporal-server and temporal-sql-tool use different envvars
- SQL_TLS_DISABLE_HOST_VERIFICATION=false
- SQL_HOST_VERIFICATION=true
And if anyone knows how to bring code-executor to life, let me know, but it probably needs a separate post.
2 Likes
Did you figure out what was going on with the code executor? In the same boat after a migration to external database. It looks like it's using the incorrect host still.
Yes, I did figure out that it doesn't work because of the privileged mode, it has nothing to do with the database. I have a plain vanilla Ubuntu where AppArmor security module will not allow you to run a container in privileged mode and you cannot reconfigure AppArmor for Docker. There are two options: either disable AppArmor completely which is a security risk for the entire machine or follow Retool's documentation and try to disable privileged mode for the container. Spoiler: it doesn't work. That's basically the point where we stopped testing Retool because the deployment process is a quest of unfolding new and new issues.