-
My goal: I need to migrate Postgres DB to an external Azure Database for Postgres
-
Issue: After updating docker.env with the credentials of the external DB all containers (except for postgres, retooldb-postgres and nginx) are in infinite restarting loop
-
Setup: Retool 3.196.11-stable self hosted on Azure VM
-
Error message(s) or screenshots:
-
What I’ve tried so far:
I have successfully migrated the data from postgres container to the external DB.
There is no connection issue, I am able to query the DB from the VM:
Also tried to addPOSTGRES_SSL_REJECT_UNAUTHORIZED=falseto docker.env and it didn't help. The application works when pointing to the container DB and it doesn't when pointing to the external one.
idk if it'll fix this, but that warning "psql major version 16, server major version 17." means the client (Retool, in this case) is using psql version 16.9 while your external database is running version 17.5. Normally the solution is to upgrade the client, but we can't do that so if you wanna make the warning go away you'll need to downgrade the server version of psql from 17.5 to 16.9.
for the infinite restarting loop though, I think it's cause by at least 1 of those containers trying to use some feature or syntax from version 16 that got changed in 17 causing that container and any others on down the pipeline to fail. The auto-restarting is on purpose to minimize any down time of the service (the alternative is to hope and pray you catch the container failing as soon as it happens to manually restart it 
)
1 Like
The issue is not with the server major version, but apparently api is not able to establish an SSL connection.
When Azure PostgreSQL server parameter require_secure_transport is OFF all containers but code-executor-1 stay alive and I can even open the application in browser, but as soon as I turn require_secure_transport ON everything breaks down.
I've downloaded Azure Certificates, bundled them and mounted to api container.
services:
api:
build:
context: .
env_file: docker.env
environment:
- SERVICE_TYPE=MAIN_BACKEND,DB_CONNECTOR,DB_SSH_CONNECTOR
ports:
- 3000:3000
networks:
- frontend
- backend
- code-executor
depends_on:
- postgres
volumes:
- ./ca_postgres/azure_postgres_bundle.crt:/retool_backend/ca/azure_postgres_bundle.crt
- ./ca_postgres/azure_postgres_bundle.crt:/etc/ssl/certs/azure_postgres_bundle.crt
# - ./ca_postgres/DigiCertGlobalRootCA.crt:/etc/ssl/certs/DigiCertGlobalRootCA.crt
# - ./ca_postgres/DigiCertGlobalRootG2.crt.pem:/etc/ssl/certs/DigiCertGlobalRootG2.crt.pem
# - ./ca_postgres/MicrosoftRSARootCertificateAuthority2017.crt:/etc/ssl/certs/MicrosoftRSARootCertificateAuthority2017.crt
restart: always
NODE_ENV=production
NODE_EXTRA_CA_CERTS=/retool_backend/ca/azure_postgres_bundle.crt
POSTGRES_SSL_REQUIRE=true
POSTGRES_SSL_CA_CERT=/etc/ssl/certs/azure_postgres_bundle.crt
POSTGRES_SSL_REJECT_UNAUTHORIZED=false
Unfortunately, no success. I get:
api-1 | Failing checking database migrations
api-1 | /retool_backend/node_modules/.pnpm/sequelize@6.31.0_patch_hash=dk6qy5gyzbjysuc4vxu23jnkrm_mysql2@2.3.3_oracledb@5.3.0_snowflake-_geltwdb3fqjepyd2eep3x2qxtu/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:146
api-1 | reject(new sequelizeErrors.ConnectionError(err));
api-1 | ^
api-1 |
api-1 | ConnectionError [SequelizeConnectionError]: no pg_hba.conf entry for host "10.1.106.212", user "pgadmin", database "retoolapp", no encryption
api-1 | at Client._connectionCallback (/retool_backend/node_modules/.pnpm/sequelize@6.31.0_patch_hash=dk6qy5gyzbjysuc4vxu23jnkrm_mysql2@2.3.3_oracledb@5.3.0_snowflake-_geltwdb3fqjepyd2eep3x2qxtu/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:146:24)
api-1 | at Client._handleErrorWhileConnecting (/retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/client.js:364:19)
api-1 | at Client._handleErrorMessage (/retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/client.js:384:19)
api-1 | at Connection.emit (node:events:518:28)
api-1 | at /retool_backend/node_modules/.pnpm/pg@8.11.3_patch_hash=5z2kru6wp55sob2v2ec4ejgszq/node_modules/pg/lib/connection.js:117:12
api-1 | at Parser.parse (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:40:17)
api-1 | at Socket.<anonymous> (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/index.js:11:42)
api-1 | at Socket.emit (node:events:518:28)
api-1 | at addChunk (node:internal/streams/readable:561:12)
api-1 | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) {
api-1 | parent: error: no pg_hba.conf entry for host "10.1.106.212", user "pgadmin", database "retoolapp", no encryption
api-1 | at Parser.parseErrorMessage (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:294:98)
api-1 | at Parser.handlePacket (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:126:29)
api-1 | at Parser.parse (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/parser.js:39:38)
api-1 | at Socket.<anonymous> (/retool_backend/node_modules/.pnpm/pg-protocol@1.6.0_patch_hash=bc532sl7pnvzzwfziu2qpbx674/node_modules/pg-protocol/dist/index.js:11:42)
api-1 | at Socket.emit (node:events:518:28)
api-1 | at addChunk (node:internal/streams/readable:561:12)
api-1 | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
api-1 | at Readable.push (node:internal/streams/readable:392:5)
api-1 | at TCP.onStreamRead (node:internal/stream_base_commons:191:23) {
api-1 | length: 160,
api-1 | severity: 'FATAL',
api-1 | code: '28000',
api-1 | detail: undefined,
api-1 | hint: undefined,
api-1 | position: undefined,
api-1 | internalPosition: undefined,
api-1 | internalQuery: undefined,
api-1 | where: undefined,
api-1 | schema: undefined,
api-1 | table: undefined,
api-1 | column: undefined,
api-1 | dataType: undefined,
api-1 | constraint: undefined,
api-1 | file: 'auth.c',
api-1 | line: '630',
api-1 | routine: 'ClientAuthentication'
api-1 | },
Additionally, I have checked whether node pg module from retool_backend is able to connect to the database. And it does connect both ways: when require_secure_transport is OFF and ON.
retool@Retool-VM:~/retool-onpremise$ sudo docker compose exec api env POSTGRES_HOST="$POSTGRES_HOST" POSTGRES_PORT="$POSTGRES_PORT" POSTGRES_USER="$POSTGRES_USER" POSTGRES_PASSWORD="$POSTGRES_PASSWORD" POSTGRES_DB="$POSTGRES_DB" POSTGRES_SSL_REQUIRE="$POSTGRES_SSL_REQUIRE" POSTGRES_SSL_CA_CERT="$POSTGRES_SSL_CA_CERT" POSTGRES_SSL_REJECT_UNAUTHORIZED="$POSTGRES_SSL_REJECT_UNAUTHORIZED" NODE_EXTRA_CA_CERTS="$NODE_EXTRA_CA_CERTS" node /retool_backend/test_db_connection.js
Attempting connection with config: {
host: '10.1.106.228',
port: 5432,
user: 'pgadmin',
password: '[REDACTED]',
database: 'retoolapp',
ssl: { rejectUnauthorized: false, ca: 'CA_CERT_PATH_PROVIDED' }
}
SUCCESS: Successfully connected to PostgreSQL database!
Query result: {
current_user: 'pgadmin',
current_database: 'retoolapp',
version: 'PostgreSQL 16.9 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.2.0, 64-bit'
}
Connection closed.
But api itself which is supposed to use the same module is never able to connect with SSL.
Some additional remarks here.
-
Not sure if
depends_on: - postgresis stil needed in compose.yaml. -
code-executor-1 exits with code 1 regardless of SSL but that seems to be another issue...
code-executor-1 | {"level":"info","message":"Configuring StatsD for development...","timestamp":"2025-07-17T17:31:54.135Z"}
code-executor-1 | {"level":"info","message":"Installing request logging middleware...","timestamp":"2025-07-17T17:31:54.142Z"}
code-executor-1 | {"level":"info","message":"Installing request logging middleware...","timestamp":"2025-07-17T17:31:54.143Z"}
code-executor-1 | {"level":"info","message":"Checking if nsjail can be used","timestamp":"2025-07-17T17:31:54.145Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][1] initCloneNs():382 mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[F][2025-07-17T17:31:54+0000][1] runChild():483 Launching child process failed","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][24] standaloneMode():275 Couldn't launch the child process","timestamp":"2025-07-17T17:31:54.178Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"info","message":"cleaning up job - /tmp/jobs/1a9dae09-d8ec-47bd-813d-0de57770043b","timestamp":"2025-07-17T17:31:54.182Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.187Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.187Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"3081305015324953560","trace_id":"7379302178545120928"},"level":"error","message":"Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal","stack":"Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND host.docker.internal\n at handleCallback (/retool/node_modules/.pnpm/hot-shots@9.3.0/node_modules/hot-shots/lib/statsd.js:358:32)\n at process.processTicksAndRejections (node:internal/process/task_queues:81:21)","timestamp":"2025-07-17T17:31:54.195Z"}
code-executor-1 | {"dd":{"env":"development","service":"code_executor_service","span_id":"7379302178545120928","trace_id":"7379302178545120928"},"level":"error","message":"[E][2025-07-17T17:31:54+0000][1] initCloneNs():382 mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied\n[F][2025-07-17T17:31:54+0000][1] runChild():483 Launching child process failed\n[E][2025-07-17T17:31:54+0000][24] standaloneMode():275 Couldn't launch the child process","timestamp":"2025-07-17T17:31:54.196Z"}
code-executor-1 | {"level":"info","message":"can use nsjail: false","timestamp":"2025-07-17T17:31:54.196Z"}
code-executor-1 | {"level":"error","message":"Error when executing code in sandbox. Is the container configured with privileged and cloneUserNs permissions? If you do not need code sandboxing, set ALLOW_UNSAFE_CODE_EXECUTION='true' environment variable.","timestamp":"2025-07-17T17:31:54.196Z"}
workflows-backend-1 | {"level":"info","message":"Not configuring StatsD...","timestamp":"2025-07-17T17:31:54.851Z"}
workflows-worker-1 | {"level":"info","message":"Not configuring StatsD...","timestamp":"2025-07-17T17:31:54.944Z"}
code-executor-1 exited with code 1