Rebooted AWS instance and can not access ReTool

mrmacph · January 20, 2022, 5:11pm

Hi,

I just rebooted the EC2 instance that I have Retool running on and can not access the webui anymore (by either domain or IP). Looks like a couple of the docker containers are not successfully starting. Currently running just the trail version as we evaluate the product.

 sudo docker-compose ps
               Name                              Command               State               Ports            
------------------------------------------------------------------------------------------------------------
retool-onpremise_api_1                docker-entrypoint.sh bash  ...   Exit 0                               
retool-onpremise_db-connector_1       docker-entrypoint.sh bash  ...   Up       3000/tcp, 3001/tcp, 3002/tcp
retool-onpremise_db-ssh-connector_1   docker-entrypoint.sh bash  ...   Up       3000/tcp, 3001/tcp, 3002/tcp
retool-onpremise_jobs-runner_1        docker-entrypoint.sh bash  ...   Exit 1                               
retool-onpremise_postgres_1           docker-entrypoint.sh postgres    Up       5432/tcp

Kabirdas · January 22, 2022, 1:04am

Hi @mrmacph

Can you check the logs for those containers that aren't starting by running either docker-compose logs (for all containers) or docker logs (for a specific container)?

mrmacph · January 24, 2022, 5:20pm

Looks like all the errors are related to a password authentication issue for "retool_internal_user", logs below. Thanks.

api_1               | wait-for-it.sh: waiting 15 seconds for postgres:5432
api_1               | wait-for-it.sh: postgres:5432 is available after 0 seconds
api_1               | not untarring the bundle
api_1               | [process service types] [ 'MAIN_BACKEND' ]
api_1               | Failing checking database migrations
api_1               | (node:22) UnhandledPromiseRejectionWarning: SequelizeConnectionError: password authentication failed for user "retool_internal_user"
api_1               |     at /snapshot/backend/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:182:24
api_1               |     at Connection.connectingErrorHandler (/snapshot/backend/node_modules/pg/lib/client.js:194:14)
api_1               |     at Connection.emit (events.js:315:20)
api_1               |     at Socket.<anonymous> (/snapshot/backend/node_modules/pg/lib/connection.js:134:12)
api_1               |     at Socket.emit (events.js:315:20)
api_1               |     at addChunk (_stream_readable.js:295:12)
api_1               |     at readableAddChunk (_stream_readable.js:271:9)
api_1               |     at Socket.Readable.push (_stream_readable.js:212:10)
api_1               |     at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
api_1               | (node:22) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
api_1               | (node:22) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
api_1               | wait-for-it.sh: waiting 15 seconds for postgres:5432
api_1               | wait-for-it.sh: postgres:5432 is available after 0 seconds
api_1               | not untarring the bundle
api_1               | [process service types] [ 'MAIN_BACKEND' ]
api_1               | Failing checking database migrations
api_1               | (node:24) UnhandledPromiseRejectionWarning: SequelizeConnectionError: password authentication failed for user "retool_internal_user"
api_1               |     at /snapshot/backend/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:182:24
api_1               |     at Connection.connectingErrorHandler (/snapshot/backend/node_modules/pg/lib/client.js:194:14)
api_1               |     at Connection.emit (events.js:315:20)
api_1               |     at Socket.<anonymous> (/snapshot/backend/node_modules/pg/lib/connection.js:134:12)
api_1               |     at Socket.emit (events.js:315:20)
api_1               |     at addChunk (_stream_readable.js:295:12)
api_1               |     at readableAddChunk (_stream_readable.js:271:9)
api_1               |     at Socket.Readable.push (_stream_readable.js:212:10)
api_1               |     at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
api_1               | (node:24) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
api_1               | (node:24) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

jobs-runner_1       | wait-for-it.sh: waiting 15 seconds for postgres:5432
jobs-runner_1       | wait-for-it.sh: postgres:5432 is available after 0 seconds
jobs-runner_1       | not untarring the bundle
jobs-runner_1       | [process service types] [ 'JOBS_RUNNER' ]
jobs-runner_1       | Failing checking database migrations
jobs-runner_1       | 
jobs-runner_1       | ---------------------------
jobs-runner_1       | -     EXITING RETOOL      -
jobs-runner_1       | ---------------------------
jobs-runner_1       | 
jobs-runner_1       | Error running database migrations: SequelizeConnectionError: password authentication failed for user "retool_internal_user"
jobs-runner_1       | wait-for-it.sh: waiting 15 seconds for postgres:5432
jobs-runner_1       | wait-for-it.sh: postgres:5432 is available after 0 seconds
jobs-runner_1       | not untarring the bundle
jobs-runner_1       | [process service types] [ 'JOBS_RUNNER' ]
jobs-runner_1       | Failing checking database migrations
jobs-runner_1       | 
jobs-runner_1       | ---------------------------
jobs-runner_1       | -     EXITING RETOOL      -
jobs-runner_1       | ---------------------------
jobs-runner_1       | 
jobs-runner_1       | Error running database migrations: SequelizeConnectionError: password authentication failed for user "retool_internal_user"

db-ssh-connector_1  | [deprecated] do nothing.
db-ssh-connector_1  | [process service types] [ 'DB_SSH_CONNECTOR' ]
db-ssh-connector_1  | Setting http and https agent maxSockets to 25
db-ssh-connector_1  | Starting database connector server on port 3002
db-ssh-connector_1  | Running node v12.18.1
db-ssh-connector_1  | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
db-ssh-connector_1  | Thu, 20 Jan 2022 16:35:20 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-ssh-connector_1  | Thu, 20 Jan 2022 16:35:20 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
db-ssh-connector_1  | dbconnector boot-up: 4565.815ms
db-ssh-connector_1  | [deprecated] do nothing.
db-ssh-connector_1  | [process service types] [ 'DB_SSH_CONNECTOR' ]
db-ssh-connector_1  | Setting http and https agent maxSockets to 25
db-ssh-connector_1  | Starting database connector server on port 3002
db-ssh-connector_1  | Running node v12.18.1
db-ssh-connector_1  | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
db-ssh-connector_1  | Thu, 20 Jan 2022 16:57:36 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-ssh-connector_1  | Thu, 20 Jan 2022 16:57:36 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
db-ssh-connector_1  | dbconnector boot-up: 4840.147ms

db-connector_1      | [process service types] [ 'DB_CONNECTOR' ]
db-connector_1      | Setting http and https agent maxSockets to 25
db-connector_1      | Starting database connector server on port 3002
db-connector_1      | Running node v12.18.1
db-connector_1      | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
db-connector_1      | Thu, 20 Jan 2022 16:35:20 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-connector_1      | Thu, 20 Jan 2022 16:35:20 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
db-connector_1      | dbconnector boot-up: 4749.382ms
db-connector_1      | [process service types] [ 'DB_CONNECTOR' ]
db-connector_1      | Setting http and https agent maxSockets to 25
db-connector_1      | Starting database connector server on port 3002
db-connector_1      | Running node v12.18.1
db-connector_1      | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
db-connector_1      | Thu, 20 Jan 2022 16:57:36 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-connector_1      | Thu, 20 Jan 2022 16:57:36 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
db-connector_1      | dbconnector boot-up: 4577.166ms

postgres_1          | 
postgres_1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1          | 
postgres_1          | 2022-01-20 16:35:05.333 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1          | 2022-01-20 16:35:05.334 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1          | 2022-01-20 16:35:05.340 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1          | 2022-01-20 16:35:05.385 UTC [25] LOG:  database system was shut down at 2022-01-20 16:20:45 UTC
postgres_1          | 2022-01-20 16:35:05.458 UTC [1] LOG:  database system is ready to accept connections
postgres_1          | 2022-01-20 16:35:05.602 UTC [32] LOG:  incomplete startup packet
postgres_1          | 2022-01-20 16:35:06.793 UTC [33] FATAL:  password authentication failed for user "retool_internal_user"
postgres_1          | 2022-01-20 16:35:06.793 UTC [33] DETAIL:  Password does not match for user "retool_internal_user".
postgres_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
postgres_1          | 2022-01-20 16:35:15.896 UTC [34] LOG:  incomplete startup packet
postgres_1          | 2022-01-20 16:35:16.971 UTC [35] FATAL:  password authentication failed for user "retool_internal_user"
postgres_1          | 2022-01-20 16:35:16.971 UTC [35] DETAIL:  Password does not match for user "retool_internal_user".
postgres_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
postgres_1          | 2022-01-20 16:57:09.109 UTC [1] LOG:  received fast shutdown request
postgres_1          | 2022-01-20 16:57:09.112 UTC [1] LOG:  aborting any active transactions
postgres_1          | 2022-01-20 16:57:09.114 UTC [1] LOG:  background worker "logical replication launcher" (PID 31) exited with exit code 1
postgres_1          | 2022-01-20 16:57:09.117 UTC [26] LOG:  shutting down
postgres_1          | 2022-01-20 16:57:09.133 UTC [1] LOG:  database system is shut down
postgres_1          | 
postgres_1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1          | 
postgres_1          | 2022-01-20 16:57:30.177 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1          | 2022-01-20 16:57:30.177 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1          | 2022-01-20 16:57:30.184 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1          | 2022-01-20 16:57:30.243 UTC [25] LOG:  database system was shut down at 2022-01-20 16:57:09 UTC
postgres_1          | 2022-01-20 16:57:30.263 UTC [1] LOG:  database system is ready to accept connections
postgres_1          | 2022-01-20 16:57:31.162 UTC [32] LOG:  incomplete startup packet
postgres_1          | 2022-01-20 16:57:31.336 UTC [33] LOG:  incomplete startup packet
postgres_1          | 2022-01-20 16:57:32.850 UTC [34] FATAL:  password authentication failed for user "retool_internal_user"
postgres_1          | 2022-01-20 16:57:32.850 UTC [34] DETAIL:  Password does not match for user "retool_internal_user".
postgres_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"
postgres_1          | 2022-01-20 16:57:33.031 UTC [35] FATAL:  password authentication failed for user "retool_internal_user"
postgres_1          | 2022-01-20 16:57:33.031 UTC [35] DETAIL:  Password does not match for user "retool_internal_user".
postgres_1          | 	Connection matched pg_hba.conf line 95: "host all all all md5"

Kabirdas · January 27, 2022, 11:21pm

On initial setup the password for retool_internal_user is stored in your docker.env file and should be set to randomstring:

Can you verify whether or not these have been changed?

jhcoinl · March 10, 2022, 8:47pm

I’m having this same issue, I changd the pw to randomstring and it still does not work.

docker logs e46f25e85cfa
wait-for-it.sh: waiting 15 seconds for postgres:5432
wait-for-it.sh: postgres:5432 is available after 0 seconds
not untarring the bundle
[process service types] [ 'MAIN_BACKEND' ]
Failing checking database migrations
(node:27) UnhandledPromiseRejectionWarning: SequelizeConnectionError: password authentication failed for user "retool_internal_user"
at /snapshot/backend/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:182:24
at Connection.connectingErrorHandler (/snapshot/backend/node_modules/pg/lib/client.js:194:14)
at Connection.emit (events.js:315:20)
at Socket. (/snapshot/backend/node_modules/pg/lib/connection.js:134:12)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at Socket.Readable.push (_stream_readable.js:212:10)
at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
(node:27) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without acatch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see Command-line API | Node.js v17.7.1 Documentation). (rejection id: 1)
(node:27) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.