Retools failing to start after the database maintenance yesterday

I am using an on-premise deployment of retool on an unbuntu aws ec2 server. It was working fine, but am now unable to connect after the database maintenance yesterday.

Here is the results of running sudo docker-compose ps

retoolonpremise_api_1 docker-entrypoint.sh bash ... Exit 1
retoolonpremise_db-connector_1 docker-entrypoint.sh bash ... Up 3000/tcp, 3001/tcp, 3002/tcp
retoolonpremise_db-ssh-connector_1 docker-entrypoint.sh bash ... Up 3000/tcp, 3001/tcp, 3002/tcp
retoolonpremise_https-portal_1 /init Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
retoolonpremise_jobs-runner_1 docker-entrypoint.sh bash ... Up 3000/tcp, 3001/tcp, 3002/tcp
retoolonpremise_postgres_1 docker-entrypoint.sh postgres Up 5432/tcp
retoolonpremise_user-postgres_1 docker-entrypoint.sh postgres Up 5432/tcp

And here are the results of running sudo docker-compose logs --tail

api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 |
api_1 | ---------------------------
postgres_1 | ERROR: duplicate key value violates unique constraint "page_user_key"
postgres_1 | DETAIL: Key ("pageId", "userId")=(3, 2) already exists.
postgres_1 | STATEMENT: INSERT INTO "recently_visited_apps" ("userId","pageId","visitType","createdAt","updatedAt") VALUES (2,3,'view','2021-10-22 19:15:08.914 +00:00','2021-10-22 19:15:08.914 +00:00') RETURNING *;
api_1 | - EXITING RETOOL -
jobs-runner_1 | [pid: 57] Memory Stats for process: { rss: '111.607808 MB',
postgres_1 | ERROR: duplicate key value violates unique constraint "page_user_key"
api_1 | ---------------------------
postgres_1 | DETAIL: Key ("pageId", "userId")=(3, 2) already exists.
postgres_1 | STATEMENT: INSERT INTO "recently_visited_apps" ("userId","pageId","visitType","createdAt","updatedAt") VALUES (2,3,'view','2021-10-22 19:15:26.021 +00:00','2021-10-22 19:15:26.021 +00:00') RETURNING *;
postgres_1 | ERROR: duplicate key value violates unique constraint "page_user_key"
jobs-runner_1 | heapTotal: '64.069632 MB',
postgres_1 | DETAIL: Key ("pageId", "userId")=(3, 2) already exists.
api_1 |
jobs-runner_1 | heapUsed: '55.358696 MB',
jobs-runner_1 | percentageHeapUsed: '86.4%' }
api_1 | There are 93 pending database migrations. Retool will not start
jobs-runner_1 | [pid: 57] CPU Usage % for process: 0.10%
jobs-runner_1 | [pid: 57] Memory Stats for container: { freeSystemMemory: '257.163264 MB',
postgres_1 | STATEMENT: INSERT INTO "recently_visited_apps" ("userId","pageId","visitType","createdAt","updatedAt") VALUES (2,3,'view','2021-10-22 19:15:45.033 +00:00','2021-10-22 19:15:45.033 +00:00') RETURNING *;
jobs-runner_1 | totalSystemMemory: '4124.83584 MB',
api_1 | unless all database migrations have been run.
postgres_1 | ERROR: invalid input syntax for uuid: "null" at character 330
jobs-runner_1 | percentageFreeMemory: '6.23%' }
postgres_1 | STATEMENT: SELECT "id", "name", "uuid", "lastEditedBy", "photoUrl", "isGlobalWidget", "createdAt", "updatedAt", "deletedAt", "folderId", "organizationId", "releasedTagId" FROM "pages" AS "page" WHERE (("page"."deletedAt" > '2021-10-22 19:15:45.742 +00:00' OR "page"."deletedAt" IS NULL) AND ("page"."organizationId" = 1 AND "page"."uuid" = 'null')) LIMIT 1;
api_1 |
jobs-runner_1 | [pid: 57] Memory Stats for process: { rss: '111.099904 MB',
postgres_1 | LOG: database system was interrupted; last known up at 2021-10-22 22:11:46 UTC
api_1 | To run migrate the database, run the following command:
jobs-runner_1 | heapTotal: '64.069632 MB',
postgres_1 | LOG: database system was not properly shut down; automatic recovery in progress
api_1 |
jobs-runner_1 | heapUsed: '54.713848 MB',
postgres_1 | LOG: redo starts at 0/858D0458
jobs-runner_1 | percentageHeapUsed: '85.4%' }
api_1 | yarn sequelize db:migrate
postgres_1 | LOG: invalid record length at 0/858D62E0: wanted 24, got 0
jobs-runner_1 | [pid: 57] CPU Usage % for process: 0.10%
jobs-runner_1 | [pid: 57] Memory Stats for container: { freeSystemMemory: '257.253376 MB',
postgres_1 | LOG: redo done at 0/858D62B8
postgres_1 | LOG: last completed transaction was at log time 2021-10-22 22:15:16.356046+00
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
postgres_1 | LOG: incomplete startup packet
postgres_1 | LOG: incomplete startup packet
postgres_1 | LOG: incomplete startup packet
jobs-runner_1 | totalSystemMemory: '4124.83584 MB',
jobs-runner_1 | percentageFreeMemory: '6.24%' }
postgres_1 | LOG: incomplete startup packet
jobs-runner_1 | [pid: 57] Memory Stats for process: { rss: '111.37024 MB',
jobs-runner_1 | heapTotal: '64.069632 MB',
jobs-runner_1 | heapUsed: '55.058688 MB',
jobs-runner_1 | percentageHeapUsed: '85.94%' }
jobs-runner_1 | [pid: 57] CPU Usage % for process: 0.10%
jobs-runner_1 | [pid: 57] Memory Stats for container: { freeSystemMemory: '257.155072 MB',
jobs-runner_1 | totalSystemMemory: '4124.83584 MB',
jobs-runner_1 | percentageFreeMemory: '6.23%' }
jobs-runner_1 | [pid: 57] Memory Stats for process: { rss: '111.640576 MB',
jobs-runner_1 | heapTotal: '64.069632 MB',
jobs-runner_1 | heapUsed: '55.403488 MB',
jobs-runner_1 | percentageHeapUsed: '86.47%' }
jobs-runner_1 | [pid: 57] CPU Usage % for process: 0.10%
jobs-runner_1 | [pid: 57] Memory Stats for container: { freeSystemMemory: '257.155072 MB',
jobs-runner_1 | totalSystemMemory: '4124.83584 MB',
jobs-runner_1 | percentageFreeMemory: '6.23%' }

Looks like there are some database migration issues, would appreciate any help I could get with resolving this.

Hey Eric,

I just wanted to clarify that the DB maintenance this past weekend would only affect cloud hosted Retool, not self hosted.

That in mind, I am not quite sure how your instance ended up in this state, as the migration which added this constraint is ~2 years old.

Are you able to share the output of a \dt on the recently_visited_apps table?

Hi Mark,

I set up this on-prem deployment during September of last year.
My team hasn't used retools at all outside of the web app. How can we connect to the database from the ec2 server to get this table?

Eric

You should be able to use

sudo docker-compose exec postgres bash

to get into the postgres container, then

psql "dbname=$POSTGRES_DB host=$POSTGRES_HOST user=$POSTGRES_USER password=$POSTGRES_PASSWORD port=$POSTGRES_PORT"

to access to the db, and finally

\dt recently_visited_apps

to list the recently_visited_apps table.

Let me know if yon have any trouble!

Thanks for the instructions.
Here's the output of \dt recently_visited_apps
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+----------------------
public | recently_visited_apps | table | retool_internal_user
(1 row)

hmmm we might actually need \d+ in order to see the constraints.

Here's the output of \d+ recently_visited_apps

Table "public.recently_visited_apps"
Column | Type | Modifiers | Storage | Stats target | Description
-----------+--------------------------+-----------+----------+--------------+-------------
userId | integer | not null | plain | |
pageId | integer | not null | plain | |
visitType | character varying(255) | not null | extended | |
createdAt | timestamp with time zone | not null | plain | |
updatedAt | timestamp with time zone | not null | plain | |
Indexes:
"page_user_key" UNIQUE CONSTRAINT, btree ("pageId", "userId")
Foreign-key constraints:
"recently_visited_apps_pageId_fkey" FOREIGN KEY ("pageId") REFERENCES pages(id) ON DELETE CASCADE
"recently_visited_apps_userId_fkey" FOREIGN KEY ("userId") REFERENCES users(id) ON DELETE CASCADE

Hmm, it looks like the migration already ran, which is very weird. Can you confirm which version of Retool you are encountering this on, as well as contents of the SequelizeMeta table? ie:

SELECT
  *
FROM
  "SequelizeMeta"

I never set a specific version for retool so I'm not sure which version it's currently on.
Here's the contents of SequelizeMeta:

20170517054913-create-organization-and-user.js
20170519200903-create-page.js
20170519213338-create-page-save.js
20170521042445-create-resource.js
20170605055605-remove-non-null-constraint-on-organizations-domain.js
20170626044218-relax-constraints-for-non-db-resources.js
20170627021252-change-user-google-token-to-text-type.js
20170718030743-create-embed.js
20170725114241-create-retool-tables.js
20170801045015-add-database-id-table-name-uniq-constraint.js
20170801051119-add-ssl-column-to-resources.js
20170827134431-add-edit-privilege-to-resources.js
20170901033536-create-user-invites.js
20170904120915-add-option-column-to-resource.js
20170905163559-add-claimed-data-to-user-invites.js
20170908044721-camel-case-organization-id.js
20170908045940-camel-case-foreign-keys.js
20170908072236-create-groups.js
20170910220055-create-group-pages.js
20170911174211-add-unique-constraints-to-group-relations.js
20170913082842-change-json-columns-to-jsonb.js
20171007195513-create-user-invite-groups.js
20171128020917-add-password-to-embeds.js
20171211042535-add-verified-to-organizations.js
20171211043343-add-unique-constraint-to-organizations-name.js
20180105093546-create-audit-trail.js
20180121060558-add-last-logged-in-to-users.js
20180121081341-add-subdomain-to-organizations.js
20180125074833-add-metadata-to-audit-trail-event.js
20180314185313-create-folders.js
20180315005524-create-root-folders-for-orgs.js
20180315070506-add-uuid-to-pages.js
20180315202022-migrate-retool-buttons.js
20180325033649-fix-root-archive-structure.js
20180411230500-create-block.js
20180501214644-add-all-users-groups.js
20180509014253-create-tags-for-page-saves.js
20180512065556-add-enabled-to-users.js
20180525085038-add-environment-to-resources.js
20180601052843-create-user-session-state.js
20180621213631-convert-user-session-state-value-to-text.js
20180626225357-add-dynamically-queryable-to-resources.js
20180717013847-add-trial-to-cloud-users.js
20180719084641-create-sessions.js
20180720192034-add-reset-password.js
20180722013936-add-preload-javascript-and-javascript-links-to-organization.js
20180722022334-add-git-url-to-organization.js
20180722034753-add-changes-record-to-page-saves.js
20180724231554-add-user-to-page-save.js
20180730035637-create-plan.js
20180730041107-create-feature.js
20180806190311-add-stripe-customer-to-orgs.js
20180807005222-store-card-identifiers.js
20180807231432-add-stripe-reference-to-plans.js
20180808035938-add-subscriptions-to-organizations.js
20180820212043-add-billing-metadata-to-organizations.js
20180823030603-add-min-seats-to-plans.js
20180910200222-add-preloaded-css.js
20180910201302-convert-preloaded-to-text.js
20181001021553-add-public-id-to-users.js
20181004031428-add-phone-and-company.js
20181025035442-add-user-name-to-users.js
20190116012339-create-ssh-keys.js
20190121071059-add-git-branch-to-organizations.js
20190128005542-add-indices-to-audit-trail-events.js
20190131050840-add-2fa-token-to-users.js
20190203223343-add-more-indices-to-audit-trail-events.js
20190206130245-add-2fa-required-to-organizations.js
20190501042420-create-playground-query-and-playground-query-save.js
20190505070623-add-last-active-to-users.js
20190517153817-create-page-favorites.js
20190519110549-add-column-pages-photo-url.js
20190523163036-add-column-homepage-preloaded-css.js
20190526195058-create-group-resources.js
20190528141948-create-page-user-heartbeats.js
20190529143048-add-column-pages-deleted.js
20190629001631-add-column-onboarding-checklist.js
20190710194607-set-playground-query-save-on-delete-cascade.js
20190804182451-add-column-tag-description.js
20190809180847-create-experiments.js
20190812022830-create-experiments-deleted-at-column.js
20190820051248-create-idpMetadataXML-column.js
20190825193917-idpMetadataXML-to-text.js
20190830021143-add-state-to-sessions.js
20190913004256-create-retool-managed-note.js
20190926185409-add-uuid-to-playground-query.js
20190926225559-create-page-save-query-save-table.js
20191004073210-add-jit-saml-user-provisioning-to-organizations.js
20191007004015-drop-organizations-verified-column.js
20191007225034-create-approval-table.js
20191022003213-create-retool-managed-note-comment.js
20191025190326-google-photo-as-text-field.js
20191029012607-create-api-keys.js
20191115211147-add-library-access-controls-to-groups.js
20191116032755-add-page-id-query-id-to-page-save-playground-query-save.js
20191119183055-backfill-page-save-playground-query-save.js
20191204200513-approval-task-on-delete-cascade.js
20191204224653-create-app-metadata-tables.js
20200117001411-add-indices-to-metadata-tables.js
20200117221101-add-page-id-sorted-index-to-page-saves.js
20200314001058-add-onboardingStage-to-user.js
20200317004832-remove-onboardingStage-from-user.js
20200317020210-create-recently_visited_apps-table.js
20200318230009-wipe-out-onboardingChecklist.js
20200319001351-add-salesCTADismissed-to-user.js
20200410092658-add-lastEditedBy.js
20200416084658-create-user-invite-suggestions-table.js
20200519224444-add-licenseVerification-to-organizations.js
20200705213318-add-column-pages-is-global-widget.js
20200825185322-add-page-id-index-to-tags.js
20200319001405-add-tutorialCTADismissed-to-user.js
20200410161606-add-theme-id-to-organization-and-create-themes.js
20200419010412-add-favorited-to-folders.js
20200522163652-add-column-required-and-unique-to-flow-input-schemas.js
20200803175920-add-user-list-access-to-groups.js
20200319080412-add-onboardingStage-to-user.js
20200412171634-create-workspaces.js
20200513233128-create-v1-flows-tables.js
20200623174842-create-flow-task-histories.js
20200804000000-convert-validation-url-to-configurable-rest-call.js
(120 rows)

You can check the version in the Dockerfile -- the first line should be something like:

FROM tryretool/backend:2.77.19

It says FROM tryretool/backend:latest

Ah... that can cause some issues actually.

Can you try changing it to FROM tryretool/backend:2.77.19 (the current latest version)?

After making this change you will need to restart the containers.

I have set it to that version and restarted the containers. Still seems to be getting the same issue

db-connector_1 | Starting database connector server on port 3002
db-connector_1 | Running node v12.18.1
db-connector_1 | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
jobs-runner_1 | totalSystemMemory: '4124.83584 MB',
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
jobs-runner_1 | percentageFreeMemory: '4.13%' }
jobs-runner_1 | [pid: 57] Memory Stats for process: { rss: '109.09696 MB',
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
user-postgres_1 | LOG: MultiXact member wraparound protections are now enabled
db-connector_1 | Wed, 27 Oct 2021 22:10:58 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
jobs-runner_1 | heapTotal: '63.545344 MB',
postgres_1 | LOG: autovacuum launcher started
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
user-postgres_1 | LOG: autovacuum launcher started
db-connector_1 | Wed, 27 Oct 2021 22:10:58 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
jobs-runner_1 | heapUsed: '54.11628 MB',
postgres_1 | LOG: incomplete startup packet
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
db-ssh-connector_1 | Setting http and https agent maxSockets to 25
user-postgres_1 | LOG: database system is ready to accept connections
jobs-runner_1 | percentageHeapUsed: '85.16%' }
postgres_1 | LOG: incomplete startup packet
db-ssh-connector_1 | Starting database connector server on port 3002
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
db-ssh-connector_1 | Running node v12.18.1
db-connector_1 | dbconnector boot-up: 9026.842ms
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
jobs-runner_1 | [pid: 57] CPU Usage % for process: 0.66%
db-ssh-connector_1 | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
db-ssh-connector_1 | Wed, 27 Oct 2021 22:10:58 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-ssh-connector_1 | Wed, 27 Oct 2021 22:10:58 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
jobs-runner_1 | [pid: 57] Memory Stats for container: { freeSystemMemory: '450.49856 MB',
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 | There are pending database migrations, checking for up-to-date schema in 1 second(s)...
api_1 |
api_1 | ---------------------------
api_1 | - EXITING RETOOL -
api_1 | ---------------------------
api_1 |
api_1 | There are 93 pending database migrations. Retool will not start
api_1 | unless all database migrations have been run.
api_1 |
api_1 | To run migrate the database, run the following command:
api_1 |
api_1 | yarn sequelize db:migrate
db-ssh-connector_1 | dbconnector boot-up: 8510.748ms

Hmm, we can try manually running the migrations by exec-ing into the jobs-runner container:

sudo docker-compose exec jobs-runner bash

then running:

yarn sequelize db:migrate

Didn't seem to do anything:

yarn run v1.22.4

warning package.json: Potential typo "script", did you mean "scripts"?

warning package.json: License should be a valid SPDX license expression

$ ./node_modules/.bin/sequelize --config "$(test -d 'transpiled' && echo './transpiled' || echo '.')/server/config/config.js" db:migrate

Sequelize CLI [Node: 10.22.0, CLI: 5.5.0, ORM: 4.43.0]

NODE_ENV: 'production'

COOKIE_INSECURE: true

Loaded configuration file "server/config/config.js".

Using environment "production".

sequelize deprecated String based operators are now deprecated. Please use Symbol based operators for better security, read more at http://docs.sequelizejs.com/manual/tutorial/querying.html#operators node_modules/sequelize/lib/sequelize.js:242:13

No migrations were executed, database schema was already up to date.

Done in 0.75s.

Can you try running from the api container instead?

Here's the logs from running it on the api container:
yarn run v1.22.5
$ ./node_modules/.bin/sequelize --config "$(test -d 'transpiled' && echo './transpiled' || echo '.')/server/config/config.js" db:migrate

Sequelize CLI [Node: 12.22.5, CLI: 5.5.0, ORM: 5.22.3]

[process service types] [ 'MAIN_BACKEND' ]
Loaded configuration file "server/config/config.js".
Using environment "production".
== 20200917185402-create_organization_email_domains: migrating =======
== 20200917185402-create_organization_email_domains: migrated (0.065s)

== 20200923000000-create_partially_registered_users: migrating =======
== 20200923000000-create_partially_registered_users: migrated (0.044s)

== 20201001025033-change-flow_task_inputs-value-to-text: migrating =======
== 20201001025033-change-flow_task_inputs-value-to-text: migrated (0.016s)

== 20201006142314-add-platform-level-auth-steps-to-organization: migrating =======
== 20201006142314-add-platform-level-auth-steps-to-organization: migrated (0.011s)

== 20201010024343-add-column-deletedAt-to-flowTasks: migrating =======
== 20201010024343-add-column-deletedAt-to-flowTasks: migrated (0.011s)

== 20201014213857-create-blob: migrating =======
== 20201014213857-create-blob: migrated (0.018s)

== 20201022002601-disable-update_users_last_active_trigger: migrating =======
== 20201022002601-disable-update_users_last_active_trigger: migrated (0.012s)

== 20201022172751-add_partial_index_to_user_invite_suggestions: migrating =======
== 20201022172751-add_partial_index_to_user_invite_suggestions: migrated (0.019s)

== 20201022175548-add_auto_join_column_to_organization_email_domains: migrating =======
== 20201022175548-add_auto_join_column_to_organization_email_domains: migrated (0.020s)

== 20201025221224-create-instrumentation-integrations: migrating =======
== 20201025221224-create-instrumentation-integrations: migrated (0.030s)

== 20201027183344-remove-citext: migrating =======
== 20201027183344-remove-citext: migrated (0.011s)

== 20201106000000-remove-non-null-from-partially-registered-user-password: migrating =======
== 20201106000000-remove-non-null-from-partially-registered-user-password: migrated (0.010s)

== 20201110182704-create-app-themes: migrating =======
== 20201110182704-create-app-themes: migrated (0.030s)

== 20201111203850-fix_organization_email_domain_column_names: migrating =======
== 20201111203850-fix_organization_email_domain_column_names: migrated (0.034s)

== 20201116013135-create-usage-tracked-properties: migrating =======
== 20201116013135-create-usage-tracked-properties: migrated (0.023s)

== 20201120000001-page_add_protected: migrating =======
== 20201120000001-page_add_protected: migrated (0.034s)

== 20201120000002-add_branches_table: migrating =======
== 20201120000002-add_branches_table: migrated (0.030s)

== 20201120000003-add-page-save-branch: migrating =======
== 20201120000003-add-page-save-branch: migrated (0.010s)

== 20201120000004-add_commits_table: migrating =======
== 20201120000004-add_commits_table: migrated (0.028s)

== 20201120000005-page_add_synced: migrating =======
== 20201120000005-page_add_synced: migrated (0.036s)

== 20201120000006-add_protected_repo: migrating =======
== 20201120000006-add_protected_repo: migrated (0.013s)

== 20201123185949-add_organization_id_index_to_users: migrating =======
== 20201123185949-add_organization_id_index_to_users: migrated (0.013s)

== 20201129213514-add_page_save_id_index_to_tags: migrating =======
== 20201129213514-add_page_save_id_index_to_tags: migrated (0.012s)

== 20201208190222-drop_profile_photo_pkey_constraint: migrating =======
== 20201208190222-drop_profile_photo_pkey_constraint: migrated (0.012s)

== 20201211162854-create-experiment-strategy: migrating =======
== 20201211162854-create-experiment-strategy: migrated (0.041s)

== 20201214030645-add-cacheQueriesPerUser-to-organization.js: migrating =======
== 20201214030645-add-cacheQueriesPerUser-to-organization.js: migrated (0.009s)

== 20201229190223-add_resource_preview_hint_tables: migrating =======
== 20201229190223-add_resource_preview_hint_tables: migrated (0.027s)

== 20210121031255-backfill-onprem-experiments: migrating =======
== 20210121031255-backfill-onprem-experiments: migrated (0.016s)

== 20210122004828-add-instrumentation-config: migrating =======
== 20210122004828-add-instrumentation-config: migrated (0.009s)

== 20210122203434-add-creator-and-releaser-user-id-to-tag: migrating =======
== 20210122203434-add-creator-and-releaser-user-id-to-tag: migrated (0.012s)

== 20210126010727-org_add_protected_base_url: migrating =======
== 20210126010727-org_add_protected_base_url: migrated (0.008s)

== 20210201010727-org_add_protected_ui_url: migrating =======
== 20210201010727-org_add_protected_ui_url: migrated (0.010s)

== 20210205194413-add-description-to-pages: migrating =======
== 20210205194413-add-description-to-pages: migrated (0.009s)

== 20210212190920-addPasswordExpiredToUsers: migrating =======
== 20210212190920-addPasswordExpiredToUsers: migrated (0.008s)

== 20210217020424-add-display-name-to-resources: migrating =======
== 20210217020424-add-display-name-to-resources: migrated (0.014s)

== 20210306210856-create-group-folder-default: migrating =======
== 20210306210856-create-group-folder-default: migrated (0.019s)

== 20210308191233-add-onprem-subscription-to-organization: migrating =======
== 20210308191233-add-onprem-subscription-to-organization: migrated (0.009s)

== 20210310152048-add-planid-to-org: migrating =======
== 20210310152048-add-planid-to-org: migrated (0.014s)

== 20210311193528-create-user-viewed-features: migrating =======
== 20210311193528-create-user-viewed-features: migrated (0.021s)

== 20210317160820-add-is-admin-column-to-user-groups: migrating =======
== 20210317160820-add-is-admin-column-to-user-groups: migrated (0.019s)

== 20210317160846-backfill-is-admin-column-on-user-groups: migrating =======
Updating 38 user_groups records, in batches of 1000
Updating 38 records, 0 remaining
== 20210317160846-backfill-is-admin-column-on-user-groups: migrated (0.033s)

== 20210317160911-make-is-admin-on-user-groups-non-nullable: migrating =======
== 20210317160911-make-is-admin-on-user-groups-non-nullable: migrated (0.009s)

== 20210318011526-create-page-readme: migrating =======
== 20210318011526-create-page-readme: migrated (0.016s)

== 20210322000000-add-checksum-to-page-save: migrating =======
== 20210322000000-add-checksum-to-page-save: migrated (0.009s)

== 20210324141659-extend-page-readme: migrating =======
== 20210324141659-extend-page-readme: migrated (0.024s)

== 20210324230508-add-last-edited-by-to-page-documentation: migrating =======
== 20210324230508-add-last-edited-by-to-page-documentation: migrated (0.010s)

== 20210326205117-add-archived-at-to-groups: migrating =======
== 20210326205117-add-archived-at-to-groups: migrated (0.010s)

== 20210407160500-add-column-pages-clonedFromTemplateName: migrating =======
== 20210407160500-add-column-pages-clonedFromTemplateName: migrated (0.008s)

== 20210420051544-add-pgcrypto-extension: migrating =======
== 20210420051544-add-pgcrypto-extension: migrated (0.031s)

== 20210420202159-add-hide-retool-pill-to-theme: migrating =======
== 20210420202159-add-hide-retool-pill-to-theme: migrated (0.019s)

== 20210429043321-add_module_uuid_key_to_theme: migrating =======
== 20210429043321-add_module_uuid_key_to_theme: migrated (0.012s)

== 20210504205959-add-pinned-to-latest-version-to-playground-query-usage: migrating =======
== 20210504205959-add-pinned-to-latest-version-to-playground-query-usage: migrated (0.013s)

== 20210506231331-backfill-pinned-to-latest-version-on-playground-query-usage: migrating =======
Updating 280 page_save_playground_query_saves records, in batches of 1000
Updating 280 records, 0 remaining
== 20210506231331-backfill-pinned-to-latest-version-on-playground-query-usage: migrated (0.020s)

== 20210506231755-make-pinned-to-latest-version-on-playground-query-usage-non-nullable: migrating =======
== 20210506231755-make-pinned-to-latest-version-on-playground-query-usage-non-nullable: migrated (0.010s)

== 20210512000000-add-canary-to-org: migrating =======
== 20210512000000-add-canary-to-org: migrated (0.019s)

== 20210526201600-add-protected-apps-sync-enabled-column-to-organizations: migrating =======
== 20210526201600-add-protected-apps-sync-enabled-column-to-organizations: migrated (0.011s)

== 20210526223041-backfill-protected-apps-sync-enabled-column-on-organizations: migrating =======
Updating 1 organizations records, in batches of 1000
Updating 1 organizations records, 0 remaining
== 20210526223041-backfill-protected-apps-sync-enabled-column-on-organizations: migrated (0.017s)

== 20210527012348-make-protected-apps-sync-enabled-column-on-organizations-non-nullable: migrating =======
== 20210527012348-make-protected-apps-sync-enabled-column-on-organizations-non-nullable: migrated (0.008s)

== 20210602200626-create-notebook-workflow: migrating =======
== 20210602200626-create-notebook-workflow: migrated (0.020s)

== 20210603161916-add-audit-log-access-to-groups: migrating =======
== 20210603161916-add-audit-log-access-to-groups: migrated (0.013s)

== 20210604211347-backfill-audit-log-access-on-groups: migrating =======
Updating 4 groups records, in batches of 1000
Updating 4 records, 0 remaining
== 20210604211347-backfill-audit-log-access-on-groups: migrated (0.013s)

== 20210604212156-make-audit-log-access-non-nullable: migrating =======
== 20210604212156-make-audit-log-access-non-nullable: migrated (0.009s)

== 20210608225513-update-resources-access-level-constraint: migrating =======
== 20210608225513-update-resources-access-level-constraint: migrated (0.013s)

== 20210609175143-update-group-resources-access-level-constraint: migrating =======
== 20210609175143-update-group-resources-access-level-constraint: migrated (0.010s)

== 20210614230814-org-startup-program: migrating =======
== 20210614230814-org-startup-program: migrated (0.012s)

== 20210615184230-add-query-id-index-to-query-saves: migrating =======
== 20210615184230-add-query-id-index-to-query-saves: migrated (0.012s)

== 20210620042700-create-notebook-workflow-run: migrating =======
== 20210620042700-create-notebook-workflow-run: migrated (0.021s)

== 20210621183636-add-intercom-toggle-to-org: migrating =======
== 20210621183636-add-intercom-toggle-to-org: migrated (0.010s)

== 20210621221154-startup-program-discount-column: migrating =======
== 20210621221154-startup-program-discount-column: migrated (0.008s)

== 20210621225353-backfill-enable-intercom-column-on-orgs: migrating =======
Updating 1 organizations records, in batches of 1000
Updating 1 records, 0 remaining
== 20210621225353-backfill-enable-intercom-column-on-orgs: migrated (0.017s)

== 20210623164832-make-enableintercom-non-nullable: migrating =======
== 20210623164832-make-enableintercom-non-nullable: migrated (0.009s)

== 20210630120915-add-show-header-logo-to-themes: migrating =======
== 20210630120915-add-show-header-logo-to-themes: migrated (0.012s)

== 20210630173841-create-workflow-save-table: migrating =======
== 20210630173841-create-workflow-save-table: migrated (0.021s)

== 20210701120000-backfill-show-header-logo-on-themes: migrating =======
Updating 0 themes records, in batches of 1000
== 20210701120000-backfill-show-header-logo-on-themes: migrated (0.008s)

== 20210706175948-remove-enable-intercom-column: migrating =======
== 20210706175948-remove-enable-intercom-column: migrated (0.008s)

== 20210707162930-make-show-header-logo-non-nullable: migrating =======
== 20210707162930-make-show-header-logo-non-nullable: migrated (0.011s)

== 20210708002200-add-show-login-logo-to-themes: migrating =======
== 20210708002200-add-show-login-logo-to-themes: migrated (0.013s)

== 20210708120000-backfill-show-login-logo-on-themes: migrating =======
Updating 0 themes records, in batches of 1000
== 20210708120000-backfill-show-login-logo-on-themes: migrated (0.009s)

== 20210708224112-make-show-login-logo-non-nullable: migrating =======
== 20210708224112-make-show-login-logo-non-nullable: migrated (0.010s)

== 20210715155015-create-datagrid-tables: migrating =======
== 20210715155015-create-datagrid-tables: migrated (0.062s)

== 20210809155015-add-table-id-to-datagrid-activity: migrating =======
== 20210809155015-add-table-id-to-datagrid-activity: migrated (0.012s)

== 20210818053430-create-storage-blob: migrating =======
== 20210818053430-create-storage-blob: migrated (0.023s)

== 20210824155015-add-allow-schema-edit-setting-to-grid: migrating =======
== 20210824155015-add-allow-schema-edit-setting-to-grid: migrated (0.020s)

== 20210825162745-add_pages_index: migrating =======
== 20210825162745-add_pages_index: migrated (0.011s)

== 20210826232121-add_folders_index: migrating =======
== 20210826232121-add_folders_index: migrated (0.011s)

== 20210830102542-add-column-pages-is-mobile-app: migrating =======
== 20210830102542-add-column-pages-is-mobile-app: migrated (0.010s)

== 20210902174420-add_page_save_playground_query_saves_playground_query_id_idx: migrating =======
== 20210902174420-add_page_save_playground_query_saves_playground_query_id_idx: migrated (0.011s)

== 20210903174420-add_table_for_grid_saved_filtering: migrating =======
== 20210903174420-add_table_for_grid_saved_filtering: migrated (0.026s)

== 20210908193736-add-retool-pill-appearance: migrating =======
== 20210908193736-add-retool-pill-appearance: migrated (0.015s)

== 20210908202851-backfill-retool-pill-appearance: migrating =======
Updating 0 theme records, in batches of 1000
== 20210908202851-backfill-retool-pill-appearance: migrated (0.012s)

== 20210920202851-grid-add-schema-name: migrating =======
== 20210920202851-grid-add-schema-name: migrated (0.008s)

== 20210921000000-fix-instrumentation-integration-index: migrating =======
== 20210921000000-fix-instrumentation-integration-index: migrated (0.014s)

== 20210927215315-experiment-strategy-enum-to-string: migrating =======
== 20210927215315-experiment-strategy-enum-to-string: migrated (0.020s)

Done in 3.85s.

Now the tail logs look like this after I restarted the containers:

db-ssh-connector_1  | [deprecated] do nothing.
db-ssh-connector_1  | [process service types] [ 'DB_SSH_CONNECTOR' ]
user-postgres_1     | LOG:  database system was shut down at 2021-10-28 02:16:41 UTC
jobs-runner_1       |   percentageFreeMemory: '13.57%' }
jobs-runner_1       | [pid: 57] Memory Stats for process:  { rss: '110.7968 MB',
db-connector_1      | Setting http and https agent maxSockets to 25
db-connector_1      | Starting database connector server on port 3002
db-connector_1      | Running node v12.18.1
user-postgres_1     | LOG:  MultiXact member wraparound protections are now enabled
jobs-runner_1       |   heapTotal: '64.59392 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.472624 MB","percentageHeapUsed":"80.8%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.00512 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:55:11.956Z"}
db-ssh-connector_1  | Setting http and https agent maxSockets to 25
db-ssh-connector_1  | Starting database connector server on port 3002
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.08624 MB","percentageHeapUsed":"81.57%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.345664 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.56%"}},"level":"info","timestamp":"2021-10-28T02:55:25.591Z"}
user-postgres_1     | LOG:  database system is ready to accept connections
db-ssh-connector_1  | Running node v12.18.1
jobs-runner_1       |   heapUsed: '55.117704 MB',
jobs-runner_1       |   percentageHeapUsed: '85.33%' }
db-connector_1      | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.468456 MB","percentageHeapUsed":"81.42%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.919104 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:55:31.698Z"}
postgres_1          | LOG:  database system was shut down at 2021-10-28 02:16:55 UTC
user-postgres_1     | LOG:  autovacuum launcher started
db-ssh-connector_1  | ARGV: [ '--max_old_space_size=5120', '--max-http-header-size=80000' ]
db-ssh-connector_1  | Thu, 28 Oct 2021 02:17:24 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-connector_1      | Thu, 28 Oct 2021 02:17:24 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at ../snapshot/backend/transpiled/dbconnector/app.js:0:0
db-connector_1      | Thu, 28 Oct 2021 02:17:24 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
db-connector_1      | dbconnector boot-up: 9807.302ms
jobs-runner_1       | [pid: 57] CPU Usage % for process:  0.08%
jobs-runner_1       | [pid: 57] Memory Stats for container:  { freeSystemMemory: '560.14848 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.482128 MB","percentageHeapUsed":"80.81%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.894528 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:55:31.977Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.10984 MB","percentageHeapUsed":"81.59%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.894528 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:55:45.592Z"}
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.477296 MB","percentageHeapUsed":"81.43%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.894528 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:55:51.720Z"}
jobs-runner_1       |   totalSystemMemory: '4124.83584 MB',
db-ssh-connector_1  | Thu, 28 Oct 2021 02:17:24 GMT body-parser deprecated undefined extended: provide extended option at ../snapshot/backend/node_modules/body-parser/index.js:105:29
postgres_1          | LOG:  MultiXact member wraparound protections are now enabled
jobs-runner_1       |   percentageFreeMemory: '13.58%' }
jobs-runner_1       | [pid: 57] Memory Stats for process:  { rss: '111.067136 MB',
db-ssh-connector_1  | dbconnector boot-up: 9772.110ms
postgres_1          | LOG:  database system is ready to accept connections
jobs-runner_1       |   heapTotal: '64.59392 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.49172 MB","percentageHeapUsed":"80.82%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.894528 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:55:51.998Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.133608 MB","percentageHeapUsed":"81.61%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.919104 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:56:05.593Z"}
postgres_1          | LOG:  autovacuum launcher started
jobs-runner_1       |   heapUsed: '55.4584 MB',
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.485992 MB","percentageHeapUsed":"81.43%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.919104 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:56:11.741Z"}
jobs-runner_1       |   percentageHeapUsed: '85.86%' }
jobs-runner_1       | [pid: 57] CPU Usage % for process:  0.08%
postgres_1          | LOG:  incomplete startup packet
jobs-runner_1       | [pid: 57] Memory Stats for container:  { freeSystemMemory: '559.742976 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.501104 MB","percentageHeapUsed":"80.83%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.919104 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:56:12.007Z"}
postgres_1          | LOG:  incomplete startup packet
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.157232 MB","percentageHeapUsed":"81.63%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.632384 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:56:25.594Z"}
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.494776 MB","percentageHeapUsed":"81.44%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.14848 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:56:31.762Z"}
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.510696 MB","percentageHeapUsed":"80.83%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.14848 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:56:32.015Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.181032 MB","percentageHeapUsed":"81.64%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"560.14848 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:56:45.595Z"}
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.50344 MB","percentageHeapUsed":"81.45%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.14848 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:56:51.783Z"}
jobs-runner_1       |   totalSystemMemory: '4124.83584 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.520112 MB","percentageHeapUsed":"80.84%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.14848 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:56:52.036Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.20468 MB","percentageHeapUsed":"81.66%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.742976 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:57:05.597Z"}
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.512184 MB","percentageHeapUsed":"81.46%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.742976 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:57:11.804Z"}
jobs-runner_1       |   percentageFreeMemory: '13.57%' }
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.52964 MB","percentageHeapUsed":"80.85%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"559.742976 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.57%"}},"level":"info","timestamp":"2021-10-28T02:57:12.057Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"199.307264 MB","heapTotal":"127.602688 MB","heapUsed":"104.228448 MB","percentageHeapUsed":"81.68%"},"CPU Usage % for process":"0.08%","Memory Stats for container":{"freeSystemMemory":"559.5136 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.56%"}},"level":"info","timestamp":"2021-10-28T02:57:25.598Z"}
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.52088 MB","percentageHeapUsed":"81.46%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.095232 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:57:31.825Z"}
jobs-runner_1       | [pid: 57] Memory Stats for process:  { rss: '111.599616 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.53908 MB","percentageHeapUsed":"80.86%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.095232 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.58%"}},"level":"info","timestamp":"2021-10-28T02:57:32.078Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"198.28736 MB","heapTotal":"127.602688 MB","heapUsed":"103.243592 MB","percentageHeapUsed":"80.91%"},"CPU Usage % for process":"0.07%","Memory Stats for container":{"freeSystemMemory":"560.726016 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.59%"}},"level":"info","timestamp":"2021-10-28T02:57:45.599Z"}
jobs-runner_1       |   heapTotal: '64.59392 MB',
api_1               | {"namespace":"memoryUsage","pid":47,"message":{"pid":47,"Memory Stats for process":{"rss":"199.643136 MB","heapTotal":"127.0784 MB","heapUsed":"103.529664 MB","percentageHeapUsed":"81.47%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.676864 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.59%"}},"level":"info","timestamp":"2021-10-28T02:57:51.846Z"}
jobs-runner_1       |   heapUsed: '55.793688 MB',
jobs-runner_1       |   percentageHeapUsed: '86.38%' }
jobs-runner_1       | [pid: 57] CPU Usage % for process:  0.08%
jobs-runner_1       | [pid: 57] Memory Stats for container:  { freeSystemMemory: '559.5136 MB',
api_1               | {"namespace":"memoryUsage","pid":45,"message":{"pid":45,"Memory Stats for process":{"rss":"198.549504 MB","heapTotal":"126.816256 MB","heapUsed":"102.54864 MB","percentageHeapUsed":"80.86%"},"CPU Usage % for process":"0.05%","Memory Stats for container":{"freeSystemMemory":"560.676864 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.59%"}},"level":"info","timestamp":"2021-10-28T02:57:52.099Z"}
api_1               | {"namespace":"memoryUsage","pid":21,"message":{"pid":21,"Memory Stats for process":{"rss":"198.28736 MB","heapTotal":"127.602688 MB","heapUsed":"103.267368 MB","percentageHeapUsed":"80.93%"},"CPU Usage % for process":"0.07%","Memory Stats for container":{"freeSystemMemory":"560.828416 MB","totalSystemMemory":"4124.83584 MB","percentageFreeMemory":"13.6%"}},"level":"info","timestamp":"2021-10-28T02:58:05.601Z"}

Ah, so it looks like manually running the migrations worked then, as those post-restart logs look fairly normal.

Are you able to log into the instance again now?

The docker containers are all running now but I still can't access the web app or ping the ec2 server.

Huh.. that is very strange. When you try to access the web app, does it give a particular error (4xx/5xx/etc), or does it just fail to load at all?