Helm k8s postgres deployment blocked - retool version 2.121.3-enterprise

[ifelsefi]

Hi

I am getting this error when trying to deploy retool using helm:

There are 401 pending database migrations, checking for up-to-date schema in 1 second(s)...

db migrate does not work as described:

Self-hosted on AWS ECS, problem with db:migrate

Hi, I deployed Retool on AWS ECS Fargate, and have problem with database migration. There is a log after first run: There are pending database migrations, checking for up-to-date schema in 1 second(s)... --------------------------- - EXITING RETOOL - --------------------------- There are 233 pending database migrations. Retool will not start unless all database migrations have been run. To run migrate the database, run the following command: yarn sequelize db:migrate So I ran tha…

yarn does not work either:

Retools failing to start after the database maintenance yesterday

I am using an on-premise deployment of retool on an unbuntu aws ec2 server. It was working fine, but am now unable to connect after the database maintenance yesterday. Here is the results of running sudo docker-compose ps retoolonpremise_api_1 docker-entrypoint.sh bash ... Exit 1 retoolonpremise_db-connector_1 docker-entrypoint.sh bash ... Up 3000/tcp, 3001/tcp, 3002/tcp retoolonpremise_db-ssh-connector_1 docker-entrypoint.sh bash ... Up 3000/tcp, …

I have a pretty vanilla configuration in my values.yaml.

We are using an external postgres 13 server:

  # IMPORTANT: Incompatible with postgresql subchart
  # Please disable the subchart in order to use a managed or external postgres instance.
  postgresql:
    # Specify if postgresql subchart is disabled
    host: "pg.example.com"
    port: "5432"
    db: "retool"
    user: "postgres"
    password: "password"
    ssl_enabled: false

[Kabirdas]

Hey @ifelsefi!

Database migrations are typically handled by the Retool jobs runner - would you mind sharing logs from the pod where you're hosting the jobs runner container?

It may also be worth pulling a later version like 2.121.13 since there have been some bug fixes for 2.121.

[ifelsefi]

Hi Kabirdas,

Thanks for checking into this issue. We are now trying 3.0.4-enterprise.

I see the below error:

kubectl logs pod/retool-jobs-runner-78d6845c5f-6c87s -n retool
not untarring the bundle
{"level":"info","message":"[process service types] JOBS_RUNNER","timestamp":"2023-07-12T16:05:11.138Z"}
Acquiring lock to run migrations...
Acquired lock
Running database migrations...
== 20180315070506-add-uuid-to-pages: migrating =======

---------------------------
-     EXITING RETOOL      -
---------------------------

Error running database migrations: SequelizeDatabaseError: could not open extension control file "/usr/pgsql-13/share/extension/uuid-ossp.control": No such file or directory

Pod state:

NAME                                      READY   STATUS             RESTARTS       AGE
pod/retool-796d4ccf5b-bqt8s               0/1     Running            6 (116s ago)   12m
pod/retool-796d4ccf5b-dr6bg               0/1     Running            6 (117s ago)   12m
pod/retool-jobs-runner-78d6845c5f-6c87s   0/1     CrashLoopBackOff   6 (114s ago)   12m

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/retool   ClusterIP   172.30.215.99   <none>        3000/TCP   12m

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/retool               0/2     2            0           12m
deployment.apps/retool-jobs-runner   0/1     1            0           12m

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/retool-796d4ccf5b               2         2         0       12m
replicaset.apps/retool-jobs-runner-78d6845c5f   1         1         0       12m

Persistent Volumes:

NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
retool   Bound    pvc-cc03679f-07aa-41ef-a16a-793881dcc434   15Gi       RWX            vastdata-filesystem   12m

We are using a ReadWriteMany volume as the application would not deploy with ReadWriteOnce.

I am assuming that /usr exists within the job runner container but the pod keeps crash looping so I cannot get a shell.

The persistent volume seems to be defined properly:

    Mounts:
      /retool_backend/pv-data from retool-pv (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8r2qm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  retool-pv:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  retool
    ReadOnly:   false

Do you have any suggestions?

[ifelsefi]

Same error with backend:2.123.14-enterprise:

{"level":"info","message":"[process service types] JOBS_RUNNER","timestamp":"2023-07-12T16:33:06.736Z"}
Acquiring lock to run migrations...
Acquired lock
Running database migrations...
== 20180315070506-add-uuid-to-pages: migrating =======

---------------------------
-     EXITING RETOOL      -
---------------------------

Error running database migrations: SequelizeDatabaseError: could not open extension control file "/usr/pgsql-13/share/extension/uuid-ossp.control": No such file or directory

The container args:

  Args:
      bash
      -c
      chmod -R +x ./docker_scripts; sync; ./docker_scripts/wait-for-it.sh -t 0 "external-pg-server.domain.example.com":"5432"; ./docker_scripts/start_api.sh

[Kabirdas]

Thanks for posting your logs! it looks as though the external Postgres server may be missing the uuid-ossp module. How are you hosting it?

[ifelsefi]

That was the problem. Thank you!