Dynamic config permission denied; can't enable workflows

[Nyoma_Diamond]

Hello,

I am attempting to run a local self-hosted retool instance, but have been running into problems enabling workflows. Specifically, when attempting to run the Temporal container, the following error occurs after connecting to the postgres database, and then the container immediately exits:

2023/08/03 22:25:26 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2023-08-03T22:25:26.999Z","msg":"Build info.","git-time":"2022-11-15T23:13:23.000Z","git-revision":"00bb51389c001bf3c8cb20ef77e5c598d4c55ba9","git-modified":false,"go-arch":"amd64","go-os":"linux","go-version":"go1.18.8","cgo-enabled":false,"server-version":"1.18.5","logging-call-at":"main.go:142"}
Unable to create dynamic config client. Error: unable to validate dynamic config: dynamic config: config/dynamicconfig/development-sql.yaml: stat config/dynamicconfig/development-sql.yaml: permission denied

To my knowledge, my docker.env, docker-compose.yml, and other related files are all set up correctly, however for some reason the container gets a permission denied error when attempting attempting to access the development-sql.yaml file. All containers have been started using sudo. I've tried giving full access to the file (chmod 777), but still received the error. Some stackoverflow pages have suggested it to be a SELinux problem, however setting setenforce 0 does not help.

I have followed the documentation at https://docs.retool.com/self-hosted/guides/workflows-deployment and https://docs.retool.com/self-hosted/quickstarts/local directly. If I had to take a guess, I may have made a mistake relating to this instruction:

Make sure to copy over the dynamicconfig directory in retool-onpremise onto your deployment instance.

However, if what I did was wrong, I am unclear on what "your deployment instance" refers to if not the system that is already running the docker containers and where I would copy dynamicconfig over to, since the statement is pretty vague.

Potentially relevant note: the Linux instance is RHEL 8, and uses podman instead of docker due to technical limitations.

Help would be greatly appreciated!

[Nyoma_Diamond]

Upon further inspection, it appears that the entire config/dynamicconfig/ directory in the temporal docker container is inaccessible. The contents and permissions of the directory are as follows:

$ ls -alh config/
total 28K
drwxr-xr-x    1 temporal temporal      25 Aug 14 17:16 .
drwxr-xr-x    1 root     root          20 May 15 19:09 ..
-rw-rw-r--    1 root     root       18.2K Nov 15  2022 config_template.yaml
-rw-r--r--    1 temporal temporal    4.8K Aug 14 17:16 docker.yaml
drwxr-x---    2 41547    41547         34 Aug 14 16:05 dynamicconfig

However, attempting to look into dynamicconfig produces a permission denied error (even when entering the container as root):

$ ls -alh config/dynamicconfig/
ls: can't open 'config/dynamicconfig/': Permission denied
total 0

I have tried changing user and group ownership of the directory outside of the container to root. I also tried chmod 777 on the dynamicconfig directory, resulting in the following permissions as expected:

drwxrwxrwx    2 41547    41547         34 Aug 14 16:05 dynamicconfig

however I still get a permission denied error when trying to access it.

I'm at a loss as to how to get the container to be able to access the volume mounted there.

[nroeder]

Did you get this figured out, trying to deploy and the instructions are still lacking..

[Nyoma_Diamond]

Nope, still haven't been able to figure it out. I also got in contact with Retool's support via their chat and they were unsure of the issue as well; It seems 100% local instances w/ Workflows via Docker Compose deployments are not yet fully supported. If using Docker Compose, they seem to promote using a Temporal cloud instance to handle Workflows, however unfortunately that can't be done in my use-case

I'm guessing that this is an issue with my system configuration (i.e., RHEL on a pre-configured VM which may have security settings in place that conflict with the requirements for Workflows), in which case until Retool improves their documentation and/or finds a workaround, there's nothing I can do to get Workflows working.

The bug itself seems very strange, though. Based on my exploration, it is a permissioning error, but giving full access permissions doesn't resolve it. So I am unsure what the actual source of the issue is.

[Artemm]

Hello! Is there any update?
Facing the same problem with temporal container.

[zelterNN]

I also encountered the same issue. My workaround:

Create a Temporal.Dockerfile in the main folder (default is their retool-onpremise-master). Content:

FROM temporalio/auto-setup:1.23.1

USER root

RUN touch /etc/temporal/config/dynamicconfig/development-sql.yaml
RUN chown -R root:temporal /etc/temporal

On docker-compose.yml replace the original temporal section with below:

  temporal:
    container_name: temporal
    build:
      context: ./
      dockerfile: Temporal.Dockerfile
    env_file: ./docker.env
    environment:
      - DB=postgresql
      - POSTGRES_SEEDS=postgres
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml
      # To enable TLS between temporal and external postgres, set both below variables to true
      - SQL_TLS_ENABLED=false
      - SQL_TLS=false
      # Defined twice because temporal-server and temporal-sql-tool use different envvars
      - SQL_TLS_DISABLE_HOST_VERIFICATION=true
      - SQL_HOST_VERIFICATION=false
    # image: tryretool/one-offs:retool-temporal-1.1.2
    networks:
      - intra-temporal-network
      - temporal-network
    ports:
      - "127.0.0.1:7233:7233"
    # volumes:
    #   - ./dynamicconfig:/etc/temporal/config/dynamicconfig

what it does is

• to build the temporal container from the freshly created Temporal.Dockerfile instead of the virgin image from dockerhub
• to disable the dynamicconfig volumns (which causes the issue)

However, I also encountered some other issues after this, if you guys have seen similar one while enabling workflow on self-host retool with self-host temporal and know the solution, please let me know:

Retool Workflow Worker constantly restart with self-host temporal

Hi Retool Community, I'm currently trying to setup a self-host temporal to work with the workflow. I used the docker-compose-with-temporal.yml as the startpoint. After starting all containers, i can: access self-host Retool without issue (except the Workflow) access self-host Temporal UI without issue However, the workflow is still not working and ask me to 'upgrade deployment'. What I can see on the docker is the Workflow Worker is constantly restarting. So I guess the issue is restricted …

[zelterNN]

Update: use below code in Temporal.Dockerfile

# FROM temporalio/auto-setup:1.23.1
FROM tryretool/one-offs:retool-temporal-1.1.2

USER root

RUN touch /etc/temporal/config/dynamicconfig/development-sql.yaml
RUN chown -R root:temporal /etc/temporal

Or else the worker won't start properly

[yordis]

You shouldn't need to create a custom Image to change the USER, instead pass the user property in your docker-compose file or using the --user flag

services:
  temporal:
    container_name: temporal
    image: temporalio/auto-setup:latest
    user: root # <--- your OS user

Make sure that the user value matches your user name when starting the container

[zelterNN]

I use the root in the custom image is basically to allow the creation of the blank development-sql.yaml in the /etc folder
I'm not exactly sure what's the underlying issue described in my OP but it seems the Retool default image failed to create the file in my environment