Password reset for self-hosted instance does not send an email

💬 Self Hosted Retool
1

Hi,

We have deployed a self-hosted instance of retool. Upon creating our organization, our password manager has not automatically saved the credentials. Therefore, we tried to use the password reset functionality of the application. In the network console, the request is a POST https://{ourDomain}/api/user/resetPassword and we receive a 200 response with the following message :

{
    "error": false,
    "message": "Check your inbox at email@ourdomain.com for further instructions"
}

The password reset email is never sent (or never reaches our inbox). Retool support could not help us and suggested we post here. Furthermore ...

  • There are a few threads about this topic, but this case is not about Retool On-prem Portal portal - we can connect just fine on that platform and we know they use different sets of credentials
  • We have validated with our DevOps team that the networking requirements were applied, including CIDR / Individual IPs / domains whitelisting (Self-hosted Retool requirements | Retool Docs). Our firewall allows traffic to leave the instance
  • We have already checked to make sure the password reset wasn't in a spam/filtered box
  • This is a docker deployment within Azure
  • Our billing platform indicates a free plan, although the deployment (advanced -> billing) indicated that we were on enterprise even after sync'ing the licence.

Thanks in advance!!

2

Hey @rcbigdata have you been able to access your instance at any point or is this your first time attempting to log in?

If this is the first time, have you able to go to your instance/auth/signup to create the organization (docs)?

I see the ticket you created with our team - is it your email that you initially used to sign up? I can check if these emails are getting blocked if this isn't your first time attempting to sign up.

Thanks

3

Hi @LaurenM
I was able to access the instance at first, yes - after creating the organization through auth/signup. I then realized the password wasn't saved, so I logged out to reset the password - hence the current situation. My email is not the one used to create the org. Can I DM you the email instead of publishing it here?

FYI we also have a production deployment - to which we DO have the master credentials - and we also can't reset the password.

4

Hi @ rcbigdata, welcome to the forum! :wave:

The fact that you are seeing this response:

Screenshot 2024-05-02 at 3.48.47 PM
Screenshot 2024-05-02 at 3.48.47 PM1174×378 19.3 KB

means it's most likely a network issue, unless there was a typo when typing the email but I'm sure you discarded this possibility already.

If you share the logs from the firewall right after sending the reset password email, we could verify if it has been blocked or not. Could you also share your firewall settings?

On the other hand, do you happen to see this error when you try to sign up a new user?

image
image948×474 28.9 KB

5

Hi Paulo! Thanks.

I've tried to create a new organization via the auth/signup endpoint and I actually get an EXPIRED AUTHORIZATION TOKEN error.

Trying to log in with an invalid password results in an Invalid credentials for this organization. Please try again.

I'm afraid I can't share more details at the moment on the Firewall configuration. All I know is that the requirements mentioned here were all set - we also have a networking rule that allows traffic to leave our Azure instance. I'll try to get more details and get back to you.

6

It would be great if we could get more details. On the other hand, if you were the only user in the org, and there is no data we could risk losing, we could try deleting all docker volumes and containers. Then running the sudo docker-compose up -d to redeploy from scratch. We should be able to register on the newly deployed instance.

7

Hi @Paulo , I'm working with @rcbigdata on this issue and I'm the DevOps that was in charge of setting up the infrastructure and deployment.

Here is a quick rundown of how our infrastructure is currently setup for this deployment, hope this clears up any questioning you might have about it.

  • The deployment was done in an Azure VM, and is being run using docker compose, with the exception of the PostgreSQL database which is hosted externally in Azure Database for PostgreSQL, thus giving us the flexibility to easily redeploy the VM without impacting any of the data/configuration stored in the database.
  • All traffic going from the Azure VM to the internet is routed directly to a firewall instance in Azure (Fortigate), which does not block any outgoing traffic on any port (an "any/any" rule is configured for outbound internet traffic).
  • There is no NSG applied at either the VM's NIC or subnet level that would be blocking the traffic; all traffic goes from the VM to the Fortigate instance via a route table, and from there the traffic goes through a public Azure Load Balancer with a static Public IP.

A few things of note that might differ from certain other setups:

  • The domain that is configured in our instance is not publicly accessible; it is only registered in our internal DNS.
  • The ReTool application is not accessed "directly", as in: We have an Nginx reverse proxy that handles all incoming traffic and also handles SSL. The proxy configuration is as simple as it gets:
server {
    listen 443 ssl;
    server_name <retool_url>;

    ssl_certificate             <path_to_ssl_certificate>;
    ssl_certificate_key         <path_to_ssl_certificate_key>;
    ssl_password_file           <path_to_ssl_certificate_key_password>;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_headers_hash_max_size     512;
        proxy_headers_hash_bucket_size  128;
    }
}
  • The reason we are doing so is because everything we manage in Azure comes "prepackaged/preconfigured" in an image that is then deployed to the VM, and managing the SSL configuration according to ReTool's documentation (Configure SSL and custom certificates | Retool Docs) is not feasible since we need to modify the docker-compose.yml file, which might be modified at any time in a new ReTool version.

Hope this helps clear up a few things about our setup!

8

Hi @Alex_Pilon, welcome to the forum, and thank you for expanding on the details of your deployment. :slightly_smiling_face:

After we make the request to send the password reset email, could you share the logs from the instance and Fortigate?

From the message that @rcbigdata sees on the Network tab, the request is making it to the server. The logs from this request should help us identify why the email is not making it through. While a "any/any" rule typically allows outbound traffic, there could still be specific firewall rules or security policies that are affecting email traffic.

9

Hi Paulo,

I've opened a ticket with our corporate support, since they manage the inbound rules for our corporate email accounts. They should be able to figure out if the emails are stuck in the corporate spam prior to reaching our inbox/spam inbox.

Do you know from which domain the emails should be coming from?

10

Great observation! I just tested it on my own Self-hosted instance deployed with Docker, the email comes from: invites@email.retool.com

1 Like
11

The reason for this ask to our corporate IT team is because prior to deploying our Development environment with the above configuration, we had deployed a proof of concept in a very "dumb" manner, meaning: A simple VM with its own public IP, no firewalls (besides a very basic NSG allowing port 443 inbound from our corporate IP address range), and @rcbigdata did a test yesterday to see if that POC instance could receive password reset emails and they were also not getting through, so firewalls are definitely not the issue.

12

Please let us know if that was the case Otherwise, we are happy to take a deeper look at what is going on.

1 Like
13

Hi everyone! Quick update :

The corporate IT team has confirmed that no emails from any domain related to retool.com were blocked. Everything sent by such domain has been transmitted to our inboxes. This includes promotional emails (devon@retool.com), support emails as well as invitations to collaborate on Retool (invites@email.retool.com).

With that in mind, as well as details provided by Alex, I believe it is safe to assume that the corporate email filters and firewall are not the root of the problem here.

14

Could you share the logs from the instance right after we click the reset password button?

15

This is what we are looking for in the api service logs:

self-hosted-codebase_api_1  | {"level":"info","message":{"http":{"method":"POST","url_base":"https://<VM-IP:PORT">,"url_path":"/api/user/resetPassword"},"type":"REQUEST_BEGIN"},"pid":115,"requestId":"3a2f842c-5311-44b6-8a4d-ac209ea10483","timestamp":"2024-05-15T22:08:48.609Z"}
self-hosted-codebase_api_1  | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Beginning workflows for retool-event","organizationId":1,"pid":115,"requestId":"3a2f842c-5311-44b6-8a4d-ac209ea10483","timestamp":"2024-05-15T22:08:48.635Z"}
self-hosted-codebase_api_1  | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Retrieved 0 workflows for retool-event","numWorkflows":0,"organizationId":1,"pid":115,"requestId":"3a2f842c-5311-44b6-8a4d-ac209ea10483","timestamp":"2024-05-15T22:08:48.637Z"}
self-hosted-codebase_api_1  | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Finished workflows for retool-event","organizationId":1,"pid":115,"requestId":"3a2f842c-5311-44b6-8a4d-ac209ea10483","timestamp":"2024-05-15T22:08:48.637Z"}
self-hosted-codebase_api_1  | {"endpoint":null,"level":"info","message":{"http":{"method":"POST","request":{"time":0.44336911100149157},"status_code":200,"url_base":"https://34.220.202.242:3000","url_path":"/api/user/resetPassword"},"type":"REQUEST_FINISH"},"pid":115,"requestId":"3a2f842c-5311-44b6-8a4d-ac209ea10483","timestamp":"2024-05-15T22:08:49.053Z"}
self-hosted-codebase_api_1  | {"level":"info","message":{"cpuUsagePercentForProcess":"0.02","isRequestHandlerProcess":false,"memoryStatsForContainer":{"freeSystemMemoryMb":28178.857984,"percentageFreeMemory":83.73,"totalSystemMemoryMb":33653.235712},"memoryStatsForProcess":{"heapTotalMb":247.025664,"heapUsedMb":230.319712,"percentageHeapUsed":93.24,"rssMb":415.903744},"namespace":"memoryUsage","pid":22},"timestamp":"2024-05-15T22:08:52.660Z"}
self-hosted-codebase_api_1  | {"level":"info","message":{"cpuUsagePercentForProcess":"0.03","isRequestHandlerProcess":true,"memoryStatsForContainer":{"freeSystemMemoryMb":28178.86208,"percentageFreeMemory":83.73,"totalSystemMemoryMb":33653.235712},"memoryStatsForProcess":{"heapTotalMb":389.718016,"heapUsedMb":367.02976,"percentageHeapUsed":94.18,"rssMb":577.753088},"namespace":"memoryUsage","pid":115},"timestamp":"2024-05-15T22:08:54.497Z"}
16

@Paulo , here is the requested information:

api-1                 | {"level":"info","message":{"http":{"method":"POST","url_base":"https://<retool_url>","url_path":"/api/user/resetPassword"},"type":"REQUEST_BEGIN"},"pid":53,"requestId":"410042f6-2c71-4fd1-8631-9c4aff75f4a3","timestamp":"2024-05-15T22:57:38.689Z"}
api-1                 | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Beginning workflows for retool-event","organizationId":1,"pid":53,"requestId":"410042f6-2c71-4fd1-8631-9c4aff75f4a3","timestamp":"2024-05-15T22:57:38.725Z"}
api-1                 | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Retrieved 0 workflows for retool-event","numWorkflows":0,"organizationId":1,"pid":53,"requestId":"410042f6-2c71-4fd1-8631-9c4aff75f4a3","timestamp":"2024-05-15T22:57:38.730Z"}
api-1                 | {"eventType":"RESET_PASSWORD_REQUEST","level":"info","message":"[RetoolEvents][queueWorkflowsForRetoolEvent] Finished workflows for retool-event","organizationId":1,"pid":53,"requestId":"410042f6-2c71-4fd1-8631-9c4aff75f4a3","timestamp":"2024-05-15T22:57:38.730Z"}
api-1                 | {"endpoint":null,"level":"info","message":{"http":{"method":"POST","request":{"time":0.4823500452041626},"status_code":200,"url_base":"https://<retool_url>","url_path":"/api/user/resetPassword"},"type":"REQUEST_FINISH"},"pid":53,"requestId":"410042f6-2c71-4fd1-8631-9c4aff75f4a3","timestamp":"2024-05-15T22:57:39.172Z"}
17

Could you share which environmental variables you added to the docker.env file? Feel free to hide all sensitive values.

18

@Paulo , here is the redacted content of our docker.env file:

## Set node environment to production
NODE_ENV=production

## Set the JWT secret for the API server
JWT_SECRET="<base64_encoded_string"

## Set and generate postgres credentials
POSTGRES_DB=hammerhead_production
POSTGRES_USER=<username>
POSTGRES_HOST=<server_name>.postgres.database.azure.com
POSTGRES_PORT=5432
POSTGRES_PASSWORD="<password>"

## Set and generate retooldb postgres credentials
RETOOLDB_POSTGRES_DB=retool
RETOOLDB_POSTGRES_USER=<username>
RETOOLDB_POSTGRES_HOST=<server_name>.postgres.database.azure.com
RETOOLDB_POSTGRES_PORT=5432
RETOOLDB_POSTGRES_PASSWORD="<password>"

# Change 'retool-poc.az.radio-canada.ca' to retool.yourcompany.com to set up SSL properly
DOMAINS=<retool_url> -> http://api:3000

## Used to create links for your users, like new user invitations and forgotten password resets
## The backend tries to guess this, but it can be incorrect if there’s a proxy in front of the website
BASE_DOMAIN=https://<retool_url>

## Set key to encrypt and decrypt database passwords, etc.
ENCRYPTION_KEY=<base64_encoded_string>

## License key
LICENSE_KEY=<license_key>

## Uncomment this line if HTTPS is not set up
#COOKIE_INSECURE=true

## This variable augments the database query timeout
DBCONNECTOR_QUERY_TIMEOUT_MS=360000
19

Could you DM the email account we are trying to send emails to?

20

I did not hear back from you. If we are still having this issue, something that may worth doing is:

  1. Delete the org and user manually from the external db.
  2. Redeploy and sign up with a non corporate email.
  3. Try to get a reset password email there.