SSH Tunneling error - "All configured authentication methods failed"

[aking43]

Hoping to get some help here. I'm setting up SSH tunneling to a few databases in AWS - RDS and DocumentDB. I've verified that I can connect to the bastion server and then get to the database via software such as Studio 3T using a username and password on the bastion host.
I've created the retool user on the AWS Linux box using the "sudo adduser retool --password NP" command.
I then followed the remaining steps, copied the data from the public key to the authorized_keys file, set permissions, as well as whitelisted the Retool IPs.
No matter what I do, I get the error below with the statement that it can't authenticate. The user is created, the public key is there....I'm out of ideas. Any help would be great.

Login as root

sudo su

Create the authorized_keys file if it does not exist yet

mkdir -p /home/retool/.ssh
touch /home/retool/.ssh/authorized_keys

Use your favorite editor to add Retool's public key to the file

vim /home/retool/.ssh/authorized_keys

Set permissions on the authorized_keys file

chmod 644 /home/retool/.ssh/authorized_keys

Change owner of authorized_keys file to Retool

chown retool:retool /home/retool/.ssh/authorized_keys

SSH tunneling

[Kabirdas]

Hey @aking43!

It certainly looks from my end as though you have things set up correctly. Would you mind sharing logs from your SSH host so that we can see if there's any additional information there? Logs from your Retool containers may be helpful as well.

[aking43]

Hey @Kabirdas ,

Here's what I see in the audit.log on the EC2 instance when trying to connect via retool. Also, we currently are completing a POC using the free version of Retool, so we do not have access to logs, unless there is another way to get the logs?

type=CRYPTO_KEY_USER msg=audit(1679661124.753:1550): pid=29832 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:a2:06:1e:98:96:af:55:3e:5b:f5:80:ef:10:39:61:c3:93:31:4a:92:50:69:be:c4:0c:76:ee:b8:fc:19:a0:df direction=? spid=29832 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" SUID="root"
type=USER_LOGIN msg=audit(1679661124.753:1551): pid=29832 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="retool" exe="/usr/sbin/sshd" hostname=? addr=35.90.103.132 terminal=ssh res=failed'UID="root" AUID="unset"
type=CRYPTO_KEY_USER msg=audit(1679661125.063:1552): pid=29835 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:a2:06:1e:98:96:af:55:3e:5b:f5:80:ef:10:39:61:c3:93:31:4a:92:50:69:be:c4:0c:76:ee:b8:fc:19:a0:df direction=? spid=29835 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" SUID="root"
type=CRYPTO_SESSION msg=audit(1679661125.133:1553): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes128-gcm@openssh.com ksize=128 mac=<implicit> pfs=curve25519-sha256@libssh.org spid=29835 suid=74 rport=47647 laddr=10.40.162.41 lport=22  exe="/usr/sbin/sshd" hostname=? addr=35.90.103.132 terminal=? res=success'UID="root" AUID="unset" SUID="sshd"
type=CRYPTO_SESSION msg=audit(1679661125.133:1554): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-client cipher=aes128-gcm@openssh.com ksize=128 mac=<implicit> pfs=curve25519-sha256@libssh.org spid=29835 suid=74 rport=47647 laddr=10.40.162.41 lport=22  exe="/usr/sbin/sshd" hostname=? addr=35.90.103.132 terminal=? res=success'UID="root" AUID="unset" SUID="sshd"
type=CRYPTO_KEY_USER msg=audit(1679661125.543:1555): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=29835 suid=74 rport=47647 laddr=10.40.162.41 lport=22  exe="/usr/sbin/sshd" hostname=? addr=35.90.103.132 terminal=? res=success'UID="root" AUID="unset" SUID="sshd"
type=CRYPTO_KEY_USER msg=audit(1679661125.543:1556): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:a2:06:1e:98:96:af:55:3e:5b:f5:80:ef:10:39:61:c3:93:31:4a:92:50:69:be:c4:0c:76:ee:b8:fc:19:a0:df direction=? spid=29835 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" SUID="sshd"
type=USER_ERR msg=audit(1679661125.543:1557): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=35.90.103.132 addr=35.90.103.132 terminal=ssh res=failed'UID="root" AUID="unset"
type=CRYPTO_KEY_USER msg=audit(1679661125.543:1558): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:a2:06:1e:98:96:af:55:3e:5b:f5:80:ef:10:39:61:c3:93:31:4a:92:50:69:be:c4:0c:76:ee:b8:fc:19:a0:df direction=? spid=29834 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" SUID="root"
type=USER_LOGIN msg=audit(1679661125.543:1559): pid=29834 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="retool" exe="/usr/sbin/sshd" hostname=? addr=35.90.103.132 terminal=ssh res=failed'UID="root" AUID="unset"

[Kabirdas]

What are your permissions on the .ssh folder itself? Are they 755 or less as mentioned in this StackExchange post?

And as to the logs - I had assumed you were self-hosted for some reason, sorry as long as you're on Cloud additional information will be in our backend.

[aking43]

Np!
I verified that the folder and file permissions are correct according to that article.

I appreciate the help!

[Kabirdas]

Got it, thanks! I'm checking in with the team about this as well, these setups can be a bit tricky. I know you mentioned copying the ssh key into /home/retool/.ssh/authorized_keys but would you mind verifying that it's correct by running something like

less home/retool/.ssh/authorized_keys

and checking that against the key you downloaded from Retool?

If you could also check your OpenSSH version as well that may be helpful. Retool's public key is generated using ssh-rsa which was deperecated with version 8.8. So, if you're using a newer version, you can try adding

PubkeyAcceptedKeyTypes +ssh-rsa

to your ssh_config file as mentioned here.

[aking43]

Thanks @Kabirdas .
I have attached some filtered screenshots, and I have verified that the public key that's downloaded from Retool is in the authorized_keys.
I've also added the PubkeyAcceptedKeyTypes +ssh-rsa to the ssh_config file. I restarted the service and I still get the same error.

[aking43]

Also, just to mention, I can switch to the retool user without any password requirements so I take it the user works just fine from the Retool perspective?

[Kabirdas]

Can you try the following to see if it gives us any more information?

1. Create an ssh keypair
2. Use the public key in place of Retool key in authorizedKeys
3. Attempt to connect to your server using the private key. Retool uses tunnel-ssh to connect to your host, so it would be good to try using that as well. You can find the library here.

Taking another look at the screenshot of your permissions that you sent over, it looks like retool is the owner of /home/retool but not /home/retool/.ssh/authorized_keys, you might try running the following again as well:

chown retool:retool /home/retool/.ssh/authorized_keys

[aking43]

Hey @Kabirdas,
I was able to get the authentication to work for a test user - I set it up the same as the retool user. Once I got it setup and working I matched the settings of the test user to the retool user. It still gives the same error in retool.

[aking43]

I setup the SSH tunnel on a ubuntu server and got it working. Now I'm just getting an error that looks to be related to the bastion host not being able to find the database server, so just working on that.
I appreciate the help you've provided!

[aking43]

Got it working! Thanks @Kabirdas for your help.
Ran these commands for the users on a new Ubuntu server-

Added the PubkeyAcceptedAlgorithms +ssh-rsa to the sshd_config, restarted the service.

Ensured the security groups have 22 inbound - restricted to Retool's IPs and then allowed all ports on the DocumentDB database. Also changed the port to a different port on the DocumentDB side.

[Kabirdas]

Awesome @aking43! Glad to hear you were able to get things working and thanks for summarizing your steps here in case other folks come across this thread!

Would you mind my asking if there's any difference you can point to between this new Ubuntu server and the one that wasn't working?

[aking43]

Np!
My main focus and troubleshooting was on a AWS Linux OS server. I had tried an ubuntu server earlier, but I think I had missed the permissions on the .ssh folder and the sshd_config file. This was earlier on so I then continued my troubleshooting on the AWS Linux OS server - which would never work.
Hope that helps! Appreciate your help as well!

[harelguy]

I just encountered this same issue, using a new Amazon Linux 2023 server.
Apparently they specifically deprecated the ssh-rsa algorithm as it is no longer considered secure enough, so I needed to follow their instructions here. You might need to reboot the server for this to take effect.

Retool - any chance of upgrading your SSH client so that it can work with the newer algorithms?

Default SSH server configuration - Amazon Linux 2023

[Kabirdas]

Thanks for sharing this @harelguy

Updating the SSH algorithm used for key generation is something on the dev team's radar. There aren't concrete plans for it at the moment but I can pass it along here if it gets included!