Intermittent "Invalid Login Credentials" with Genesys API Using OAuth2 Client Credentials in Retool

Lawrence_Drayton · October 2, 2024, 4:35pm

Hello Retool Community,

I'm experiencing a critical issue with Retool workflows and apps that connect to the Genesys API using OAuth2 client credentials. This problem is causing significant disruptions to our operations, and I'm seeking your assistance to resolve it.

Issue Summary:

Setup: For over a year, I've been using a Retool resource configured with OAuth2 Client Credentials to authenticate with the Genesys API (https://login.mypurecloud.com.au/oauth/token). This setup has been stable and reliable until recently.
Problem Start Time: Around midday Sydney time (~7 PM ET) on 28/09, the resource began intermittently failing.
Error Message:

statusCode: 400
error: Bad Request
message: {"message":"Invalid login credentials.","code":"bad.credentials","status":401}

Impact: All workflows and apps that depend on this resource are failing. We run approximately 100k workflow runs per month, so this issue is critical.

Troubleshooting Steps Taken:

Verified Credentials Outside Retool:

Tested the same client credentials using Postman and cURL, and they work perfectly.
Example cURL Request:

curl -X POST \
  -u "client_id:client_secret" \
  -d "grant_type=client_credentials" \
  https://login.mypurecloud.com.au/oauth/token

2.Response:*

{
  "access_token": "[ACCESS_TOKEN]",
  "token_type": "bearer",
  "expires_in": 86399
}

Used the obtained access_token to make API calls successfully.

Switched to Authorization Code Grant:

Changed the resource in Retool to use OAuth2 Authorization Code Grant as a temporary workaround.
This method works but is not ideal because it requires manual re-authentication and isn't suitable for public apps.

Created New Resources in Retool:

Created new resources with the same client credentials.
Observed inconsistent behavior:
- Sometimes, one resource works while an identical one fails with the same error.
- The working and failing resources can switch without any changes made to them.

Observed Inconsistent Behavior Within Workflows:

In a single workflow, one step using the resource succeeds, and the next step fails with the "Invalid login credentials" error.
No changes are made between these steps.

Additional Details:

No Changes on Our End: The client credentials and resource configurations haven't been altered.
Intermittent Issue: The problem occurs sporadically without a discernible pattern.
Retool Support Response:
- Suggested that the issue might be with the Genesys API.
- Pointed me to external forums, but similar issues there were related to incorrect region settings, which isn't the case here.
Genesys API Confirmation:
- Since the credentials work outside Retool consistently, it seems unlikely that the issue is with Genesys.

My Hypothesis:

Retool OAuth2 Handling Issue:
- It appears there might be a problem with how Retool handles OAuth2 client credentials, possibly with token caching or renewal.
- The intermittent nature suggests it might be related to how tokens are stored or retrieved within Retool.

Questions for the Community:

Has anyone else experienced intermittent OAuth2 authentication issues with client credentials in Retool?
Are there known issues or configurations in Retool that could cause this behavior?
What are the best practices for setting up OAuth2 client credentials in Retool to ensure consistent authentication?
Is there a way to debug or get more detailed logs from Retool to pinpoint the issue?

Impact and Urgency:

This issue is critical and affects all our customer-facing operations.
We're planning to move to an enterprise plan due to our usage volume, but we need to resolve this issue urgently.

Any assistance or guidance would be greatly appreciated.

Thank you for your time!

Best regards,

Lawrence

Darren · October 3, 2024, 4:09am

Thanks for reaching out, @Lawrence_Drayton! And welcome to the community.

Just to confirm - you aren't experiencing any of the same issues after switching it to use the "Authorization Code Grant" flow? Assuming Genesys gives you a refresh_token, this can hopefully be a relatively long-term solution with minimal re-authentication required. You can even choose to "Share credentials with users" via the resource's configuration in order to truly mimic the client credentials experience.

Shifting into problem-solving mode, the error you're seeing indicates that Retool is sending an invalid auth token with the API request. There are a few possibilities here; it could be expired, invalidated, malformed, or even empty. The best way to narrow it down is to set up your resource to point to an endpoint that you control so that you can manually inspect the request. There are a variety of services for easily setting up such an endpoint, such as Beeceptor, if you don't have infra for quickly doing so.

Given the high volume of requests, my very tentative guess is that Genesys is enforcing some sort of rate limit by invalidating tokens. It's a few years old at this point but this post in the Genesys forums suggests that it's at least feasible. I can't think of anything on the Retool side that might be causing you to hit this theoretical limit now, when you never did previously, but it's conceivable that something is causing excessive re-authentication requests. I'll see how deep that rabbit hole goes and get back to you if I find anything interesting! In the meantime - and if you have access to any Genesys logs - I'd be interested to know if you see anything that might corroborate this theory.

All that said, I do think there's a fairly straightforward workaround that you can implement if there really is an issue with the Retool OAuth2 implementation - namely, a "Custom Auth" pattern that pretty much puts all the control in your hands, as if you were using cURL or Postman:

I hope that gives you a place to start looking, at the very least, and maybe even a long-term solution. I'll do some more digging on my end and let you know if I find anything.