Extremelly slow resource queries since 15th Feb

Hello,

We have several apps that we have built on Retool and have been using for over a year. Many of these apps utilise OpenAPI resources which have always worked well for us.

Since the morning of the 15th of February, many of these OpenAPI query requests have been running extremely slowly (taking in excess of 90 seconds to complete the query). When I test the endpoints myself via Postman, these same requests take only 100-200ms. This represents an increase of 450%+ in execution time in Retool! As a result, many of our Retool apps have become effectively unusable.

Our API is processing requests in a timely manner (~200ms) when they do not come from Retool. I have also see someone else post a similar problem in this forum relating to their MongoDB resource.

I have also noticed this incredibly long execution time even when running "Test connection" on the OpenAPI connection in the Resources tab.

Does anybody know what could be causing this?

Hey @Isaac1 - Thanks for flagging, seeing the same thing on my side. We're opening an incident to investigate this now!

2 Likes

We have folks online digging into this more, we're going to keep our status page updated with the latest!

1 Like

Hey @Isaac1 - We've been digging a lot into this today and definitely see the slowdown in response times starting for your org yesterday, but doesn't seem much more widespread than that in general. Is it specific to a particular spec, and did that spec change recently?

We're also going to follow up on the Mongo aspect, but those aren't jumping out as much in our logs as the openAPI ones.

Hi @jmann, thank you for looking into this.

We recently updated our FastAPI version (which is generating our OpenAPI spec) from 0.70.1 to 0.109.2, which included an update of OpenAPI version from 3.0.2 to 3.1.0. I can see that if I try and use the last version of our spec on OpenAPI version 3.0.2, there is no delay in the queries, but the first version of the spec using version 3.1.0 has the excessive execution time.

I can't seem to tell why this is happening, but in the interim, we can downgrade back to 3.0.2 to get our apps back up and running. I'm happy to share both the last working version and the first problematic version of our OpenAPI spec with you if that would help you to debug from your end!

No problem! It does seem to be some discrepancy between the spec and the node package we're using to resolve it. Here's essentially what we're doing if you'd like to test locally to see where in the spec the hangup might be coming from:

const SwaggerClient = require('swagger-client')

SwaggerClient.resolve({ url: YOUR_SPEC_URL});

Feel free to DM me the problematic json as well for us to document, but can't yet commit to debugging on our side since at this point it does seem to be coming from the spec somewhere (and may or may not be something we could address on our side in the short term).