To answer my own question, based on other forum posts, I'm guessing it's because this is an OpenAPI 3.1 specification, and Retool is failing to properly parse our API spec because it only supports up to OpenAPI 3.0.
I've created a REST API resource for the same endpoint, and it is not having any of these performance issues. This isn't ideal, since we lose out on the ability to know all available endpoints, parameters, types and response payloads, but at least it unblocks our ability to use Retool. That said, it would be nice to know if OpenAPI 3.1 support is on the roadmap, and when we might expect this support to ship.