Since about a week ago our app users have started to report sporadic cases of "Error: 14 UNAVAILABLE: read ECONNRESET" on queries going to an array of our gRPC services.
These queries never hit our infrastructure / services, and the user is often able to retry with success.
We have a feeling something is going wrong in the proxy layer in Retool between the client and Retool itself.
Is anyone else experiencing this for gRPC resources?
Can anyone from the Retool team have a look if there is any elevated error rates overall for gRCP connections?
We've been seeing this as well, along with "socket hang up" issues. Early indications suggest it has something to do with Python blocks, but no solid explanation as of yet. Recommendation has been to migrate blocks to JS, though in our case that isn't fully possible.
Hey @Martin_Christiansen! Thanks for reaching out. It definitely looks like there's been a significant spike in these errors since the middle of June. I'll talk to the owning team and see if we can identify the root cause.
As soon as there is news to share, I'll follow up here with those updates.
The timing seems to indicate that one of our recent updates to the gRPC connection pooler is likely responsible, but it's not a widespread issue. I'm guessing that the volume of requests you're making is saturating the resource connector.
I've temporarily excluded your org from this new rule while we take a closer look and maybe tweak the pooler's settings. On your end, it's probably worth evaluating whether the volume of requests you're making can be reduced! Let me know if you notice a reduction in these errors going forward.
This seem to indeed have resolved the issue, I did not get any connecition resets when manually testing for a minute.
We don't send any redundant requests I'm afraid, we just have all our services in Microservices connected with gRPC - I could imagine we are just not that many having the amount of applications / volume to provoke the issue.
again thanks for looking into this and so far resolving it :-)!
A recent release related to gRPC reintroduced this, confirmed by the Retool team.
This is ultra critical for us as there is no recovery path when connections stalls in query chains in the app as there is no possible retry recovery besides the app being totally reloaded.
This is causing havoc with half-way executed chains in our apps as we heavily use gRPC to communicate with our services.
The connection pooler that we previously rolled out via feature flag recently went GA, meaning we no longer have the ability to exclude high-traffic orgs. Engineering is aware and prioritizing a fix for you! In the short term, we'll likely expose a toggle that will allow you to disable pooling on individual resources. In general, though, we can hopefully tune the pooler to perform more reliably at volume.
I'll continue to check in with Engineering and give you updates as they come in.
We literally have 14 days downtime on our warehouse returns and order management made in Retool due to this.. People are loosing their minds to say the least.
Yeah, disabling pooling via the new toggle that we've exposed should return your org to the opted-out state that it was in prior to the 3.375 release. I don't see any backend errors associated with your org in the last ~90 minutes or so, which is hopefully a good sign!
Thanks for getting this through at last! I think they should fix that pooler in general for gRPC connections to avoid new and other customers getting into issues. Our volume is not really that high.
Yeah no reports of it being broke still so far, so hopefully it fixed it!
I think they should fix that pooler in general for gRPC connections to avoid new and other customers getting into issues. Our volume is not really that high.