Very long request processing time

Hi @Francesko_Petrarka, @Matt_N, @dguzzo, @Mitchel_Smith, @Shawn_Optipath, @Shawn_Crocker, thanks for your patience on this one! :folded_hands:

Our team has deployed a fix and is now monitoring it through peak traffic. You can see the latest on our status page. Are you still seeing slow queries today?

Any idea yet what the root cause was and what is being done to prevent it in the future. We are still in UAT mode for all of our apps switching over to Retool and hoping this is a very isolated incident. We don't have hundreds of people doing timecards and PTO requests yet, but when that day comes an issue like this will be unacceptable.

We had really slow response times this morning around the time that was last reported on the status page so I'm guessing the fix worked.

thanks,
Mike

I have been using retool db for a few years in my companies tools and this is the first time I have noted any type of performance degradation that lasted long enough to remember. Just wanted to add a note on my experience with the system.

@mikeS, thanks for following up, and for the context on your UAT rollout.

Here’s what we know so far. Our infrastructure was overloaded during periods of high traffic. We normally scale up automatically to handle increased demand, but a combination of factors kept that from working in this case.

To prevent it from happening again, we’re updating our autoscaling and recovery mechanisms. We’re also still investigating and fixing performance issues.

We know how disruptive this was, and we’re sorry. If you notice slow response times again, please reply here with roughly when it happened. That helps the team line it up with what they’re seeing. You can also follow the status page for updates.

@Shawn_Crocker, thanks for sharing your experience too! :folded_hands:

1 Like