All Retool db queries timing out

App Building
21

Hi @Shawn_Optipath,

I was just about to say, our engineers suggested trying to use a PostgreSQL resource connection when the issue pops up, but it sounds like you already tested that and it doesn't take a performance hit, which is great to hear.

Switching over to PostgreSQL resource connections is likely the best solution in the short term given the seasonal time constraints. We can definitely look to change the throttle issue as well for a longer term solution as needed.

Our engineer was able to see from your logs that the connectivity issues occur exactly when your workflows are running :face_with_monocle:

They went on to add that the app is "making crazy amount of connections during this time and workflow queries are also failing in that time". From the image I was it looks like there are 60+ connections occurring at once.

CleanShot 2024-11-21 at 14.23.39@2x
CleanShot 2024-11-21 at 14.23.39@2x800×496 64.3 KB

CleanShot 2024-11-21 at 14.23.55@2x
CleanShot 2024-11-21 at 14.23.55@2x1594×300 65.6 KB

Given that both of these occurring in conjunction, the solution in my mind is to see if these can be alleviated in their own ways.

Figuring out why so many connections are being made and reducing the number of connections would be top priority.

I know you mentioned the workflows are triggered by webhooks from job boards, but if the workflows can be spread out or run periodically that could also open up bandwidth :sweat_smile:

Maybe some type of scheduler third party (Temporal?) could be used to work as a middleman between the incoming data and the workflows to act as a sort of a load balancer :thinking: food for thought.

22

Thanks for digging a little deeper @Jack_T. That helped me identify the culprit. I did not know the workflow schedules were GMT. I was thinking it is local time. I have the larger one running every 2 days at 12 am which is 4pm PST.

I did previously note this workflow timing out during the loop blocks. It was on my list to improve this but I had to switch gears. Now it is a priority! :sweat_smile:

For the moment I have paused the workflow until I can get the loops under control. I'll definitely create another post regarding loop optimization.

1 Like
23

Glad I could help!

This situation is making a lot more sense now :sweat_smile:

Sounds like a plan, we definitely have some resources in our docs on loop optimizations as well as lots of user posts in these forums about the subject.

Feel free to make a new post and we can help as best we can! We also have some exciting new features related to workflow loops coming out very soon so stay tuned!

1 Like