Significant Variation in Query Performance

Hi all,

We recently noticed that two of our more important queries were timing out during morning business hours (EST). Typically, the issue will resolve by 10:30 am and the queries will begin performing significantly faster. The cutoff seems to vary day to day, with some days taking longer than others.

We've done a good bit of testing on our end and have comfirmed there are no concurrent processes running on our end (PostgreSQL database on AWS) that might interfere.

I'm curious if anyone else has experienced anything like this, and if you eventually found a root cause.

Take a look at these stats from this morning: