Query takes longer when cached

I have a query in Retool, it is a simple "List all contacts" query

SELECT
  *
FROM
  contacts
ORDER BY
  contacts.first_name ASC;

When I disable retool cache this query takes about 1.2s on average. But with cache enabled it can take anywhere from 2s - 17s. This seems to be happening on multiple queries, enabling cache typically doubled run time.

FWIW I notice this most on Monday mornings and it seems to level out over the week, but that is purely anecdotal.