Audit log table cleanup

💬 Feature Requests
1

I would like to have a new feature on Retool to allow auto-cleanup of the audit log table with data older than 90 days or so. I have a situation where a deployment is being heavily used and it's generating too many entries in the audit logs (50 GB), and this is taking forever to clean up. Could you please consider adding a background process to perform this task? Thanks!

1 Like
3

Hello @Marcus_Santos!

Are you self hosted or on the cloud?

I can definitely make a feature request for such a tool to be native to Retool.

For work arounds in the short term, if you are self hosted, there are a number of tools that you can use to clean up that Postgres table containing the logs.

For your use case where you want to clear out logs that are older than 90 days you could use SQL native tools combined with automated jobs to accomplish the following steps.

  1. Partition table by month using pg_partman
  2. Schedule a weekly cron job(could potentially use a Workflow for this!) to:
  3. Drop old partitions > 3 months
  4. Call pg_repack if necessary, to reclaim data
  5. Export logs > 6 months to cold storage (optional)
4

Hi Jack,

Yes, it's a self-hosted deployment on Google Cloud, using a managed Cloud SQL instance. The client is already performing these workarounds, but it's asking for a native feature. Let me know when you create the feature request and when it's planned to be released.

Thank you!

1 Like
6

Hi @Marcus_Santos thanks for getting back to me with those details!

I have made the feature request and will update this thread with any news I hear. But I am not sure how close this would be on our feature roadmap.

Self-hosted SQL instance management falls outside of the scope of Retool's product :sweat: but if you follow the above steps, it should be very manageable to cut down the table size with partitions automatically and consistently remove the old logs.

7

Hey there,

@Jack_T, thanks a lot for creating the feature request.

a huge +1 for this feature request. This would be so helpful to keep the database of Retool at a manageable size. As a customer, the unbound and quick growth of that table took us by surprise.

Retool manages all other aspects of this table, i.e., table creation, index creation, writing, and reading. As a customer, going inside this database and changing the table partitioning would put us at risk that the next update of Retool doesn't run cleanly, when this update performs a db migration of that table.

With audit log cleanup as a native feature, Retool retains agency to change db schema, without breaking customer installations.

All the best
Michael

2 Likes