On-Prem diskstore-*.json files eating up all the disk space

💬 Self Hosted Retool
1

Some queries, especially ones with large results (like ~2kb), started to fail across our instance recently. We saw a "Disk out of space error". Sure enough, our 75GB storage volume had no empty space.

After SSH'ing into the server, we found certain /var/lib/docker/**/cache folders were growing like crazy, full of files that match the diskstore-*.json format. Looking at one of the files, it just appears to be a GraphQL query specific to our GraphQL backend.

I'm unsure how to safely proceed here - queries are failing, I've shut cacheing off across the platform, but how can I get rid of all of these diskstore-*.json files? Can I manually delete them?

2

Hey @markmarkoh

Welcome to the community,

Yes, you can safely delete these diskstore-*.json files manually because they are just cache files and can be recreated by the system if needed.

  1. Stop the Docker containers or the backend service using the cache.
  2. Delete the cache files manually with a command like:
sudo find /var/lib/docker -type f -name 'diskstore-*.json' -delete
  1. Check your disk space (df -h) to confirm space is freed.
  2. Restart your backend service or Docker containers.
  3. Monitor disk usage and consider keeping caching off or setting cache limits to avoid this issue in the future.
2 Likes
3

Thanks @WidleStudioLLP, that makes sense. I preemptively deleted a small batch of the diskstore-*.json files to just get our instance back into a good state, good to know I can delete the lot of the them.

4

@markmarkoh

If you want, I can help exact commands.

1 Like
5

Welcome to the community, @markmarkoh! There are a couple other things you can do to help alleviate space usage, if you find yourself running low.

  1. Prune any hanging Docker images with docker image prune
  2. Thin out the page_saves table with page save compaction
1 Like
6

Thanks all. To follow up here:

I ran a docker-compose down so I could clean up some of the diskstore-*.json files, but the docker-compose down did the cleanup itself.

When I restarted the instance with docker-compose up -d, the disk space was clear.

Before:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G   59G     0 100% /

After:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G   14G   45G  23% /
7

Good to know! Thanks for sharing an update, @markmarkoh. :+1: It's generally a good idea to restart your containers every now and then anyway, whether for updating or otherwise, so I'm curious to see how long it takes for these files to accumulate.