Some queries, especially ones with large results (like ~2kb), started to fail across our instance recently. We saw a "Disk out of space error". Sure enough, our 75GB storage volume had no empty space.
After SSH'ing into the server, we found certain /var/lib/docker/**/cache folders were growing like crazy, full of files that match the diskstore-*.json format. Looking at one of the files, it just appears to be a GraphQL query specific to our GraphQL backend.
I'm unsure how to safely proceed here - queries are failing, I've shut cacheing off across the platform, but how can I get rid of all of these diskstore-*.json files? Can I manually delete them?
Hey @markmarkoh
Welcome to the community,
Yes, you can safely delete these diskstore-*.json
files manually because they are just cache files and can be recreated by the system if needed.
- Stop the Docker containers or the backend service using the cache.
- Delete the cache files manually with a command like:
sudo find /var/lib/docker -type f -name 'diskstore-*.json' -delete
- Check your disk space (
df -h
) to confirm space is freed.
- Restart your backend service or Docker containers.
- Monitor disk usage and consider keeping caching off or setting cache limits to avoid this issue in the future.
2 Likes
Thanks @WidleStudioLLP, that makes sense. I preemptively deleted a small batch of the diskstore-*.json files to just get our instance back into a good state, good to know I can delete the lot of the them.
@markmarkoh
If you want, I can help exact commands.
1 Like
Welcome to the community, @markmarkoh! There are a couple other things you can do to help alleviate space usage, if you find yourself running low.
- Prune any hanging Docker images with
docker image prune
- Thin out the
page_saves
table with page save compaction
1 Like
Thanks all. To follow up here:
I ran a docker-compose down
so I could clean up some of the diskstore-*.json files, but the docker-compose down
did the cleanup itself.
When I restarted the instance with docker-compose up -d
, the disk space was clear.
Before:
Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 59G 0 100% /
After:
Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 14G 45G 23% /
Good to know! Thanks for sharing an update, @markmarkoh.
It's generally a good idea to restart your containers every now and then anyway, whether for updating or otherwise, so I'm curious to see how long it takes for these files to accumulate.