So yeah today we realized one of our development servers had 0% free space left... after checking it out with

du -h --max-depth=1

in a few places in the filesystem, it turned out that we had a 66 gigabye Apache httpd error log.

66 GB error log


That's what you get for setting the log level to debug and not rolling over your error logs...