Cold shard eviction

[knighton]

Description of changes:

Cold shard eviction. Keep filesystem usage under some limit. Do this by dropping least recently accessed shards as needed when downloading new shards. This is a general solution that works with random access.

• Redesign shard states: MISSING, DOWNLOADING, PRESENT (evicting happens under the lock)
• Initialize shard_states by walking and normalizing stream local dirs
• Implement shard eviction
• Redesign StreamingDataset keep-related args to add cache_limit, drop keep_raw
• Bookkeeping for shard access times and tracking disk usage
• Automatically evict shards based on disk usage limit

Issue #, if available:

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

[dakinggg]

For testing, would be good to have

1. unit test for simple shard eviction (low cache limit)
2. mosaicgpt run with shard eviction to make sure it still runs, has good throughput, loss looks fine.