HDDS-4337. Implement RocksDB options cache for new datanode DB utilities.

[errose28]

What changes were proposed in this pull request

Add a cache of ColumnFamilyOptions objects to AbstractDatanodeStore, so that each ConfigurationSource object used to construct a container will result in the same RocksDB ColumnFamilyOptions object being used. Since the same configuration instance is used for all containers on a datanode, this results in all containers on a datanode using the same ColumnFamilyOptions, and by extension the same RocksDB metadata cache. A configuration parameter is added for adjusting the size of this cache. Each RocksDB instance will have its own DBOptions, since this requires the use of RocksDB Statistics objects which are not thread safe and cannot be shared.

The DBStoreBuilder class was refactored to make configuration easier. It is now possible to use the same configurable column family options for all column families, and it is easier to reason about which configurations will be applied. It is no longer an error to register a table twice with the builder, the second call will simply override the first, which is usually the expected behavior in setters for builders. There should be no other functionality changes to this class.

What is the link to the Apache JIRA

HDDS-4337

How was this patch tested?

Automated Testing

• DBStoreBuilder unit tests were updated.
• A unit test was added to TestKeyValueContainer to test that ColumnFamilyOptions objects are shared between containers with the same configuration.

Manual Testing

Manual testing was also done with a docker cluster to verify that the same cache is used for all containers on a datanode. Steps:

1. Start an ozone docker cluster and enter a datanode.
2. Write 100 keys: ozone freon ockg -n 100
3. Close the pipeline those keys were written to.
   • ozone admin pipeline list to get the pipeline IDs.
   • ozone admin pipeline close <id> to close the pipeline.
4. Write 100 more keys.
5. Get the address of the cache object from the RocksDB log for each of the two containers:
   grep -A5 "block_cache:" /data/hdds/hdds/*/current/containerDir0/1/metadata/1-dn-container.db/LOG
   and
   grep -A5 "block_cache:" /data/hdds/hdds/*/current/containerDir0/2/metadata/2-dn-container.db/LOG

• Make sure that the address listed after block_cache: is the same for both containers.

[sodonnel]

If I understand this correctly, in AbstractDatanodeStore.start(), it pulls the default CF options from a static variable where they are setup on first use. This creates a single LRUCache(CACHE_SIZE) which is passed into the the DBStoreBuilder for all container RocksDBs. This means they should share the LRU cache.

Its a little hard to follow the logic to ensure this cache is definitely being reused. Could you create a docker cluster, use Freon to create some containers and then check the RocksDB log files to ensure the same cache instance is being used - There is an example of what to look for to prove this in https://issues.apache.org/jira/browse/HDDS-4246.

I also think the 8MB shared cache is probably too small. Could we add a configuration for it, and make the default higher, maybe 64MB for now? What do you think?

[errose28]

Thanks, @sodonnel , I will add a setting for cache size and test this in a docker cluster. Looking at the rocks docs it appears those LRUCache objects can't be accidentally modified by callers, so I could also expose it as public and add a unit test to check that the same cache is used.

[sodonnel]

These changes look good to me. I am +1 on them, so I will go ahead and commit.