RocksDB系列二：RocksDB Option

 RocksDB用戶可以通過Options類將配置信息傳入引擎，除此之外，還可以以下其他方法設置，分別爲：

1. 通過option file生成一個option class
2. 從option string中獲取option 信息
3. 從string map中獲取option信息

option string

  用戶可以調用GetColumnFamilyOptionsFromString() or GetDBOptionsFromString()將option string 傳入方法，就可以解析出string中的option信息，也可以使擁GetBlockBasedTableOptionsFromString() and GetPlainTableOptionsFromString() 方法獲取表格中的配置信息。每個option信息在option string中以<option_name>:<option_value>傳入，多個option之間以;分割。
例如：

table_factory=PlainTable;prefix_extractor=rocksdb.CappedPrefix.13;comparator=leveldb.BytewiseComparator;compression_per_level=kBZip2Compression:kBZip2Compression:kBZip2Compression:kNoCompression:kZlibCompression:kBZip2Compression:kSnappyCompression;max_bytes_for_level_base=986;bloom_locality=8016;target_file_size_base=4294976376;memtable_huge_page_size=2557;max_successive_merges=5497;max_sequential_skip_in_iterations=4294971408;arena_block_size=1893;target_file_size_multiplier=35;min_write_buffer_number_to_merge=9;max_write_buffer_number=84;write_buffer_size=1653;max_compaction_bytes=64;max_bytes_for_level_multiplier=60;memtable_factory=SkipListFactory;compression=kNoCompression;bottommost_compression=kDisableCompressionOption;min_partial_merge_operands=7576;level0_stop_writes_trigger=33;num_levels=99;level0_slowdown_writes_trigger=22;level0_file_num_compaction_trigger=14;compaction_filter=urxcqstuwnCompactionFilter;soft_rate_limit=530.615385;soft_pending_compaction_bytes_limit=0;max_write_buffer_number_to_maintain=84;verify_checksums_in_compaction=false;merge_operator=aabcxehazrMergeOperator;memtable_prefix_bloom_size_ratio=0.4642;memtable_insert_with_hint_prefix_extractor=rocksdb.CappedPrefix.13;paranoid_file_checks=true;force_consistency_checks=true;inplace_update_num_locks=7429;optimize_filters_for_hits=false;level_compaction_dynamic_level_bytes=false;inplace_update_support=false;compaction_style=kCompactionStyleFIFO;purge_redundant_kvs_while_flush=true;hard_pending_compaction_bytes_limit=0;disable_auto_compactions=false;report_bg_io_stats=true;compaction_filter_factory=mpudlojcujCompactionFilterFactory;

options map

類似的，用戶可以通過string map獲取配置信息，需要調用以下函數：GetColumnFamilyOptionsFromMap(), GetDBOptionsFromMap(), GetBlockBasedTableOptionsFromMap() or GetPlainTableOptionsFromMap()。
例如：

 std::unordered_map<std::string, std::string> cf_options_map = {
      {"write_buffer_size", "1"},
      {"max_write_buffer_number", "2"},
      {"min_write_buffer_number_to_merge", "3"},
      {"max_write_buffer_number_to_maintain", "99"},
      {"compression", "kSnappyCompression"},
      {"compression_per_level",
       "kNoCompression:"
       "kSnappyCompression:"
       "kZlibCompression:"
       "kBZip2Compression:"
       "kLZ4Compression:"
       "kLZ4HCCompression:"
       "kXpressCompression:"
       "kZSTD:"
       "kZSTDNotFinalCompression"},
      {"bottommost_compression", "kLZ4Compression"},
      {"compression_opts", "4:5:6:7"},
      {"num_levels", "8"},
      {"level0_file_num_compaction_trigger", "8"},
      {"level0_slowdown_writes_trigger", "9"},
      {"level0_stop_writes_trigger", "10"},
      {"target_file_size_base", "12"},
      {"target_file_size_multiplier", "13"},
      {"max_bytes_for_level_base", "14"},
      {"level_compaction_dynamic_level_bytes", "true"},
      {"max_bytes_for_level_multiplier", "15.0"},
      {"max_bytes_for_level_multiplier_additional", "16:17:18"},
      {"max_compaction_bytes", "21"},
      {"soft_rate_limit", "1.1"},
      {"hard_rate_limit", "2.1"},
      {"hard_pending_compaction_bytes_limit", "211"},
      {"arena_block_size", "22"},
      {"disable_auto_compactions", "true"},
      {"compaction_style", "kCompactionStyleLevel"},
      {"verify_checksums_in_compaction", "false"},
      {"compaction_options_fifo", "23"},
      {"max_sequential_skip_in_iterations", "24"},
      {"inplace_update_support", "true"},
      {"report_bg_io_stats", "true"},
      {"compaction_measure_io_stats", "false"},
      {"inplace_update_num_locks", "25"},
      {"memtable_prefix_bloom_size_ratio", "0.26"},
      {"memtable_huge_page_size", "28"},
      {"bloom_locality", "29"},
      {"max_successive_merges", "30"},
      {"min_partial_merge_operands", "31"},
      {"prefix_extractor", "fixed:31"},
      {"optimize_filters_for_hits", "true"},
  };

可用配置

  不論是在option string還是option map中，option name是目標類中的變量名，這些包括：DBOptions, ColumnFamilyOptions, BlockBasedTableOptions, or PlainTableOptions。DBOptions and ColumnFamilyOptions中的變量名和變量描述信息可以在options.h中找到，BlockBasedTableOptions, and PlainTableOptions中的變量信息可以在table.h中找到。
  需要注意的是，儘管絕大部分的配置項都可以在option string和option map中支持，仍然有一些例外。RocksDB支持的所有配置項可以在db_options_type_info, cf_options_type_info and block_based_table_type_info中查閱，源文件是util/options_helper.h。
  如果配置項是一個回調類，比如:comparators, compaction filter, and merge operators，需要將回調類的指針作爲option value傳入。當然也有一些例外情況可以支持通過option string和option map傳入回調類。

• Prefix extractor
  option value可以這樣傳入：rocksdb.FixedPrefix.<prefix_length> or rocksdb.CappedPrefix.<prefix_length>
• Filter policy
  option name：filter_poilcy
  option value：bloomfilter:<bits_per_key>:<use_block_based>
• table factory
  option name： table_factory
  option value： BlockBasedTable or PlainTable
  *Memtable Factory
  option name： memtable_factory
  option value ： skip_list/prefix_hash/hash_linkedlist/vector/cukoo

option file

  假設，我們打開了一個RocksDB實例，創建了一個列族，然後關閉實例，代碼如下

s = DB::Open(rocksdb_options, path_to_db, &db);
...
// Create column family, and rocksdb will persist the options.
ColumnFamilyHandle* cf;
s = db->CreateColumnFamily(ColumnFamilyOptions(), "new_cf", &cf);
...
// close DB
delete cf;
delete db;

  自從4.3版本以後，每個RocksDB實例都會自動存儲最新的配置信息到一個配置文件中，這個配置文件可以在下次再打開數據庫時使用。這和4.2版本或者更老版本差別還是很大的，在老版本，用戶需要記錄每個列族的配置信息，這樣下次纔可以成功打開DB實例。
  首先，調用調用LoadLatestOptions()來加載目標RocksDB的最新配置信息

DBOptions loaded_db_opt;
std::vector<ColumnFamilyDescriptor> loaded_cf_descs;
LoadLatestOptions(path_to_db, Env::Default(), &loaded_db_opt,
                  &loaded_cf_descs);

  由於c++沒有映射機制，以下用戶自定義的函數和類型指針必須在初始化時默認指定，更詳細的信息可以在rocksdb/utilities/options_util.h找到。

* env
* memtable_factory
* compaction_filter_factory
* prefix_extractor
* comparator
* merge_operator
* compaction_filter
* cache in BlockBasedTableOptions
* table_factory other than BlockBasedTableFactory

  這些不支持的用戶自定義函數，開發者需要手動指定。例如：下面這個例子我們初始化BlockBasedTableOptions and CompactionFilter中的cache配置。

for (size_t i = 0; i < loaded_cf_descs.size(); ++i) {
  auto* loaded_bbt_opt = reinterpret_cast<BlockBasedTableOptions*>(
      loaded_cf_descs[0].options.table_factory->GetOptions());
  loaded_bbt_opt->block_cache = cache;
}

loaded_cf_descs[0].options.compaction_filter = new MyCompactionFilter();

  接下來，我們可以做sanity check來保證可以正確安全地打開目標數據庫

Status s = CheckOptionsCompatibility(
    kDBPath, Env::Default(), db_options, loaded_cf_descs);

  如果check 返回OK，我們就可以繼續使用上面load的配置集合來打開數據庫了。

s = DB::Open(loaded_db_opt, kDBPath, loaded_cf_descs, &handles, &db);

  RocksDB配置文件是INI 文件格式，每個配置文件都有一個版本區塊，DBOption 區塊、CFOption區塊、TableOption區塊，每一個列族都有這幾類配置區塊。以下是一個完整的配置文件示例

[Version]
  rocksdb_version=4.3.0
  options_file_version=1.1
[DBOptions]
  stats_dump_period_sec=600
  max_manifest_file_size=18446744073709551615
  bytes_per_sync=8388608
  delayed_write_rate=2097152
  WAL_ttl_seconds=0
  ...
[CFOptions "default"]
  compaction_style=kCompactionStyleLevel
  compaction_filter=nullptr
  num_levels=6
  table_factory=BlockBasedTable
  comparator=leveldb.BytewiseComparator
  compression_per_level=kNoCompression:kNoCompression:kNoCompression:kSnappyCompression:kSnappyCompression:kSnappyCompression
  ...
[TableOptions/BlockBasedTable "default"]
  format_version=2
  whole_key_filtering=true
  skip_table_builder_flush=false
  no_block_cache=false
  checksum=kCRC32c
  filter_policy=rocksdb.BuiltinBloomFilter
  ....