存儲系統的分類

之前收集了一些存儲產品,最近又重新整理了一下,對他們進行了簡單的分類。每個對存儲的分類可能不僅相同,我的分類完全按照自己的喜好來分,如和您的分類不同,僅供參考。只是做了蒐集和分類,少量產品加了寫介紹,希望以後有時間,加更多更詳細的介紹。

1.存儲引擎

1.1 Hash Table

1.1.1 dbm (database manager)

https://en.wikipedia.org/wiki/Dbm

The dbm library stores arbitrary data by use of a single key (a primary key) in fixed-size buckets and uses hashing techniques to enable fast retrieval of the data by key.

1.2 btree

1.2.1 berkerlydb

https://en.wikipedia.org/wiki/Berkeley_DB

http://baike.baidu.com/view/1281930.htm

Key/value數據模型

Berkeley DB最初開發的目的是以新的HASH訪問算法來代替舊的hsearch函數和大量的dbm實現(如AT&T的dbm,Berkeley的 ndbm,GNU項目的gdbm)

oracle

Written in C, java

BTREE, HASH, QUEUE, RECNO storage

https://www.oracle.com/database/berkeley-db/db.html

http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html

1.2.2 LMDB (Lightning Memory-Mapped Database)

LMDB is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc’s or memcpy’s occur during data fetches.

https://symas.com/products/lightning-memory-mapped-database/

https://github.com/LMDB/lmdb

http://www.lmdb.tech/doc/

1.2.3 BoltDB

Bolt is a pure Go key/value store inspired by Howard Chu’s LMDB project.

https://github.com/boltdb

https://github.com/boltdb/bolt

1.3 LSM

1.3.1 LevelDB

https://github.com/google/leveldb

C++, google

1.3.2 RocksDB

http://rocksdb.org/

https://github.com/facebook/rocksdb/

C++/java

1.3.3GoLevelDB

https://github.com/syndtr/goleveldb

1.3.4gorocksdb

https://github.com/tecbot/gorocksdb

Go wrapper for RocksDB

1.3.5 levigo

Go wrapper for LevelDB

https://github.com/jmhodges/levigo

1.3.6 mongo-rocks

RocksDB Storage Engine Module for MongoDB

https://github.com/mongodb-partners/mongo-rocks

C++

1.4 LSH

1.4.1 bitcask

https://github.com/basho/bitcask

採用bitcask模型的有:beandb, Riak

erlang

日誌結構的key/value存儲系統Bitcas

http://blog.chinaunix.net/uid-20196318-id-154750.html

Bitcask存儲模型

http://blog.csdn.net/qq910894904/article/details/37756377

1.5 FractalTree

1.5.1 PerconaFT

https://github.com/percona/PerconaFT

https://www.percona.com/doc/percona-server-for-mongodb/perconaft.html

https://github.com/percona/PerconaFT/wiki

Mysql存儲引擎之TokuDB以及它的數據結構Fractal tree(分形樹)

http://www.fxysw.com/thread-5061-1-1.html

https://en.wikipedia.org/wiki/Fractal_tree_index

TokuDB的索引結構–分形樹的實現

http://www.cnblogs.com/chaosheng/p/5250037.html

1.6 dbm系列

1.6.1 QDBM (Quick DataBase Manager)

http://sourceforge.net/projects/qdbm/

http://qdbm.sourceforge.net/

http://qdbm.sourceforge.net/benchmark.pdf

used by nmdb

1.6.2 ndbm (New Database Manager)

https://en.wikipedia.org/wiki/Ndbm

http://infolab.stanford.edu/~ullman/dbsi/win98/ndbm.html

ndbm (standing for New Database Manager) is a Berkeley produced version from 1986 of the AT&T dbm database.

ndbm stores arbitrary data by use of a single key in fixed-size buckets.

1.6.3 SDBM (Substitute Database Manager)

Substitute Database Manager

https://github.com/davidar/sdbm

http://www.cse.yorku.ca/~oz/sdbm.bun

1.6.4 GDBM (GNU Database Manager)

GNU Database Manager

http://www.gnu.org.ua/software/gdbm/

1.6.5 tdb (Trivial Database)

Trivial Database

http://sourceforge.net/projects/tdb/

http://tdb.sourceforge.net/

1.6.6 CDB

http://cr.yp.to/cdb.html

1.6.7 TinyCDB

Tiny Constant Database

http://www.corpit.ru/mjt/tinycdb.html

1.7 雙類型

1.7.1 Wiredtiger (btree, LSM)

C語言的

WiredTiger 本身也支持 LSM option (默認是 btree )

mangodb

https://github.com/wiredtiger/wiredtiger

http://www.wiredtiger.com/

http://source.wiredtiger.com/2.8.0/architecture.html#cache

1.7.2 Tokyo Cabinet and Kyoto Cabinet (B+tree,hash table)

http://baike.baidu.com/view/2640411.htm

http://fallabs.com/tokyocabinet/

https://en.wikipedia.org/wiki/Tokyo_Cabinet_and_Kyoto_Cabinet

Tokyo Cabinet and Kyoto Cabinet are two libraries of routines for managing key-value databases.

Kyoto Cabinet is the designated successor of Tokyo Cabinet

Tokyo Cabinet 是一個DBM的實現

Tokyo Cabinet features on-disk B+ trees and hash tables for key-value storage

Used by nmdb,Kyoto TreeDB

1.7.3 RaptorDB key value store (B+ 樹 或者 MurMur 哈希索引)

一個很小的、快速的嵌入式 NoSQL 存儲模塊,使用 B+ 樹 或者 MurMur 哈希索引

Implemented in .NET

http://raptordbkv.codeplex.com/

http://www.codeproject.com/Articles/190504/RaptorDB

http://www.codeproject.com/Articles/316816/RaptorDB-The-Key-Value-Store-V

1.8 SQL引擎類

1.8.1 InnoDB

http://www.oschina.net/p/innodb/

http://baike.baidu.com/view/1238935.htm

https://en.wikipedia.org/wiki/InnoDB

1.9 document類

1.9.1 RaptorDB document store

http://raptordb.codeplex.com/

http://www.codeproject.com/Articles/375413/RaptorDB-the-Document-Store

2 嵌入式

https://en.wikipedia.org/wiki/Embedded_database

2.1 SQLite

https://en.wikipedia.org/wiki/SQLite

2.2 UnQLite

盤點移動開發中最流行的5個數據庫

http://www.evget.com/article/2014/11/21/21843.html

3 單機存儲

3.1 單值KV存儲

3.1.1 Memcache

3.1.2 nmdb

https://blitiri.com.ar/p/nmdb/

use qdbm, berkeley db, tokyo cabinet or tdb as database backends

C語言

3.1.3 Memcachedb

C語言,新浪

http://memcachedb.org/

a distributed key-value storage system designed for persistent
It conforms to memcache protocol
uses Berkeley DB as a storing backend

write 18868 w/s
read 44444 r/s

using replication for master/slave
6 policy for replication:

http://memcachedb.org/memcachedb-guide-1.0.pdf

3.1.4 Kyoto Tycoon

http://fallabs.com/kyototycoon/

C/C++,FAL Labs

Kyoto Tycoon is a lightweight database server with auto expiration mechanism, which is useful to handle cache data and persistent data of various applications. Kyoto Tycoon is also a package of network interface to the DBM called Kyoto Cabinet.

3.1.5 ThruDB

建立在Apache Thrift framework下的簡單服務

支持多個數據存儲後端,包括BerkeleyDB、Disk、MySQL,還擁有Memcache和Spread集成

http://code.google.com/p/thrudb/

Thrudb is a set of simple services built on top of the Apache Thrift framework that provides indexing and document storage services

3.2 結構化KV存儲

3.2.1 Redis

3.2.2 ssdb

https://github.com/ideawu/ssdb

http://ssdb.io/zh_cn/

http://ssdb.io/docs/replication.html

http://ssdb.io/docs/config.html

https://github.com/ideawu/ssdb/blob/master/ssdb.conf

採用ssd,使用leveldb作爲存儲引擎,兼容redis接口

C/C++

3.2.3 ssdb-rocks

https://github.com/ideawu/ssdb-rocks

ssdb的另一個版本,採用ssd,使用rocksdb作爲存儲引擎,兼容redis接口

3.3.4 ardb

https://github.com/yinqiwen/ardb

redis-protocol compatible persistent nosql, it support multiple storage engines as backend like Google’s LevelDB, Facebook’s RocksDB, OpenLDAP’s LMDB, WiredTiger, the default backend is Facebook’s RocksDB.

C++

http://yinqiwen.github.io/ardb/2014/08/23/ardbintroduction/

3.2.5 (reborndb)QDB

兼容redis協議

Rocksdb and LevelDB

https://github.com/reborndb/qdb

3.2.6 Pika

Pika 的存儲引擎, 基於Rocksdb 修改. 封裝Hash, List, Set, Zset等數據結構

https://github.com/Qihoo360/pika

http://git.oschina.net/baotiao/pika

http://www.jianshu.com/p/d4f23120cbe4

首發丨360開源的類Redis存儲系統:Pika

https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653547160&idx=1&sn=befd195e2aa788775aaf1cc3b6f6fab3&scene=1&srcid=0512FqLKcLjVNH0zbKVlVBSO&key=b28b03434249256b1da1489d74564ea1a9d5b15207160026adc2a6ce0622b47c084d23b75e909dfe14f6173662cbdf5b&ascene=0&uin=MjMxMzM3NjIyMw%3D%3D&devicetype=iMac+MacBookPro12%2C1+OSX+OSX+10.11.4+build(15E65)&version=11020201&pass_ticket=WU1aAnx4aVkuID0Quq0HGuKNQB68CvjQzaTnnIhjJFZwLcPqGk1zilYX4uRvF9sd

首發丨360開源的類Redis存儲系統:Pika

https://media.weibo.cn/article?id=2309403974295628970629

3.2.7 LedisDB

A high performance NoSQL like Redis powered by Go

LevelDB, goleveldb, LMDB, RocksDB, BoltDB or Memory

http://ledisdb.com/

https://github.com/siddontang/ledisdb

3.3 文檔型

3.3.1 SisoDB

C#編寫的,專門提供給SQL Server面向文檔的db-provider

http://www.sisodb.com

3.4 SQL

3.4.1 MySQL

3.4.2 innostore

https://github.com/basho/innostore

Innostore is a simple Erlang API to Embedded InnoDB

4 單機存儲的proxy集羣方案

4.1 KV/Redis類

4.1.1 Twenproxy

http://www.oschina.net/p/twemproxy

https://github.com/twitter/twemproxy

靜態的分佈式Redis方案

4.1.2 Reborndb

https://github.com/reborndb

https://github.com/reborndb/reborn/blob/master/doc/tutorial_zh.md

4.1.3 Codis

http://www.oschina.net/p/codis

https://github.com/wandoulabs/codis

https://github.com/CodisLabs/codis

4.1.4 Netflix Dynomite

https://github.com/Netflix/dynomite

基於dynamo的思想

Dynomite: NetFlix對dynamo的開源通用實現

http://www.infoq.com/cn/news/2014/11/dynomite-netflix-dynamo

Netflix open sources Dynomite to make any datastore distributed

https://gigaom.com/2014/11/03/netflix-open-sources-dynomite-to-make-any-datastore-distributed/

A generic dynamo implementation for different k-v storage engines
inspired by Dynamo whitepaper

4.1.5 dbcached

http://code.google.com/p/dbcached/

a distributed key-value memory caching system for QDBM or Berkeley DB base on Memcached and NMDB

4.2 SQL類

4.2.1 Mycat

http://www.mycat.org.cn/

MyCat:開源分佈式數據庫中間件
http://mp.weixin.qq.com/s?__biz=MzAwNjMxNjQzNA==&mid=208187004&idx=1&sn=60aba39c148711e95f00ec7ca2e13bb1&scene=0#rd

4.2.2 MySQL Fabric

4.2.3 TDDL

4.2.4 Cobar

4.2.5 Atlas

4.2.6 Heisenberg

4.2.7 Vitess

5 KV存儲

5.1 riak

http://www.oschina.net/p/riak/

https://github.com/basho/riak

採用bitcask模型

Riak的實現是基於Amazon的Dynamo論文

erlang

Riak是以 Erlang 編寫的一個高度可擴展的分佈式數據存儲,Riak的實現是基於Amazon的Dynamo論文,Riak的設計目標之一就是高可用。

http://docs.basho.com/riak/kv/2.2.3/

5.2 beandb

https://github.com/douban/beansdb

https://github.com/douban/beanseye

採用bitcask模型

distributed key-value storage system

took the ideas from Amazon’s Dynamo

5.3 Project Voldemort

http://www.project-voldemort.com/voldemort/

Voldemort is a distributed key-value storage system

http://www.project-voldemort.com/voldemort/design.html

https://github.com/voldemort/voldemort

BDB-JE, MySQL, Read-Only

LinkedIn

5.4 Scalaris

http://scalaris.zib.de/

分佈式key value

erlang

supported the ACID properties for multi-key transactions

http://code.google.com/p/scalaris/

5.5 Aeospike

www.aerospike.com

http://www.aerospike.com/technologies/

http://www.aerospike.com/docs/

http://www.aerospike.com/docs/architecture/index.html

5.6 Tair

http://code.taobao.org/p/tair/

http://www.oschina.net/p/tair/

https://github.com/alibaba/tair

c/c++, 自研的mdb,fdb

5.7 dynomite

https://github.com/moonpolysoft/dynomite/wiki

http://www.oschina.net/p/dynomite/

6 文檔型存儲

6.1 MongoDB

6.2 CouchDB

CouchDB是文檔型存儲

http://couchdb.apache.org/

6.3 Membase / Couchbase

http://www.couchbase.com/

http://www.oschina.net/p/membase

http://www.oschina.net/p/couchbase-server

https://en.wikipedia.org/wiki/Couchbase_Server

http://www.couchbase.com/wiki/display/couchbase/Home

面向文檔的 NoSQL 數據庫管理系統

6.4 SequoiaDB

一個類mongodb的文檔型存儲
http://www.sequoiadb.com/cn/

6.5 RavenDB

http://ravendb.net/
a .net document database built on the Windows ESENT storage system

支持Linq,可以使用C#的Linq語法查詢數據

6.6 OrientDB

雖然是文檔型數據庫,但是它的關係管理方式卻和圖形數據庫相類似

http://www.orientechnologies.com/

http://orientdb.com/orientdb/

Distributed Graph Database with the flexibility of Documents

是兼具文擋數據庫的靈活性和圖形數據庫管理鏈接 能力的可深層次擴展的文檔-圖形數據庫管理系統。可選無模式、全模式或混合模式下。支持許 多高級特性,諸如ACID事務、快速索引,原生和SQL查詢功能。可以JSON格式導入、導出文檔。若不執行昂貴的JOIN操作的話,如同關係數據庫可在 幾毫秒內可檢索數以百記的鏈接文檔圖

Even if it is a document-based database, the relationships are managed as in graph databases with direct connections between records.

It supports schema-less, schema-full and schema-mixed modes.

6.7 RethinkDB

http://www.rethinkdb.com/

https://en.wikipedia.org/wiki/RethinkDB

http://baike.baidu.com/link?url=IaJEn3OOWg3i_q4HSIyqBfxRUATc2bpsoNfzcHdyAVa7vbs02R6PsIIyR4F2V4zZy9Wq3FJYW550B-d6IF6MV_

RethinkDB 1.14 (Brazil) 發佈,分佈式數據庫
http://www.oschina.net/news/54771/rethinkdb-1-14-brazil

7 列式存儲

7.1 HBase

7.2 Cassandra

7.3 Accumulo

https://accumulo.apache.org/
Apache Accumulo is based on Google’s BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift.

7.4 Hypertable

http://hypertable.org/
an open source database system inspired by publications on the design of Google’s BigTable.
Hypertable runs on top of a distributed file system such as the Apache HDFS, GlusterFS or the Kosmos File System (KFS).

7.5 Scylla

http://www.scylladb.com/
https://github.com/scylladb/scylla

8 NewSQL

8.1 Actordb

http://m.oschina.net/p/actordb

http://www.actordb.com/

https://github.com/biokoda/actordb

8.2 Cockroachdb

Go語言

https://www.cockroachlabs.com/

https://github.com/cockroachdb/cockroach

https://groups.google.com/forum/#!forum/cockroachdb-users

https://groups.google.com/forum/#!forum/cockroach-db

https://www.cockroachlabs.com/docs/

Design Documents
https://github.com/cockroachdb/cockroach/blob/master/docs/design.md

8.3 FoundationDB

https://github.com/FoundationDB

https://foundationdb.com/

8.4 Oceanbase

https://github.com/alibaba/oceanbase

OceanBase架構介紹
http://wenku.baidu.com/link?url=cknPHaARI1_Z6tHGeFfF4Vn_hDjAsfHElNu8yOX_jX2qX4eIKaf4UrY2e02TcCf5ib7GOUf437IulU5lUPrixrrgMFOnEJUMUIkBDP__jA7

8.5 SnappyData

8.6 TiDB

https://github.com/pingcap/tidb
Go 語言

https://github.com/pingcap/tikv
Rust語言

https://github.com/pingcap/tidb/blob/master/docs/QUICKSTART.md

從零開始寫分佈式數據庫

https://github.com/ngaut/builddatabase

9 圖數據庫

9.1 Neo4j

9.2 Infinite Graph

http://www.objectivity.com/products/infinitegraph/

an enterprise distributed graph database

10 File存儲

10.1 Ceph

http://ceph.com/

https://github.com/ceph

Ceph:一個 Linux PB 級分佈式文件系統

http://www.ibm.com/developerworks/cn/linux/l-ceph/

10.2 FastDFS

http://code.google.com/p/fastdfs/

10.3 HDFS

10.4 MogileFs

https://github.com/mogilefs/

10.5 MooseFS

http://www.moosefs.org/

a fault tolerant, network distributed file system

10.6 TFS

http://tfs.taobao.org/

10.7 GlusterFS

http://www.gluster.org/

GlusterFS:異地備份(Geo-replication)源碼分析

http://blog.chinaunix.net/uid-22166872-id-4392777.html

Gluster Geo-replication工作原理[轉載]

http://blog.163.com/szy8706@yeah/blog/static/62713185201363163131800/

http://disclu.blogspot.com/2012/11/gluster-geo-replication.html

10.8 kosmosfs

http://code.google.com/p/kosmosfs/

11 In-Memory 存儲

11.1 Redis cluster

11.2 Mysql cluster

11.3 Gemfire/Gemde

11.4 VoltDB

http://voltdb.com/

內存數據庫

12 私有存儲

12.1 Amazon

12.1.1 Amazon Dynamo

Key-value

http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Amazon’s Dynamo paper

Merkle trees

Gossiping of membership

Gossiped synchronization of partitions

12.2 Google

12.2.1 BigTable

Bigtable: A Distributed Storage System for Structured Data

http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf

12.2.2 F1

F1: A Distributed SQL Database That Scales

http://research.google.com/pubs/pub41344.html

http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf

12.2.3 Spanner

http://research.google.com/archive/spanner.html

Spanner: Google’s Globally-Distributed Database

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/zh-CN//archive/spanner-osdi2012.pdf

Exclusive: Inside Google Spanner, the Largest Single Database on Earth

http://www.wired.com/2012/11/google-spanner-time/all/

解析全球級分佈式數據庫Google Spanner

http://www.csdn.net/article/2012-09-19/2810132-google-spanner-next-database-datacenter

12.2.4 Megastore

Megastore: Providing Scalable, Highly Available Storage for Interactive Services

https://research.google.com/pubs/pub36971.html

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf

12.3 Baidu

12.3.1 Baidu Mola

Key-value

12.3.2 Baidu BDRP

12.3.3 Baidu DDBS

12.4 騰訊

單機MySQL到NoSQL集羣 騰訊存儲進階路

http://tech.it168.com/a2017/0322/3105/000003105672.shtml

騰訊十多個人管理一萬多臺NoSQL存儲服務器的祕密

http://m.techweb.com.cn/article/2016-01-06/2253310.shtml

12.4.1 騰訊CKV

騰訊CKV海量分佈式存儲系統

http://www.csdn.net/article/2014-03-11/2818723

12.4.2 QuorumKV

微信PaxosStore內存雲揭祕:十億Paxos/分鐘的挑戰

http://mp.weixin.qq.com/s?__biz=MzI4NDMyNTU2Mw%3D%3D&mid=2247483804&idx=1&sn=a6629ebdaefbc2470c2ecbf12577daff

12.6 京東

12.6.1 京東JIMDB

解讀JIMDB 京東分佈式緩存與高速KV存儲

http://m.chinabyte.com/storage/463/13344963_mi.shtml

解讀JIMDB 京東分佈式緩存與高速KV存儲

http://m.it168.com/article_1720792.html

12.7 滴滴

12.7.1 滴滴Rockstable

滴滴高性能KV存儲系統實踐

http://m.it168.com/article_3091989.html

12.8 美團

12.8.1 Cellar

基於tair研發的新一代KV存儲服務

美團cellar講座筆記

http://blog.leanote.com/post/yuannight/cellar%E8%AE%B2%E5%BA%A7%E7%AC%94%E8%AE%B0

12.9 360

12.9.1 360 Bada

Key-value

360自研分佈式存儲系統Bada的架構設計和應用

http://www.chinacloud.cn/wap.aspx?cid=16&nid=21103

1.9.2 HustStore

https://github.com/Qihoo360/huststore

HustStore 360高性能分佈式存儲服務

http://www.oschina.net/p/huststore?fromerr=ug3CEPNP

13 雲產品

13.1 Amazon DynamoDB

13.2 AWS Aurora

https://aws.amazon.com/cn/rds/aurora/

2.3 Google雲

2.4 阿里雲

2.5 美團雲

Mangix:分佈式對象存儲

Mangix: 美團雲分佈式對象存儲系統

http://docs.huihoo.com/infoq/qconbeijing/2016/day2/%E4%BA%91%E5%B9%B3%E5%8F%B0%E6%9E%B6%E6%9E%84%E4%B8%93%E9%A2%98/2-6-%E7%BE%8E%E5%9B%A2%E4%B8%87%E4%BA%BF%E7%BA%A7%E5%AF%B9%E8%B1%A1%E5%AD%98%E5%82%A8%E7%B3%BB%E7%BB%9F-%E6%9D%8E%E5%87%AF.pdf

發佈了32 篇原創文章 · 獲贊 13 · 訪問量 6萬+
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章