RGW的實現基於Rados的pool、對象、xattrs和omap。RGW將數據分爲3種：metadata，
bucket index和數據。
- metadata
包括user，bucket和bucket.instance。其中user是用戶信息。bucket是bucket名字和
bucket instance的映射。bucket.instance中是每個bucket instance的信息。
- bucket index
bucket index在rados對象的omap中保存了k/v映射。其中key是用戶put的object的名字，
value保存了該對象的一些基本元數據信息。每個omap中都有一個header，其中存放了bucket
的統計信息(比如，存放的對象個數，佔用的空間大小等)。//???
- data
用戶put的對象的數據內容存放在一個或者多個rados對象中。
1. rgw bucket怎麼表示?
rgw bucket 和 Amazon S3 一致。和Swift的 container 對應。1.1 bucket 的元數據信息
----------------------- [root@yhg-2 centos]# radosgw-admin metadata get bucket:xiamenbucket --cluster yhgtest
{
"key": "bucket:xiamenbucket",
"ver": {
"tag": "_lGZYdGzrVGYu2PYmKUcqs6S",
"ver": 1
},
"mtime": 1457681931,
"data": {
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"owner": "osgw-william1",
"creation_time": 1457681931,
"linked": "true",
"has_bucket_info": "false"
}
} .. Note:: bucket_id/marker的組成：
zone_name.$instance_id.$bucket_id'。
詳細內容見rgw/rgw_rados.cc:644
1.2 bucket的表示
--------------------------- bucket index 對象
作用：
在該對象的omap中保存了屬於本bucket的所有key的列表。
名稱：
.dir.yhgtest-yhgtest.4463.1 (.dir.$bucket_id)內容：
有兩個key，kehekey和xiamenkey。1.1.1/2.2.3 屬於hehekey，3.3.2/4.4.3屬於xiamenkey。
每次put新數據都會創建兩個新的�0_0000000000AA.AA.2/�0_0000000000BB.BB.3，其中
BB=AA+1。
[root@yhg-2 cmds]# rados -p .rgw.buckets.index listomapkeys .dir.yhgtest-yhgtest.4463.1 --cluster yhgtest
hehekey
xiamenkey
�0_00000000001.1.2
�0_00000000002.2.3
�0_00000000003.3.2
�0_00000000004.4.3
�0_00000000005.5.2
�0_00000000006.6.3
�0_00000000007.7.2
�0_00000000008.8.3
[root@yhg-2 cmds]# rados -p .rgw.buckets.index listomapvals .dir.yhgtest-yhgtest.4463.1 --cluster yhgtest
hehekey
value: (221 bytes) :
0000 : 08 03 d7 00 00 00 07 00 00 00 68 65 68 65 6b 65 : ..........heheke
0010 : 79 03 00 00 00 00 00 00 00 01 04 03 83 00 00 00 : y...............
0020 : 01 00 00 50 00 00 00 00 00 ef ce e7 56 00 00 00 : ...P........V...
0030 : 00 20 00 00 00 62 61 37 38 64 32 36 37 62 62 33 : . ...ba78d267bb3
0040 : 34 39 63 62 34 66 65 38 32 32 61 38 66 36 62 36 : 49cb4fe822a8f6b6
0050 : 64 61 64 39 37 0d 00 00 00 6f 73 67 77 2d 77 69 : dad97....osgw-wi
0060 : 6c 6c 69 61 6d 31 15 00 00 00 5a 6f 6e 65 20 75 : lliam1....Zone u
0070 : 73 65 72 20 66 6f 72 20 79 68 67 74 65 73 74 18 : ser for yhgtest.
0080 : 00 00 00 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f : ...application/o
0090 : 63 74 65 74 2d 73 74 72 65 61 6d 00 00 50 00 00 : ctet-stream..P..
00a0 : 00 00 00 00 00 00 00 00 00 00 00 01 01 02 00 00 : ................
00b0 : 00 01 03 08 17 00 00 00 79 68 67 74 65 73 74 2d : ........yhgtest-
00c0 : 79 68 67 74 65 73 74 2e 34 36 33 33 2e 32 35 00 : yhgtest.4633.25.
00d0 : 00 00 00 00 00 00 00 00 00 00 00 00 00 : .............
...
[root@yhg-2 cmds]# radosgw-admin bucket stats xiamenbucket --cluster yhgtest
...
"xiamenbucket",
{
"bucket": "xiamenbucket",
"pool": ".hdd_pool",
"index_pool": ".rgw.buckets.index",
"id": "yhgtest-yhgtest.4463.1",
"marker": "yhgtest-yhgtest.4463.1",
"owner": "osgw-william1",
"ver": "0#9",
"master_ver": "0#0",
"mtime": "2016-03-11 07:38:51.000000",
"max_marker": "0#00000000008.8.3", // 最新的更新
"usage": {
"rgw.main": {
"size_kb": 10240,
"size_kb_actual": 10240,
"num_objects": 2
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},另外，還可以直接和對象的osd守護進程交互，獲取.dir_XXX對象上的omap的信息。
# ceph daemon --cluster yhg /var/run/ceph/yhg-osd.1.asok getomap $POOL $OBJ所在pool:
bucket.index_pool (.rgw.buckets.index)- bucket meta 對象
作用：
其內容爲bucket的描述信息，在其xattr中還有user.rgw.acl信息。
名稱：
.bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 (.bucket.meta.$bucket_name:$bucket_id)內容：
[root@yhg-2 cmds]# rados -p .rgw get .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 /tmp/m --cluster yhgtest
[root@yhg-2 cmds]# ceph-dencoder type RGWBucketInfo import /tmp/m decode dump_json
{
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"creation_time": 1457681931,
"owner": "osgw-william1",
"flags": 0,
"region": "yhgtest",
"placement_rule": "default-placement",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"num_shards": 0,
"bi_shard_hash_type": 0
}
[root@yhg-2 cmds]# rados -p .rgw listxattr .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 --cluster yhgtest
ceph.objclass.version
user.rgw.acl
[root@yhg-2 cmds]# rados -p .rgw getxattr .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 user.rgw.acl --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWAccessControlPolicy import /tmp/m decode dump_json
{
"acl": {
"acl_user_map": [
{
"user": "osgw-william1",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "osgw-william1",
"grant": {
"type": {
"type": 0
},
"id": "osgw-william1",
"email": "",
"permission": {
"flags": 15
},
"name": "Zone user for yhgtest",
"group": 0
}
}
]
},
"owner": {
"id": "osgw-william1",
"display_name": "Zone user for yhgtest"
}
}所在pool：
zone.domain_root (.rgw)實現函數：
// rgw/rgw_rados.cc
7463 int ret = put_bucket_instance_info(info, exclusive, mtime, pattrs)- bucket 對象
作用：
???
名稱：
xiamenbucket (用戶指定的bucket名字)內容：
和bucket meta對象內容類似。
[root@yhg-2 cmds]# rados -p .rgw listxattr xiamenbucket --cluster yhgtest
ceph.objclass.version所在pool：
zone.domain_root (.rgw）2. rgw object怎麼表示?
對應至少一個 Rados 對象。每個 rgw object 分爲兩部分，一個對象邏輯頭(object logical head/olh)和一個
可選 tail。olh 最大 512KB，tail 按照固定大小切片。切片的大小可以配置，默認4MB。
olh 的對象內容爲上傳文件的前512KB。如果文件小於該值，則沒有tail。
數據會分爲header和4M的其他片段。
- header
[root@yhg-2 cmds]# rados -p .hdd_pool listxattr yhgtest-yhgtest.4463.1_hehekey --cluster yhgtest
user.rgw.acl
user.rgw.content_type
user.rgw.etag
user.rgw.idtag
user.rgw.manifest.. Note:: s3對象的 用戶自定義的metadata也存放在這裏
查看header對象所在的pg:
[root@yhg-2 cmds]# ceph osd map .hdd_pool yhgtest-yhgtest.4463.1_hehekey --cluster yhgtest
osdmap e72 pool '.hdd_pool' (1) object 'yhgtest-yhgtest.4463.1_hehekey' -> pg 1.beb53f29 (1.29) -> up ([0], p0) acting ([0], p0)- manifest
rgw object的 manifest 指示了tail部分的分片信息，用於讀取對象時查找分片。
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.manifest --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWObjManifest import /tmp/m decode dump_json
{
"objs": [],
"obj_size": 5242880,
"explicit_objs": "false",
"head_obj": {
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"key": "",
"ns": "",
"object": "hehekey",
"instance": ""
},
"head_size": 524288,
"max_head_size": 524288,
"prefix": ".eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_",
"tail_bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"rules": [
{
"key": 0,
"val": {
"start_part_num": 0,
"start_ofs": 524288,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
]
}manifest的prefix字段指示了rgw對象的當前版本的分片的名稱信息。
[root@yhg-2 cmds]# rados -p .hdd_pool ls --cluster yhgtest | grep eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4
yhgtest-yhgtest.4463.1__shadow_.eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_2
yhgtest-yhgtest.4463.1__shadow_.eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_1- idtag
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.idtag --cluster yhgtest
yhgtest-yhgtest.4633.25- etag
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.etag --cluster yhgtest
ba78d267bb349cb4fe822a8f6b6dad97- content type
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.content_type --cluster yhgtest
application/octet-stream- acl
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.acl --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWAccessControlPolicy import /tmp/m decode dump_json
{
"acl": {
"acl_user_map": [
{
"user": "osgw-william1",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "osgw-william1",
"grant": {
"type": {
"type": 0
},
"id": "osgw-william1",
"email": "",
"permission": {
"flags": 15
},
"name": "Zone user for yhgtest",
"group": 0
}
}
]
},
"owner": {
"id": "osgw-william1",
"display_name": "Zone user for yhgtest"
}
}3. 怎樣列舉某個用戶的buckets
在zone.user_uid_pool (pool '.users.uid') 中，爲每個用戶創建了一個$user_name.buckets對象(比如，osgw-william1.buckets)。在該對象的omap中存放了該用戶的buckets。在更新
bucket內容，更新獲取bucket的統計信息時會訪問這些信息。
內容：
[root@yhg-2 cmds]# rados -p .users.uid listomapkeys osgw-william1.buckets --cluster yhgtest
xiamenbucket
xiamenbucket2
xiamenbucket3
xiamenbucket4
::
[root@yhg-2 cmds]# rados -p .users.uid listomapvals osgw-william1.buckets --cluster yhgtest
xiamenbucket
value: (170 bytes) :
0000 : 06 05 a4 00 00 00 00 00 00 00 00 00 a0 00 00 00 : ................
0010 : 00 00 0b 76 e2 56 02 00 00 00 00 00 00 00 07 03 : ...v.V..........
0020 : 7d 00 00 00 0c 00 00 00 78 69 61 6d 65 6e 62 75 : }.......xiamenbu
0030 : 63 6b 65 74 09 00 00 00 2e 68 64 64 5f 70 6f 6f : cket.....hdd_poo
0040 : 6c 16 00 00 00 79 68 67 74 65 73 74 2d 79 68 67 : l....yhgtest-yhg
0050 : 74 65 73 74 2e 34 34 36 33 2e 31 16 00 00 00 79 : test.4463.1....y
0060 : 68 67 74 65 73 74 2d 79 68 67 74 65 73 74 2e 34 : hgtest-yhgtest.4
0070 : 34 36 33 2e 31 12 00 00 00 2e 72 67 77 2e 62 75 : 463.1.....rgw.bu
0080 : 63 6b 65 74 73 2e 69 6e 64 65 78 12 00 00 00 2e : ckets.index.....
0090 : 72 67 77 2e 62 75 63 6b 65 74 73 2e 65 78 74 72 : rgw.buckets.extr
00a0 : 61 00 00 a0 00 00 00 00 00 01 : a.........
xiamenbucket2
value: (171 bytes) :
... ---------------------------
其他
---------------------------[1] zone map
[root@yhg-2 cmds]# radosgw-admin zone get --cluster yhgtest
{
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": {
"access_key": "66D5C0ZBSXYX6HGHH2D8",
"secret_key": "BgDVnjKmeM5bMAXQXGPg1CyxzIQKzmlCCpKrfm4Q"
},
"placement_pools": [
{
"key": ".hdd_pool",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra"
}
},
{
"key": "default-placement",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra"
}
}
]
}