rgw数据存储分析
metadata
ceph rgw的元数据分成四类：
[root@muhongtao-ceph-3 rgw]# radosgw-admin metadata list
[
"bucket",
"bucket.instance",
"otp", //暂忽略不分析
"user"
]
这里只分析三类：
- bucket
- bucket.instance
- user
这三类元数据都是存放在 meta pool 中的, 可以通过命令：
[root@mceph-3 ~]# rados -p meta.pool ls --all //查看meta.pool中存储的元数据有哪些
root bucket1
root .bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
users.keys 3F998GKDQHGGOI7060CN
root bucket2
users.uid 1009-user-01
root sensebucket
root .bucket.meta.sensebucket:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.1
root .bucket.meta.bucket2:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.2
users.uid 1009-user-01.buckets
[root@mceph-3 ~]#
#ns=root [1]
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta ls --namespace=root
bucket1
.bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
bucket2
.bucket.meta.bucket2:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.2
sensebucket
.bucket.meta.sensebucket:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.1
[root@ceph-3 ~]#
#ns=users.uid [2]
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta ls --namespace=users.uid
1009-user-01
1009-user-01.buckets
[root@ceph-3 ~]#
#ns=users.keys [3]
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta ls --namespace=users.keys
3F998GKDQHGGOI7060CN
针对元数据信息查询,都可以使用这一范式：
- radosgw-admin metadata get bucket:bucket_name
- radosgw-admin metadata get bucket.instance:bucket_name.instance
- radosgw-admin metadata get bucket:bucket_name
- [1] namespace=root: 上述在 namespace=root 下存储6哥 rados对象,分为三组，每组包含一个 rados bucket对象 和 一个 rados bucket.instance 对象。
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta ls --namespace=root
bucket1
.bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
bucket2
.bucket.meta.bucket2:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.2
sensebucket
.bucket.meta.sensebucket:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.1
我们以bucket1组为例，来分别看一下这两个 rados对象中都包含了什么信息：
<1>. 查看这两个rados对象的xattr属性
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta listxattr --namespace=root bucket1
ceph.objclass.version
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta listxattr --namespace=root .bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
ceph.objclass.version
user.rgw.acl
user.rgw.lc
针对rados对象：bucket1来讲, ceph.objclass.version是其存储在meta pool中的数据信息，该rados对象:bucket1仅仅包括一个扩展属性：xattr, 通过 getxattr获取该属性的值，显示为乱码：
[root@muhongtao-ceph-3 ~]# rados -p class_hdd_pool_1.meta getxattr --namespace=root bucket1 ceph.objclass.version
$_sFEdSejSKb39KBN0PS-IReM
//TODO: 怎么将该属性解码并输出看一下呢?
针对rados对象：.bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3来讲, ceph.objclass.version、user.rgw.acl、user.rgw.lc 是其存储在meta pool中的数据信息，该rados对象仅仅包括一个扩展属性：xattr, 通过getxattr将该属性的值保存下来并解码（解码需要知道对应的类型, 上边的ceph.objclass.version不知道对应类型）：
#lc
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta getxattr --namespace=root .bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3 user.rgw.lc > lc.txt
[root@ceph-3 ~]# ceph-dencoder import lc.txt type RGWLifecycleConfiguration decode dump_json
{
"prefix_map": {
"transclod-": {
"status": true,
"dm_expiration": false,
"expiration": 0,
"noncur_expiration": 0,
"mp_expiration": 0
}
},
"rule_map": [
{
"id": "transbucket_id_0987654321_1019", //这个是之前随意定的一个 id
"rule": {
"id": "transbucket_id_0987654321_1019",
"prefix": "transclod-", //lc前缀
"status": "Enabled",
"expiration": {
"days": "",
"date": ""
},
"noncur_expiration": {
"days": "",
"date": ""
},
"mp_expiration": {
"days": "",
"date": ""
},
"filter": {
"prefix": "",
"obj_tags": {}
},
"dm_expiration": false
}
}
]
}
#acl
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta getxattr --namespace=root .bucket.meta.bucket1:8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3 user.rgw.acl > acl.txt
[root@ceph-3 ~]#
[root@ceph-3 ~]# ceph-dencoder import acl.txt type RGWAccessControlPolicy decode dump_json
{
"acl": {
"acl_user_map": [
{
"user": "1009-user-01",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "1009-user-01",
"grant": {
"type": {
"type": 0
},
"id": "1009-user-01",
"email": "",
"permission": {
"flags": 15
},
"name": "1009-user-01",
"group": 0,
"url_spec": ""
}
}
]
},
"owner": {
"id": "1009-user-01",
"display_name": "1009-user-01"
}
}
- [2] namespace=users.uid: 上述在 namespace=users.uid 下存储着两个 rados 对象： 1009-user-01 和 1009-user-01.buckets 那么这两个rados对象在 class_hdd_pool_1.meta 中都存储了什么东西呢？
[root@eph-3 rgw]# rados -p class_hdd_pool_1.meta --namespace=users.uid listxattr 1009-user-01
ceph.objclass.version
[root@ceph-3 rgw]# rados -p class_hdd_pool_1.meta --namespace=users.uid listxattr 1009-user-01.buckets
[root@ceph-3 rgw]# //空
rados对象：1009-user-01, 该对象只有一个 xattr属性, 没有omap属性。该对象本身包括了用户的基本信息, 都在该属性 xattr中。可以通过 getxattr获取该属性的值，但是显示为乱码：
[root@ceph-3 rgw]# rados -p class_hdd_pool_1.meta --namespace=users.uid getxattr 1009-user-01 ceph.objclass.version
$_yQablR9CUuAbj6nUUMdtI7t
[root@ceph-3 rgw]#
//TODO 怎么将该属性解码输出看一下呢?
rados对象：1009-user-01.buckets, 该对象本身不包含信息, 只是个占位符, 该对象真正包含的信息存放在 omap属性中, 该对象没有 xattr扩展属性。 用户在对bucket拉列表（list bucket）时候会用到这个rados对象的信息。 rados对象1009-user-01.buckets的omap中存储的都是一个个的<key, value>, 首先列举一下该对象含有的key有哪些：
[root@ceph-3 rgw]# rados -p class_hdd_pool_1.meta --namespace=users.uid listomapkeys 1009-user-01.buckets
bucket1
bucket2
sensebucket
在列举一下该rados对象含有的values：
[root@ceph-3 rgw]# rados -p class_hdd_pool_1.meta --namespace=users.uid listomapvals 1009-user-01.buckets
bucket1
value (172 bytes) :
00000000 09 05 a6 00 00 00 00 00 00 00 d1 04 00 00 00 00 |................|
00000010 00 00 7f 53 8d 5f 01 00 00 00 00 00 00 00 07 03 |...S._..........|
00000020 77 00 00 00 07 00 00 00 62 75 63 6b 65 74 31 00 |w.......bucket1.|
00000030 00 00 00 2c 00 00 00 38 66 62 32 64 65 66 37 2d |...,...8fb2def7-|
00000040 37 63 63 64 2d 34 38 30 33 2d 61 37 36 61 2d 35 |7ccd-4803-a76a-5|
00000050 36 36 35 35 34 65 32 31 62 39 65 2e 39 35 39 36 |66554e21b9e.9596|
00000060 39 2e 33 2c 00 00 00 38 66 62 32 64 65 66 37 2d |9.3,...8fb2def7-|
00000070 37 63 63 64 2d 34 38 30 33 2d 61 37 36 61 2d 35 |7ccd-4803-a76a-5|
00000080 36 36 35 35 34 65 32 31 62 39 65 2e 39 35 39 36 |66554e21b9e.9596|
00000090 39 2e 33 00 00 00 00 00 00 00 00 00 10 00 00 00 |9.3.............|
000000a0 00 00 00 01 7f 53 8d 5f 95 34 f1 18 |.....S._.4..|
000000ac
bucket2
value (172 bytes) :
00000000 09 05 a6 00 00 00 00 00 00 00 36 07 00 00 00 00 |..........6.....|
00000010 00 00 e6 b8 87 5f 01 00 00 00 00 00 00 00 07 03 |....._..........|
00000020 77 00 00 00 07 00 00 00 62 75 63 6b 65 74 32 00 |w.......bucket2.|
00000030 00 00 00 2c 00 00 00 38 66 62 32 64 65 66 37 2d |...,...8fb2def7-|
00000040 37 63 63 64 2d 34 38 30 33 2d 61 37 36 61 2d 35 |7ccd-4803-a76a-5|
00000050 36 36 35 35 34 65 32 31 62 39 65 2e 39 35 39 36 |66554e21b9e.9596|
00000060 39 2e 32 2c 00 00 00 38 66 62 32 64 65 66 37 2d |9.2,...8fb2def7-|
00000070 37 63 63 64 2d 34 38 30 33 2d 61 37 36 61 2d 35 |7ccd-4803-a76a-5|
00000080 36 36 35 35 34 65 32 31 62 39 65 2e 39 35 39 36 |66554e21b9e.9596|
00000090 39 2e 32 00 00 00 00 00 00 00 00 00 10 00 00 00 |9.2.............|
000000a0 00 00 00 01 e6 b8 87 5f d4 42 01 21 |......._.B.!|
000000ac
sensebucket
value (176 bytes) :
00000000 09 05 aa 00 00 00 00 00 00 00 d7 04 a0 00 00 00 |................|
00000010 00 00 da 30 80 5f 04 00 00 00 00 00 00 00 07 03 |...0._..........|
00000020 7b 00 00 00 0b 00 00 00 73 65 6e 73 65 62 75 63 |{.......sensebuc|
00000030 6b 65 74 00 00 00 00 2c 00 00 00 38 66 62 32 64 |ket....,...8fb2d|
00000040 65 66 37 2d 37 63 63 64 2d 34 38 30 33 2d 61 37 |ef7-7ccd-4803-a7|
00000050 36 61 2d 35 36 36 35 35 34 65 32 31 62 39 65 2e |6a-566554e21b9e.|
00000060 39 35 39 36 39 2e 31 2c 00 00 00 38 66 62 32 64 |95969.1,...8fb2d|
00000070 65 66 37 2d 37 63 63 64 2d 34 38 30 33 2d 61 37 |ef7-7ccd-4803-a7|
00000080 36 61 2d 35 36 36 35 35 34 65 32 31 62 39 65 2e |6a-566554e21b9e.|
00000090 39 35 39 36 39 2e 31 00 00 00 00 00 00 00 00 00 |95969.1.........|
000000a0 20 a0 00 00 00 00 00 01 da 30 80 5f ef 00 bd 23 | ........0._...#|
000000b0
[root@ceph-3 rgw]#
我们来列举一个bucket的metadata, 注意上边listomapvals出的bucket1的size=172 bytes.
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta --namespace=users.uid getomapval 1009-user-01.buckets bucket1 bucket1.meta
Writing to bucket1.meta
[root@ceph-3 ~]# ls -la
total 8
drwxr-xr-x 2 root root 44 Oct 21 04:56 .
drwxr-xr-x 6 root root 95 Oct 21 02:44 ..
-rw-r--r-- 1 root root 172 Oct 21 04:56 bucket1.meta
[root@ceph-3 ~]#
可以看到 buceket1.meta 大小172 bytes. 来解析一下该文件：
[root@ceph-3 ceph_index]# ceph-dencoder import bucket1.meta type cls_user_bucket_entry decode dump_json
{
"bucket": {
"name": "bucket1",
"marker": "8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3",
"bucket_id": "8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3"
},
"size": 1233, //包含的文件的总大小
"size_rounded": 4096, // ???
"creation_time": "2020-10-19 08:51:11.418460Z", //bucket的创建时间
"count": 1, //文件个数
"user_stats_sync": "true"
}
该rados对象1009-user-01.buckets中的数据存储在 omap中, 格式为<key, value>, key为bucket_name， value为”类型为cls_user_bucket_entry的序列化过的数据”
[3]:
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta ls --namespace=users.keys
3F998GKDQHGGOI7060CN //这是该用户的access key
包含了key到uid的映射信息, 可以通过 access key找到uid
[root@ceph-3 ~]# rados -p class_hdd_pool_1.meta --namespace=users.keys get 3F998GKDQHGGOI7060CN key.txt
[root@ceph-3 ~]# cat key.txt
1009-user-01
index 索引数据
bucket index是一类特殊的 metadata, 通过bucket index, 可以列出该bucket下所有的 rgw对象。 bucket index保存在 index pool。我这里是： class_hdd_pool_1.index
bucket index信息本身保存在 index pool中：
[root@ceph-3 ~]# rados -p class_hdd_pool_1.index ls
.dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.1 // .dir. + bucket_id
.dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
.dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.2
index pool 中存放三个 rados对象( .dir. + bucket_id ), 分别是三个bucket的索引数据。 这三个rados对象是存放在 index pool 中的, 并且这三个rados对象的自身数据信息都存放在 omap中： <key, val>, key是bucket中的对象名(文件名), val是bucket中文件名对应的文件内容：即该文件的索引信息。
使用 listomapkeys即可将该bucket下所有的对象名列出来：
[root@muhongtao-ceph-3 ~]# rados -p class_hdd_pool_1.index listomapkeys .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.1
test.5M.gz.clod.1009
transclod-5mb.1009
transclod-test.1009
transclod-user.md.json.1009
[root@muhongtao-ceph-3 ~]#
[root@muhongtao-ceph-3 ~]# rados -p class_hdd_pool_1.index listomapkeys .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.2
uptransfer-zone1.bak.1015
[root@muhongtao-ceph-3 ~]# rados -p class_hdd_pool_1.index listomapkeys .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
transclod-usermd.json.1019
同理使用 listomapvals 即可将该bucket下所有文件对象的索引列出来： (该bucket只含有一个对象)
[root@ceph-3 ~]# rados -p class_hdd_pool_1.index listomapvals .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3
transclod-usermd.json.1019
value (239 bytes) :
00000000 08 03 e9 00 00 00 1a 00 00 00 74 72 61 6e 73 63 |..........transc|
00000010 6c 6f 64 2d 75 73 65 72 6d 64 2e 6a 73 6f 6e 2e |lod-usermd.json.|
00000020 31 30 31 39 02 00 00 00 00 00 00 00 01 07 03 78 |1019...........x|
00000030 00 00 00 01 d1 04 00 00 00 00 00 00 a8 53 8d 5f |.............S._|
00000040 17 61 15 1a 20 00 00 00 64 32 39 64 31 63 65 32 |.a.. ...d29d1ce2|
00000050 32 36 35 31 38 38 36 37 34 35 64 35 33 35 36 30 |2651886745d53560|
00000060 66 66 61 30 39 37 63 62 0c 00 00 00 31 30 30 39 |ffa097cb....1009|
00000070 2d 75 73 65 72 2d 30 31 0c 00 00 00 31 30 30 39 |-user-01....1009|
00000080 2d 75 73 65 72 2d 30 31 0a 00 00 00 74 65 78 74 |-user-01....text|
00000090 2f 70 6c 61 69 6e d1 04 00 00 00 00 00 00 00 00 |/plain..........|
000000a0 00 00 04 00 00 00 43 4c 4f 44 00 00 00 00 00 00 |......CLOD......|
000000b0 00 00 00 01 01 03 00 00 00 81 89 02 02 20 00 00 |............. ..|
000000c0 00 5f 53 61 33 77 6a 56 56 33 69 58 61 34 59 76 |._Sa3wjVV3iXa4Yv|
000000d0 76 69 67 42 5a 64 52 37 36 6f 41 44 73 61 30 5f |vigBZdR76oADsa0_|
000000e0 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |c..............|
我们也使用 getomapval 将其列出： (getomapval是操作单个对象的命令： getomapval index_rados_obj original_obj_name)
[root@muhongtao-ceph-3 ~]# rados -p class_hdd_pool_1.index getomapval .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3 transclod-usermd.json.1019
value (239 bytes) :
00000000 08 03 e9 00 00 00 1a 00 00 00 74 72 61 6e 73 63 |..........transc|
00000010 6c 6f 64 2d 75 73 65 72 6d 64 2e 6a 73 6f 6e 2e |lod-usermd.json.|
00000020 31 30 31 39 02 00 00 00 00 00 00 00 01 07 03 78 |1019...........x|
00000030 00 00 00 01 d1 04 00 00 00 00 00 00 a8 53 8d 5f |.............S._|
00000040 17 61 15 1a 20 00 00 00 64 32 39 64 31 63 65 32 |.a.. ...d29d1ce2|
00000050 32 36 35 31 38 38 36 37 34 35 64 35 33 35 36 30 |2651886745d53560|
00000060 66 66 61 30 39 37 63 62 0c 00 00 00 31 30 30 39 |ffa097cb....1009|
00000070 2d 75 73 65 72 2d 30 31 0c 00 00 00 31 30 30 39 |-user-01....1009|
00000080 2d 75 73 65 72 2d 30 31 0a 00 00 00 74 65 78 74 |-user-01....text|
00000090 2f 70 6c 61 69 6e d1 04 00 00 00 00 00 00 00 00 |/plain..........|
000000a0 00 00 04 00 00 00 43 4c 4f 44 00 00 00 00 00 00 |......CLOD......|
000000b0 00 00 00 01 01 03 00 00 00 81 89 02 02 20 00 00 |............. ..|
000000c0 00 5f 53 61 33 77 6a 56 56 33 69 58 61 34 59 76 |._Sa3wjVV3iXa4Yv|
000000d0 76 69 67 42 5a 64 52 37 36 6f 41 44 73 61 30 5f |vigBZdR76oADsa0_|
000000e0 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |c..............|
000000ef
我们将该索引信息进行解码：
[root@ceph-3 ~]# rados -p class_hdd_pool_1.index getomapval .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3 transclod-usermd.json.1019 transclod-usermd.json.1019
Writing to transclod-usermd.json.1019
[root@ceph-3 ~]# ls
transclod-usermd.json.1019
#文件的索引内容
[root@ceph-3 ~]# ceph-dencoder import transclod-usermd.json.1019 type rgw_bucket_dir_entry decode dump_json
{
"name": "transclod-usermd.json.1019",
"instance": "",
"ver": {
"pool": 137,
"epoch": 2
},
"locator": "",
"exists": "true",
"meta": { //文件的元数据
"category": 1,
"size": 1233,
"mtime": "2020-10-19 08:51:52.437608Z",
"etag": "d29d1ce22651886745d53560ffa097cb",
"storage_class": "CLOD",
"owner": "1009-user-01",
"owner_display_name": "1009-user-01",
"content_type": "text/plain",
"accounted_size": 1233,
"user_data": "",
"appendable": "false"
},
"tag": "_Sa3wjVV3iXa4YvvigBZdR76oADsa0_c",
"flags": 0,
"pending_map": [],
"versioned_epoch": 0
}
总结一下：
- bucket index信息存储在 index pool中。
- bucket index中记录的bucket相关信息(含有哪些key, 以及key对应的vals)存储在 omap中：
- omap<key, val>, key是bucket下存储的对象名, val是这些文件名对应的索引信息, val的结构是: rgw_bucket_dir_entry
- 每个bucket还有一个bucket header存放在omap 中，记录该bucekt的对象个数，总大小等
查看一下 omap header信息：
[root@ceph-3 ~]# rados getomapheader -p class_hdd_pool_1.index .dir.8fb2def7-7ccd-4803-a76a-566554e21b9e.95969.3 header.txt
Writing to header.txt
[root@ceph-3 ~]# ceph-dencoder type rgw_bucket_dir_header import header.txt decode dump_json
{
"ver": 3,
"master_ver": 0,
"stats": [
1,
{
"total_size": 1233, //该bucket目前的存储量大小
"total_size_rounded": 4096,
"num_entries": 1, //对象个数
"actual_size": 1233 //真实大小???
}
],
"new_instance": {
"reshard_status": "not-resharding",
"new_bucket_instance_id": "",
"num_shards": -1
}
}
object info
关于对象信息请看另一篇文章：object数据格式分析
