转载

使用RADOSGW联盟网关跨域同步

Ceph 是一个高性能的PB级分布式文件系统。它能够在一个系统中提供对象存储、块存储和文件存储。本文的主要内容是如何 通过RADOSGW备份一个可用区 。

  • 对如何加载使用块存储和文件存储感兴趣可以参考 《用容器轻松搭建ceph实验环境》
  • 对RADOSGW如何暴露S3和Swift接口感兴趣可以参考 《通过RADOSGW提供ceph的S3和Swift接口》

准备工作

我们需要先安装 virtualBox 和 vagrant 。通过vagrant来驱动virtualBox搭建一个虚拟测试环境。首先在本地任意路径新建一个空文件夹比如 test ，运行以下命令：

virtual box host 
mkdir test
cd test
vagrant init minimum/ubuntu-trusty64-docker
vi Vagrantfile

里面应该有一句 config.vm.box = "minimum/ubuntu-trusty64-docker" ，在它的下面添加如下几行代码，相当于给它分配三台虚拟机，一台叫做 us-east ，它的IP是 192.168.33.15 ；另一台叫做 us-west ，它的IP是 192.168.33.16 ；第三台叫做 eu-east ，它的IP是 192.168.33.17 。

Vagrantfile 
config.vm.define "us-east" do | host |
  host.vm.hostname = "us-east"
  host.vm.network "private_network", ip: "192.168.33.15"
end

config.vm.define "us-west" do | host |
  host.vm.hostname = "us-west"
  host.vm.network "private_network", ip: "192.168.33.16"
end

config.vm.define "eu-east" do | host |
  host.vm.hostname = "eu-east"
  host.vm.network "private_network", ip: "192.168.33.17"
end

然后分别在三个终端运行以下命令启动并连接三台虚拟机。

virtual box host terminal 1 
vagrant up
vagrant ssh us-east
virtual box host terminal 2 
vagrant ssh us-west
virtual box host terminal 3 
vagrant ssh eu-east

背景介绍

Ceph对象网关（Ceph Object Gateway，radosgw） 提供了S3和Swift的API，同时也支持S3的一些概念。 辖区（region） 表明了一个地理位置，比如us。在这个辖区里可以有多个 域（zone） ，比如east和west。一个域里可以有多个实例，一个实例可以有多个 节点（node） 。同时，配置一个域需要一系列的 存储池（pool） 。如下图：

使用RADOSGW联盟网关跨域同步

对于这次练习，我们使用下图的架构：

使用RADOSGW联盟网关跨域同步

一个us辖区里有us-east和us-west两个域，每个域里各有一个实例，分别为us-east-1和us-west-1。还有一个eu的辖区，里面有一个eu-east的域，一个为eu-east-1的实例。我们将会首先实现同一个辖区（us）里的同步，然后是不同辖区的同步。相同辖区可以同步元数据和数据对象，而不同的辖区只能同步元数据而不能同步数据对象。元数据包括网关用户和存储桶（bucket）。

相同辖区的同步

首先需要安装一些ceph、radosgw的依赖包：

us-east us-west eu-east 
sudo apt-get update
sudo apt-get -y --force-yes install ceph-common radosgw radosgw-agent

为了提供HTTP服务，还需要所有虚拟机都安装apache2和FastCGI：

us-east us-west eu-east 
sudo apt-get -y --force-yes install apache2 libapache2-mod-fastcgi

然后就可以分别启动ceph/demo这个容器来轻松提供ceph服务了：

us-east 
docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=192.168.33.15 -e CEPH_NETWORK=192.168.33.15/24 --name=ceph ceph/demo
us-west 
docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=192.168.33.16 -e CEPH_NETWORK=192.168.33.16/24 --name=ceph ceph/demo

然后手动在各自虚拟机上创建一些提供给域使用的存储池，这一步不是必须的，因为我们创建的网关用户是有权限自动创建存储池的：

us-east 
docker exec -it ceph bash
ceph osd pool create .us-east.rgw.root 16 16
ceph osd pool create .us-east.rgw.control 16 16
ceph osd pool create .us-east.rgw.gc 16 16
ceph osd pool create .us-east.rgw.buckets 512 512
ceph osd pool create .us-east.rgw.buckets.index 32 32
ceph osd pool create .us-east.rgw.buckets.extra 16 16
ceph osd pool create .us-east.log 16 16
ceph osd pool create .us-east.intent-log 16 16
ceph osd pool create .us-east.usage 16 16
ceph osd pool create .us-east.users 16 16
ceph osd pool create .us-east.users.email 16 16
ceph osd pool create .us-east.users.swift 16 16
ceph osd pool create .us-east.users.uid 16 16
exit
us-west 
docker exec -it ceph bash
ceph osd pool create .us-west.rgw.root 16 16
ceph osd pool create .us-west.rgw.control 16 16
ceph osd pool create .us-west.rgw.gc 16 16
ceph osd pool create .us-west.rgw.buckets 512 512
ceph osd pool create .us-west.rgw.buckets.index 32 32
ceph osd pool create .us-west.rgw.buckets.extra 16 16
ceph osd pool create .us-west.log 16 16
ceph osd pool create .us-west.intent-log 16 16
ceph osd pool create .us-west.usage 16 16
ceph osd pool create .us-west.users 16 16
ceph osd pool create .us-west.users.email 16 16
ceph osd pool create .us-west.users.swift 16 16
ceph osd pool create .us-west.users.uid 16 16
exit

两台虚拟机对应两个实例，接下来为这两个实例分别创建密钥环（keyring），用它生成网关的用户和密钥（key），增加密钥的rwx权限并让其有权限访问Ceph对象集群：

us-east 
sudo ceph auth del client.radosgw.gateway
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-east-1 --gen-key
sudo ceph-authtool -n client.radosgw.us-east-1 --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
us-west 
sudo ceph auth del client.radosgw.gateway
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-west-1 --gen-key
sudo ceph-authtool -n client.radosgw.us-west-1 --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring

接下来创建一个apache2的配置文件，监听80端口并把请求转发到radosgw提供的FastCGI 9000端口（稍后将会配置）上：

us-east us-west 
cat << EOF > rgw.conf
FastCgiExternalServer /var/www/s3gw.fcgi -host localhost:9000

<VirtualHost *:80>
    ServerName localhost
    ServerAlias *.localhost
    ServerAdmin qinghua@ggg.com
    DocumentRoot /var/www
    RewriteEngine On
    RewriteRule  ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

    <IfModule mod_fastcgi.c>
           <Directory /var/www>
            Options +ExecCGI
            AllowOverride All
            SetHandler fastcgi-script
            Order allow,deny
            Allow from all
            AuthBasicAuthoritative Off
        </Directory>
    </IfModule>

    AllowEncodedSlashes On
    ErrorLog /var/log/apache2/error.log
    CustomLog /var/log/apache2/access.log combined
    ServerSignature Off
</VirtualHost>
EOF

sudo mv rgw.conf /etc/apache2/conf-enabled/rgw.conf

由于上述配置需要用到apache2默认未加载的 rewrite模块 ，所以需要加载并重新启动apache2：

us-east us-west 
sudo a2enmod rewrite
sudo service apache2 restart

FastCGI需要一个脚本来启用兼容S3的接口，同样也是所有虚拟机都要配，但是实例名略有区别：

us-east 
cat << EOF > s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.us-east-1
EOF

sudo mv s3gw.fcgi /var/www/s3gw.fcgi
sudo chmod +x /var/www/s3gw.fcgi
us-west 
cat << EOF > s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.us-west-1
EOF

sudo mv s3gw.fcgi /var/www/s3gw.fcgi
sudo chmod +x /var/www/s3gw.fcgi

现在到了在 ceph.conf 配置实例的时候了：

us-east 
sudo sed -i '$a rgw region root pool = .us.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zonegroup root pool = .us.rgw.root' /etc/ceph/ceph.conf
sudo sed -i $'$a ///n' /etc/ceph/ceph.conf
sudo sed -i '$a [client.radosgw.us-east-1]' /etc/ceph/ceph.conf
sudo sed -i '$a rgw region = us' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone = us-east' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone root pool = .us-east.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw dns name = us-east' /etc/ceph/ceph.conf
sudo sed -i '$a host = us-east' /etc/ceph/ceph.conf
sudo sed -i '$a keyring = /etc/ceph/ceph.client.radosgw.keyring' /etc/ceph/ceph.conf
sudo sed -i '$a rgw socket path = ""' /etc/ceph/ceph.conf
sudo sed -i '$a log file = /var/log/radosgw/client.radosgw.us-east-1.log' /etc/ceph/ceph.conf
sudo sed -i '$a rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0' /etc/ceph/ceph.conf
sudo sed -i '$a rgw print continue = false' /etc/ceph/ceph.conf
sudo sed -i $'$a ///n' /etc/ceph/ceph.conf
us-west 
sudo sed -i '$a rgw region root pool = .us.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zonegroup root pool = .us.rgw.root' /etc/ceph/ceph.conf
sudo sed -i $'$a ///n' /etc/ceph/ceph.conf
sudo sed -i '$a [client.radosgw.us-west-1]' /etc/ceph/ceph.conf
sudo sed -i '$a rgw region = us' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone = us-west' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone root pool = .us-west.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw dns name = us-west' /etc/ceph/ceph.conf
sudo sed -i '$a host = us-west' /etc/ceph/ceph.conf
sudo sed -i '$a keyring = /etc/ceph/ceph.client.radosgw.keyring' /etc/ceph/ceph.conf
sudo sed -i '$a rgw socket path = ""' /etc/ceph/ceph.conf
sudo sed -i '$a log file = /var/log/radosgw/client.radosgw.us-west-1.log' /etc/ceph/ceph.conf
sudo sed -i '$a rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0' /etc/ceph/ceph.conf
sudo sed -i '$a rgw print continue = false' /etc/ceph/ceph.conf

配置参数及其作用可以在 这里 查到，下面列出了一部分与辖区和域有关的参数：

  • rgw region root pool ：v.67版本中指定辖区所使用的存储池
  • rgw zonegroup root pool ：Jewel版本中指定辖区所使用的存储池
  • rgw region ：指定该实例的辖区名
  • rgw zone ：指定该实例的域名
  • rgw zone root pool ：指定域所使用的存储池

接下来在各自实例上生成一个相同的json格式的辖区文件：

us-east us-west 
docker exec -it ceph bash
cat << EOF > us.json
{
  "name": "us",
  "api_name": "us",
  "is_master": "true",
  "endpoints": ["http:////192.168.33.15:80//"],
  "master_zone": "us-east",
  "zones": [
    {
      "name": "us-east",
      "endpoints": ["http:////192.168.33.15:80//"],
      "log_meta": "true",
      "log_data": "true"
    },
    {
      "name": "us-west",
      "endpoints": ["http:////192.168.33.16:80//"],
      "log_meta": "true",
      "log_data": "true"
    }
  ],
  "placement_targets": [
   {
     "name": "default-placement",
     "tags": []
   }
  ],
  "default_placement": "default-placement"
}
EOF
exit

然后通过辖区文件生成us辖区并设置其为默认辖区：

us-east 
docker exec -it ceph bash
radosgw-admin region set --infile us.json --name client.radosgw.us-east-1
radosgw-admin region default --rgw-region=us --name client.radosgw.us-east-1
radosgw-admin regionmap update --name client.radosgw.us-east-1
exit
us-west 
docker exec -it ceph bash
radosgw-admin region set --infile us.json --name client.radosgw.us-west-1
radosgw-admin region default --rgw-region=us --name client.radosgw.us-west-1
radosgw-admin regionmap update --name client.radosgw.us-west-1
exit

辖区搞定之后，就轮到域啦。在各自实例上生成两个相同的json格式的域文件：

us-east us-west 
docker exec -it ceph bash
cat << EOF > us-east.json
{
  "domain_root": ".us-east.domain.rgw",
  "control_pool": ".us-east.rgw.control",
  "gc_pool": ".us-east.rgw.gc",
  "log_pool": ".us-east.log",
  "intent_log_pool": ".us-east.intent-log",
  "usage_log_pool": ".us-east.usage",
  "user_keys_pool": ".us-east.users",
  "user_email_pool": ".us-east.users.email",
  "user_swift_pool": ".us-east.users.swift",
  "user_uid_pool": ".us-east.users.uid",
  "system_key": {
    "access_key": "",
    "secret_key": ""
  },
  "placement_pools": [
    {
      "key": "default-placement",
      "val": {
        "index_pool": ".us-east.rgw.buckets.index",
        "data_pool": ".us-east.rgw.buckets"
      }
    }
  ]
}
EOF

sed 's/east/west/g' us-east.json > us-west.json
exit

然后通过域文件生成两个域并更新辖区图（region map）：

us-east 
docker exec -it ceph bash
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1
radosgw-admin regionmap update --name client.radosgw.us-east-1
exit
us-west 
docker exec -it ceph bash
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1
radosgw-admin regionmap update --name client.radosgw.us-west-1
exit

在 us-east-1 实例上，生成 us-east 的用户，并用生成的 access_key 和 secret_key 填充刚才为空的 us-east.json 文件，并将其复制到 us-west 虚拟机上：

us-east 
docker exec -it ceph bash
radosgw-admin user create --uid="us-east" --display-name="Region-US Zone-East" --name client.radosgw.us-east-1 --system | tee eastuser.txt
export SRC_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' eastuser.txt`
export SRC_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' eastuser.txt`
sed -i "s/access_key/": /"/access_key/": /"$SRC_ACCESS_KEY/g" us-east.json
sed -i "s/secret_key/": /"/secret_key/": /"$SRC_SECRET_KEY/g" us-east.json
exit

docker cp ceph:us-east.json us-east.json
scp us-east.json vagrant@192.168.33.16:/home/vagrant    # vagrant的密码也是vagrant

在 us-west-1 实例上，生成 us-west 的用户，也用生成的 access_key 和 secret_key 填充刚才为空的 us-west.json 文件，并将其复制到 us-east 虚拟机上：

us-west 
docker exec -it ceph bash
radosgw-admin user create --uid="us-west" --display-name="Region-US Zone-West" --name client.radosgw.us-west-1 --system | tee westuser.txt
export DEST_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' westuser.txt`
export DEST_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' westuser.txt`
sed -i "s/access_key/": /"/access_key/": /"$DEST_ACCESS_KEY/g" us-west.json
sed -i "s/secret_key/": /"/secret_key/": /"$DEST_SECRET_KEY/g" us-west.json
exit

docker cp ceph:us-west.json us-west.json
scp us-west.json vagrant@192.168.33.15:/home/vagrant    # vagrant的密码也是vagrant

现在两台虚拟机的用户主文件夹里都有对方的json文件，分别复制进ceph容器里：

us-east 
docker cp us-west.json ceph:/us-west.json
us-west 
docker cp us-east.json ceph:/us-east.json

接下来分别在两个实例里更新带了 access_key 和 secret_key 的各两个域：

us-east 
docker exec ceph radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1
docker exec ceph radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1
us-west 
docker exec ceph radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1
docker exec ceph radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1

都完成了以后，就可以重启ceph服务和apache2啦：

us-east us-west 
docker restart ceph
sudo /etc/init.d/radosgw start
sudo service apache2 restart

Apache2启动完成后，在浏览器打开 http://192.168.33.15/ 或 http://192.168.33.16/ 应该能看到下面的xml： 

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>anonymous</ID>
    <DisplayName/>
  </Owner>
  <Buckets/>
</ListAllMyBucketsResult>

如果只看到了500的错误，等一会儿再刷新一次即可。如果遇到麻烦，可以这样调试：输入命令 sudo lsof -i :9000 看看是否radosgw启动了这个端口。如果没有，输入命令 ps -ef | grep radosgw 看看radosgw是否正常启动。若是正常启动，应该会有一个 /usr/bin/radosgw -n client.radosgw.us-east-1 的进程。若是没有正常启动，可以检查 /ect/ceph/ceph.conf 的内容，一般都是配置有问题。

搞定ceph和apache2后，在 us-east 里用python的boto库给 us-east-1 实例创建一个名为 my-new-bucket 的存储桶，并给 ggg 的key上传一句 Hello world ：

us-east 
export SRC_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' us-east.json`
export SRC_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' us-east.json`

cat << EOF > s3test.py
import boto
import boto.s3
import boto.s3.connection
import os
from boto.s3.key import Key

access_key = os.environ["SRC_ACCESS_KEY"]
secret_key = os.environ["SRC_SECRET_KEY"]
conn = boto.connect_s3(
  aws_access_key_id = access_key,
  aws_secret_access_key = secret_key,
  host = '192.168.33.15',
  is_secure=False,
  calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.create_bucket('my-new-bucket')

k = Key(bucket)
k.key = 'ggg'
k.set_contents_from_string('Hello world')

for bucket in conn.get_all_buckets():
  print "{name}/t{created}".format(
    name = bucket.name,
    created = bucket.creation_date,
)
EOF

python s3test.py

现在就该启动 radosgw-agent 来同步数据啦：

us-west 
export SRC_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' us-east.json`
export SRC_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' us-east.json`
export DEST_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' us-west.json`
export DEST_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' us-west.json`

cat << EOF > cluster-data-sync.conf
src_zone: us-east
source: http://192.168.33.15
src_access_key: $SRC_ACCESS_KEY
src_secret_key: $SRC_SECRET_KEY
dest_zone: us-west
destination: http://192.168.33.16
dest_access_key: $DEST_ACCESS_KEY
dest_secret_key: $DEST_SECRET_KEY
log_file: /var/log/radosgw/radosgw-sync-us-east-west.log
EOF

sudo radosgw-agent -c cluster-data-sync.conf

再打开一个终端窗口，用以下命令查看 us-west-1 实例是不是已经把 my-new-bucket 同步过来啦：

us-west 
docker exec ceph radosgw-admin metadata bucket list --name client.radosgw.us-west-1

可能是由于单机 ceph/demo 容器的性能极差，在同步对象的时候基本上就一直停在 INFO:radosgw_agent.worker:syncing bucket “my-new-bucket” 上。如果有真实环境的ceph应该能够很快同步过来。如果同步成功，可以用以下命令来得到刚才的 Hello world ：

us-west 
cat << EOF > s3download.py
import boto
import boto.s3
import boto.s3.connection
import os
from boto.s3.key import Key

access_key = os.environ["SRC_ACCESS_KEY"]
secret_key = os.environ["SRC_SECRET_KEY"]
conn = boto.connect_s3(
  aws_access_key_id = access_key,
  aws_secret_access_key = secret_key,
  host = '192.168.33.16',
  is_secure=False,
  calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.get_bucket('my-new-bucket')

key = bucket.get_key('ggg')
print key.get_contents_as_string()
EOF

python s3download.py

下面是一张同辖区同步示意图：

使用RADOSGW联盟网关跨域同步

不同辖区的同步

不同的辖区只能同步元数据而不能同步数据对象。接下来我们在eu-east上，尝试同步us-east的元数据。有了相同辖区的同步的经验，这回就不详细介绍下面的命令了：

eu-east 
docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=192.168.33.17 -e CEPH_NETWORK=192.168.33.17/24 --name=ceph ceph/demo

# 创建存储池这一步也同上面一样是可选
docker exec -it ceph bash
ceph osd pool create .eu-east.rgw.root 16 16
ceph osd pool create .eu-east.rgw.control 16 16
ceph osd pool create .eu-east.rgw.gc 16 16
ceph osd pool create .eu-east.rgw.buckets 512 512
ceph osd pool create .eu-east.rgw.buckets.index 32 32
ceph osd pool create .eu-east.rgw.buckets.extra 16 16
ceph osd pool create .eu-east.log 16 16
ceph osd pool create .eu-east.intent-log 16 16
ceph osd pool create .eu-east.usage 16 16
ceph osd pool create .eu-east.users 16 16
ceph osd pool create .eu-east.users.email 16 16
ceph osd pool create .eu-east.users.swift 16 16
ceph osd pool create .eu-east.users.uid 16 16
exit

# 创建密钥环和网关用户
sudo ceph auth del client.radosgw.gateway
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.eu-east-1 --gen-key
sudo ceph-authtool -n client.radosgw.eu-east-1 --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.eu-east-1 -i /etc/ceph/ceph.client.radosgw.keyring

# 配置apache2
cat << EOF > rgw.conf
FastCgiExternalServer /var/www/s3gw.fcgi -host localhost:9000

<VirtualHost *:80>

    ServerName localhost
    ServerAlias *.localhost
    ServerAdmin qinghua@ggg.com
    DocumentRoot /var/www
    RewriteEngine On
    RewriteRule  ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

    <IfModule mod_fastcgi.c>
           <Directory /var/www>
            Options +ExecCGI
            AllowOverride All
            SetHandler fastcgi-script
            Order allow,deny
            Allow from all
            AuthBasicAuthoritative Off
        </Directory>
    </IfModule>

    AllowEncodedSlashes On
    ErrorLog /var/log/apache2/error.log
    CustomLog /var/log/apache2/access.log combined
    ServerSignature Off

</VirtualHost>
EOF

sudo mv rgw.conf /etc/apache2/conf-enabled/rgw.conf

# 配置FastCGI
cat << EOF > s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.eu-east-1
EOF

sudo mv s3gw.fcgi /var/www/s3gw.fcgi
sudo chmod +x /var/www/s3gw.fcgi

# 配置ceph
sudo sed -i '$a rgw region root pool = .eu.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zonegroup root pool = .eu.rgw.root' /etc/ceph/ceph.conf
sudo sed -i $'$a ///n' /etc/ceph/ceph.conf
sudo sed -i '$a [client.radosgw.eu-east-1]' /etc/ceph/ceph.conf
sudo sed -i '$a rgw region = eu' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone = eu-east' /etc/ceph/ceph.conf
sudo sed -i '$a rgw zone root pool = .eu-east.rgw.root' /etc/ceph/ceph.conf
sudo sed -i '$a rgw dns name = eu-east' /etc/ceph/ceph.conf
sudo sed -i '$a host = eu-east' /etc/ceph/ceph.conf
sudo sed -i '$a keyring = /etc/ceph/ceph.client.radosgw.keyring' /etc/ceph/ceph.conf
sudo sed -i '$a rgw socket path = ""' /etc/ceph/ceph.conf
sudo sed -i '$a log file = /var/log/radosgw/client.radosgw.eu-east-1.log' /etc/ceph/ceph.conf
sudo sed -i '$a rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0' /etc/ceph/ceph.conf
sudo sed -i '$a rgw print continue = false' /etc/ceph/ceph.conf
sudo sed -i $'$a ///n' /etc/ceph/ceph.conf

接下来就是辖区和域的配置了。需要设置us的辖区和eu自己的辖区，否则会报错： AssertionError: No master zone found for region default 。但是域只用设置eu自己的就好：

eu-east 
docker exec -it ceph bash

cat << EOF > us.json
{
  "name": "us",
  "api_name": "us",
  "is_master": "true",
  "endpoints": ["http:////192.168.33.15:80//"],
  "master_zone": "us-east",
  "zones": [
    {
      "name": "us-east",
      "endpoints": ["http:////192.168.33.15:80//"],
      "log_meta": "true",
      "log_data": "true"
    },
    {
      "name": "us-west",
      "endpoints": ["http:////192.168.33.16:80//"],
      "log_meta": "true",
      "log_data": "true"
    }
  ],
  "placement_targets": [
   {
     "name": "default-placement",
     "tags": []
   }
  ],
  "default_placement": "default-placement"
}
EOF

cat << EOF > eu.json
{
  "name": "eu",
  "api_name": "eu",
  "is_master": "false",
  "endpoints": ["http:////192.168.33.15:80//"],
  "master_zone": "eu-east",
  "zones": [
    {
      "name": "eu-east",
      "endpoints": ["http:////192.168.33.17:80//"],
      "log_meta": "true",
      "log_data": "true"
    }
  ],
  "placement_targets": [
   {
     "name": "default-placement",
     "tags": []
   }
  ],
  "default_placement": "default-placement"
}
EOF

# 设置辖区
radosgw-admin region set --infile us.json --name client.radosgw.eu-east-1
radosgw-admin regionmap update --name client.radosgw.eu-east-1

radosgw-admin region set --infile eu.json --name client.radosgw.eu-east-1
radosgw-admin region default --rgw-region=eu --name client.radosgw.eu-east-1
radosgw-admin regionmap update --name client.radosgw.eu-east-1

cat << EOF > eu-east.json
{
  "domain_root": ".eu-east.domain.rgw",
  "control_pool": ".eu-east.rgw.control",
  "gc_pool": ".eu-east.rgw.gc",
  "log_pool": ".eu-east.log",
  "intent_log_pool": ".eu-east.intent-log",
  "usage_log_pool": ".eu-east.usage",
  "user_keys_pool": ".eu-east.users",
  "user_email_pool": ".eu-east.users.email",
  "user_swift_pool": ".eu-east.users.swift",
  "user_uid_pool": ".eu-east.users.uid",
  "system_key": {
    "access_key": "",
    "secret_key": ""
  },
  "placement_pools": [
    {
      "key": "default-placement",
      "val": {
        "index_pool": ".eu-east.rgw.buckets.index",
        "data_pool": ".eu-east.rgw.buckets"
      }
    }
  ]
}
EOF

# 设置域
radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.eu-east-1
radosgw-admin regionmap update --name client.radosgw.eu-east-1

# 创建用户
radosgw-admin user create --uid="eu-east" --display-name="Region-EU Zone-East" --name client.radosgw.eu-east-1 --system | tee eastuser.txt
export SRC_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' eastuser.txt`
export SRC_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' eastuser.txt`
sed -i "s/access_key/": /"/access_key/": /"$SRC_ACCESS_KEY/g" eu-east.json
sed -i "s/secret_key/": /"/secret_key/": /"$SRC_SECRET_KEY/g" eu-east.json

radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.eu-east-1

exit

都完成了以后，就可以重启ceph服务和apache2啦：

eu-east 
docker restart ceph
sudo /etc/init.d/radosgw start
sudo a2enmod rewrite
sudo service apache2 restart

最后同步元数据：

eu-east 
scp vagrant@192.168.33.15:/home/vagrant/us-east.json .    # vagrant的密码也是vagrant

docker cp ceph:eu-east.json eu-east.json
export SRC_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' us-east.json`
export SRC_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' us-east.json`
export DEST_ACCESS_KEY=`sed -n 's/ *"access_key": "/(.*/)",//1/p' eu-east.json`
export DEST_SECRET_KEY=`sed -n 's/ *"secret_key": "/(.*/)"//1/p' eu-east.json`

cat << EOF > cluster-data-sync.conf
src_zone: us-east
source: http://192.168.33.15
src_access_key: $SRC_ACCESS_KEY
src_secret_key: $SRC_SECRET_KEY
dest_zone: eu-east
destination: http://192.168.33.17
dest_access_key: $DEST_ACCESS_KEY
dest_secret_key: $DEST_SECRET_KEY
log_file: /var/log/radosgw/radosgw-sync-eu-east-west.log
EOF

sudo radosgw-agent -c cluster-data-sync.conf --metadata-only

如果不加 --metadata-only ，则会报错： ERROR:root:data sync can only occur between zones in the same region 。同步完成后，我们运行以下命令查看现在 eu-east-1 实例里的存储桶：

eu-east 
docker exec ceph radosgw-admin metadata bucket list --name client.radosgw.eu-east-1

应该能够看到先前在 us-east-1 创建的 my-new-bucket 。下面是一张不同辖区同步示意图：

使用RADOSGW联盟网关跨域同步

常见命令

再介绍一些ceph的常见命令： 

rados lspools                                                # 列出所有的存储池
rados ls -p .rgw.root                                        # 列出.rgw.root存储池的所有对象
rados get zone_info.default obj.txt -p .rgw.root             # 将.rgw.root存储池的zone_info.default对象内容保存到obj.txt文件
rados rm region_info.default -p .us.rgw.root                 # 删除.us.rgw.root存储池的region_info.default对象
radosgw-admin region list --name client.radosgw.us-east-1    # 列出client.radosgw.us-east-1实例的所有辖区
radosgw-admin region get --name client.radosgw.us-east-1     # 查看client.radosgw.us-east-1实例的主辖区
radosgw-admin zone list --name client.radosgw.us-east-1      # 列出client.radosgw.us-east-1实例的所有域
radosgw-admin zone get --name client.radosgw.us-east-1       # 查看client.radosgw.us-east-1实例的主域
原文  http://qinghua.github.io/ceph-radosgw-replication/
正文到此结束
所属分类： 编程技术

热门推荐

相关文章

阿里云首购8折
说给你听

本文目录
随机标签
书籍教程
Java入门教程 bootstrap3 CSS Apache基础教程 springboot-demo php ionic 教程 Python mysql教程 eclipse Ubuntu VPS系统配置 AngularJS 教程 Struts2教程 MongoDB教程 Redis教程 Spring教程 Git教程 Jenkins进阶系列 openfire参考指南 Java设计模式 HBase教程 Maven教程 hibernate教程 Docker 教程 memcached教程 Quartz指南 SpringCloud ANTLR教程 Hive教程 java实例教程 Ant教程 Hazelcast教程 Elastic-Job-Lite XStream教程 深入浅出MyBatis SVN教程 ibaties教程 rabittmq教程 WebService CXF学习 Hadoop教程 solr教程 JPA教程 ActiveMQ中文指南 Java内存模型 dubbo教程 Linux入门视频教程
近期评论
  1. 1 能用365块大洋去解决的事，千万不要用时间
  2. 2 Spring Boot集成atomikos快速入门Demo
  3. 3 Spring Boot集成fastdfs快速入门Demo
  4. 4 Spring Boot集成Https快速入门Demo
  5. 5 Spring Boot集成easypoi快速入门Demo
  6. 6 Spring Boot集成webflux快速入门Demo
  7. 7 Spring Boot集成Graphql快速入门Demo
  8. 8 Spring Boot API 多版本快速入门Demo
  9. 9 Spring Boot内容协商快速入门Demo
  10. 10 Spring Boot集成Debezium实现postgres增量同步
网站信息
  • 文章总数：155,458 篇
  • 文件总数：468,738 个
  • 标签总数：2,264 个
  • 分类总数：90 个
  • 留言数量：2,538 条
  • 在线人数：611 人
  • 运行天数：4,191天
  • 最后更新：2024年04月18日22点
Loading...

其他链接

关于本站

本站定位：个人技术类博客

本站作用：写博客、记日志、闲聊扯淡鼓捣技术。

问题交流

[HBLOG]公众号 HBLOG HBLOG