Configuration and Deployment

Basic Configuration

To set up Ceph RBD mirroring between two OpenNebula sites, you will need to configure asynchronous block-level replication of RBD images in order to ensure that Virtual Machine disk images are synchronized.

First, you deploy two independent Ceph clusters, one per site (Site A and Site B). These clusters must use matching RBD pool names. (In this guide, we will use a pool called one). To avoid duplicate image names, Site B does not have any images in the same pool.

On Site A, you need to:

  • Set the VM images as persistent in the OpenNebula database. This means that modifications you make to each image will be preserved after the VM is terminated. You can set a VM image as persistent when you create the image or later, using the oneimage command. For details see Creating Images and Changing the Persistent Mode in the Images documentation.

On Site B, you need to:

  • Retrieve the metadata for each VM. You can do this with onevm show. For full details on this and other Virtual Machine operations, see Virtual Machine Instances.

Create Ceph Users

To use RBD mirroring, you will need to set up two users for the Ceph clusters: one for the rbd-mirror daemon on the source Ceph cluster (Site A) and one for the daemon on the target Ceph cluster (Site B). These users will enable the rbd-mirror daemon on each Ceph cluster to authenticate against each other.

This section lists the commands to create the Ceph user on the Ceph clusters for Site and Site B.

On Site A

To create the user in the source Ceph cluster, run:

ceph auth get-or-create client.rbd-mirror-peer-a mon 'profile rbd' osd 'profile rbd' -o /etc/ceph/site-a.client.rbd-mirror-peer-a.keyring

This creates a secret key for the user and outputs it to the file /etc/ceph/site-a.client.rbd-mirror-peer-a.keyring. You will need to copy this file to the Ceph cluster at Site B. To copy the contents to all hosts on Site B, you can run this one-line script, (replacing the node and site names if necessary):

for host in node{0..3}-site-b; do echo $host; scp /etc/ceph/site-a.client.rbd-mirror-peer-a.keyring root@$host:/etc/ceph/site-a.client.rbd-mirror-peer-a.keyring; done

Then, on each host change the ownership of the file to user ceph:

for host in node{0..3}-site-b; do echo $host; ssh $host chown ceph:ceph /etc/ceph/site-a.client.rbd-mirror-peer-a.keyring; done

On Site B

On the target Ceph cluster (Site B), you will need to create a local user for the rbd-mirror daemon. Here we will use $(hostname) to match the unique ID to that used for other Ceph services such as monitors.

ceph auth get-or-create client.rbd-mirror.$(hostname) mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.$(hostname).keyring

Enabling Daemon Access to the “Site A” Ceph Cluster

To enable the rbd-mirror daemon on Site B to access the Ceph cluster on Site A, you will need to copy the ceph.conf file from Site A to Site B, and name it site-a.conf.

On Site A, you can run the command below to copy the file to all nodes on Site B (replacing the node and site names if necessary):

for host in node{0..3}-site-b; do echo $host; scp /etc/ceph/ceph.conf root@$host:/etc/ceph/site-a.conf; done
root@site-a $ for host in node{0..3}-site-b; do echo $host; ssh $host chown ceph:ceph /etc/ceph/site-a.conf; done

Then, to change ownership of the file to system user ceph:

for host in node{0..3}-site-b; do echo $host; ssh $host chown ceph:ceph /etc/ceph/site-a.conf; done

Make sure that the name of the config file matches the name used in the keyring that stores the authentication information.

Enable Mirroring

The below commands illustrate how to enable mirroring on the source and target Ceph clusters.

Enable Mirroring on Site A

On Site A, you can enable mirroring in pool or image mode. To enable it in pool mode, run:

rbd mirror pool enable one pool

If you wish to enable mirroring with image mode, skip the above command and follow the steps below, then come back to this section to enable mirroring in Site B.

Enable Mirroring on Site B

On Site B, mirroring must always use pool mode. To enable it, run:

rbd mirror pool enable one pool

To verify mirroring:

rbd mirror pool info one

Next, we need to tell the pool on site B which keyring and Ceph config file it should use to connect to the peer (Site A).

rbd mirror pool peer add one client.rbd-mirror-peer-a@site-a

You can check the settings by running:

rbd mirror pool info one

For example:

root@site-b $ rbd mirror pool info one
Mode: pool
Site Name: 16c707cc-a764-47d4-b308-eefa06ff1205
Peer Sites: 
UUID: 164f8358-70a3-4f2d-a727-8729fa186b88
Name: site-a
Mirror UUID: 
Direction: rx-tx
Client: client.rbd-mirror-peer-a

The Direction field should display rx-tx and the client should be set correctly to match the keyring file. The name should also be shown correctly (site-a).

Install the rbd-mirror Daemon on Site B

To install the daemon, run:

apt install rbd-mirror

Then, enable and modify the systemd unit file for rbd-mirror:

systemctl enable ceph-rbd-mirror.target

cp /usr/lib/systemd/system/ceph-rbd-mirror@.service /etc/systemd/system/ceph-rbd-mirror@.service

Create the Mirroring Service on Site B

Next, you will need to create and start the mirroring service. Ensure to give it the same name as the local user for the Site B cluster created earlier (see above), or the Site A daemon won’t be able to authenticate against the Site B cluster.

systemctl enable --now ceph-rbd-mirror@rbd-mirror.$(hostname).service

If we check the status and logs of the ceph-rbd-mirror@rbd-mirror.<hostname>.service_ service, we should see that it comes up and does not log any authentication errors.

To check service status, run:

systemctl status ceph-rbd-mirror@rbd-mirror.ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0.service

For example:

root@site-b $ systemctl status ceph-rbd-mirror@rbd-mirror.ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0.service
● ceph-rbd-mirror@rbd-mirror.ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0.service - Ceph rbd mirror daemon
 	Loaded: loaded (/etc/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: enabled)
 	Active: active (running) since Mon 2025-06-09 12:04:50 UTC; 6 days ago
   Main PID: 17234 (rbd-mirror)
  	Tasks: 51
 	Memory: 55.5M
    	CPU: 21min 42.097s
 	CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@rbd-mirror.ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0.service
         	└─17234 /usr/bin/rbd-mirror -f --cluster ceph --id rbd-mirror.ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0 --setuser root --setgroup root

At this point, mirroring should be configured from Site A to Site B, with direction tx-only. To verify the mirroring, see below.

Verify Mirroring

On Site A, run:

rbd mirror pool info one

Output should similar to:

root@site-a $ rbd mirror pool info one
Mode: pool
Site Name: 277ddb4b-1323-425f-bf28-fa8c58c0137e

Peer Sites: 

UUID: 0ab31017-3e80-470d-b80d-1ba04d606b13
Name: 258816cf-14fb-4237-981a-e84b35c7b0b9
Mirror UUID: 5218ea1d-fcb7-404e-82f0-e48a77a05935
Direction: tx-only

To obtain detailed information about the mirroring, run:

rbd mirror pool status one --verbose

For example:

root@site-b $ rbd mirror pool status one --verbose
health: OK
daemon health: OK
image health: OK
images: 0 total

DAEMONS
service 14595:
  instance_id: 14597
  client_id: ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0
  hostname: ubuntu2204-kvm-ceph-squid-6-10-cqyoo-0
  version: 19.2.2
  leader: true
  health: OK

Enable Mirroring on Site A in image mode

If you want to use image mode for Site A mirroring, you will need to define which images should be mirrored, and enable the exclusive-lock and journal features for the images.

To enable journal-based mirroring for an image (in this example, image one-0-0-0), run:

rbd mirror image enable one/one-0-0-0 exclusive-lock,journal

Then, verify the image:

rbd image 'one-0-0-0':

For example:

root@site-a $ rbd image 'one-0-0-0':
	size 256 MiB in 64 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 39b66a80b833
	block_name_prefix: rbd_data.39b66a80b833
	format: 2
	features: layering, exclusive-lock, journaling
	op_features: 
	flags: 
	create_timestamp: Wed Jun  4 10:31:42 2025
	access_timestamp: Mon Jun  9 12:08:11 2025
	modify_timestamp: Mon Jun  9 12:08:05 2025
	parent: one/one-0@snap
	overlap: 256 MiB
	journal: 39b66a80b833
	mirroring state: enabled
	mirroring mode: journal
	mirroring global id: f0523ef9-a784-420f-8725-c3f81ff5a302
	mirroring primary: true

Next, since VM disks are just snapshots based on the image, we will need to flatten the required image:

rbd flatten one/one-0-0-0

After performing these steps, to enable mirroring on Site B go back to the section above.

Enable Two-way Mirroring for Failback

Failback is the process by which Virtual Machines are restored to the primary site, once normal operation on the site has resumed.

To failback Virtual Machines to Site A when it becomes available again after an outage, you will need to set up the rbd-mirror daemon on Site A. The daemon will connect to Site B during failback. On Site A, install the rbd-mirror daemon and enable the service by following the same steps described above for Site B: