Disaster Recovery: Deployment Blueprint on

Overview

Mon, 01 Jan 0001 00:00:00 +0000

Disaster Recovery (DR) involves anticipating and designing an adequate response for any situation that prevents the correct functioning of a system in an organization. DR plays a key role in an organization’s business and operations continuity, and is a critical aspect in the planning and maintenance of cloud infrastructure.

A complete DR solution involves two main processes:

Failover, the process of moving business operations from a primary site which has suffered an outage, to a temporary site designated and preconfigured for such emergencies.

Architecture and Specifications

Mon, 01 Jan 0001 00:00:00 +0000

Architecture

The reference architecture used in this guide consists of two OpenNebula clusters with independent Front-ends, on Site A and Site B.

Each site contains an OpenNebula Front-end, three KVM compute nodes, and an independent Ceph cluster with RDB mirroring. The Virtual Machines running production workloads reside on Site A. The Ceph clusters share the same storage network, as shown below.

Specifications

The setup tested in this reference architecture utilizes the same versions of software components, detailed below, on both sites. Note that the Ceph rbd-mirror daemon is active on both Ceph clusters.

Configuration and Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Basic Configuration

To set up Ceph RBD mirroring between two OpenNebula sites, you will need to configure asynchronous block-level replication of RBD images in order to ensure that Virtual Machine disk images are synchronized.

First, you deploy two independent Ceph clusters, one per site (Site A and Site B). These clusters must use matching RBD pool names. (In this guide, we will use a pool called one). To avoid duplicate image names, Site B does not have any images in the same pool.

Failover

Mon, 01 Jan 0001 00:00:00 +0000

This page covers failover, the process of moving business operations from Site A to Site B in the event of an outage at Site A.

High-level Steps for Failover

In this scenario, an outage at source Site A triggers a failover to target Site B.

To move production from Site A to Site B, the basic high-level steps are:

On Site A, export the VM image file for each VM that will run on Site B.
On Site B, for each VM create the VM image file from the export in the previous step.
On Site A, export the VM template for each VM that will run on Site B.
On Site B, promote the desired RBD images or the whole image pool.
On Site B, for each VM create the VM from the VM template previously exported.
On Site B, instantiate the VM.

This guide describes these steps with example commands, and provides additional steps for testing the failover procedure with both Ceph clusters active.

Failback

Mon, 01 Jan 0001 00:00:00 +0000

This page covers failback, the process of moving business operations back to Site A, after Site A’s normal operation has been restored. This process involves resynchronizing data back to the source Ceph cluster.

High-level Steps for Failback

On Site A, demote the image pool and flag images for resync.
On Site B, terminate each VM and wait for its image to mirror successfully to Site A.
On Site B, demote the images in the pool.
On Site A, promote the images in the pool.
On Site A, start each VM.

Failback Procedure

Demote and Flag Images on Site A

If recovering from a disaster on Site A, then most probably the images on Site A were not demoted. In this case, the first step is to demote them.