VM Backup Operations

Overview

Backup Operations

Backups can be operated in two modes:

• Single VM (described in this guide): Backup operations are defined and managed for a single VM. You can use this method to manage the backups of a few VMs.
• Backup Jobs are described in the backup jobs guide. They allow you to define backup operations involving multiple VMs and efficiently manage all the backups as a cohesive unit.

Backup Types

OpenNebula supports two backup types:

• Full, each backup contains a full copy of the VM disks. Libvirt version >= 5.5 is required.
• Incremental, each backup contains only the changes since the last backup. Incremental backups track changes by creating checkpoints (disk block dirty-bitmaps) using QEMU/Libvirt. Libvirt version >= 7.7 is required.

Incremental backups of qcow2 disks can use two different modes via the INCREMENT_MODE user setting:

• CBT (Changed Block Tracking). For each increment OpenNebula creates a block bitmap in the disk image to track which blocks have changed since the last backup.
• SNAPSHOT. OpenNebula tracks changes by creating a separate disk snapshot. This snapshot stores all disk changes since the last backup.

Also, for RBD disks (Ceph), FULL and INCREMENT backups are currently stored in a different way, although the difference should be transparent to the user:

• Full backups (FORMAT=raw) store the RBD export converted to a qcow2 file. The restore process involves converting it to a RAW file and importing it to the Ceph pool.
• Incremental backups (FORMAT=rbd) store the initial RBD export, as well as zero or more increment files, in the native format of Ceph exports (rbd export –export-format 2 / rbd export-diff). The restore process involves importing the initial export and applying the diff files in the same order, one by one.

The Backup Process

VM backups can be taken live or while the VM is powered off. The operation comprises three steps:

• Pre-backup: Disks (or increments) are prepared for backup. When the VM is running the filesystems of the guest are frozen (see below) and temporal disks are created so the VM can continue its normal operation. Note: backups are taken at the same time for all the VM disks (qcow2/raw images) to guarantee crash consistent backups.
• Backup: Full disk copies (or increments) are uploaded to the backup server. In this step, OpenNebula will use the specific datastore drivers for the backup system.
• Post-backup: Cleans any temporal file in the hypervisor.

Limitations

• Incremental backups are only available for KVM and qcow2/RBD disks
• Live backups are only supported for KVM
• Attaching a disk to a VM that had an incremental backup previously made will yield an error. The –reset option for the backup operation is required to recreate a new incremental chain
• Incremental backups on VMs with disk or system snapshots is not supported
• KEEP_LAST option is not supported for Incremental backups of Ceph disks

Preparing VMs for Backups

Before making backups you need to configure some aspects of the backup process (e.g., the backup mode). This can be done for VM templates or Virtual Machines.

Virtual Machine Templates

You can configure backups in the VM Template, so every VM created will have a preconfigured backup setup. The following example shows a VM template with incremental backups configured:

NAME   = "Template  - Backup"
CPU    = "1"
MEMORY = "2048"

DISK = [
  IMAGE_ID = "1" ]

BACKUP_CONFIG = [
  FS_FREEZE = "NONE",
  KEEP_LAST = "4",
  MODE = "INCREMENT" ]

To configure using the Sunstone GUI, select the Backup tab:

Virtual Machines

For running VMs you can set (or update) backup configuration attributes through the updateconf API call or CLI command. For example, to configure a VM with the above settings, add the following attribute:

$ onevm updateconf 0

BACKUP_CONFIG = [
   FS_FREEZE = "NONE",
   KEEP_LAST = "4",
   MODE = "INCREMENT"
]
...

You should be able to see the configuration of the VM by showing its information with onevm show command:

$ onevm show 0

VIRTUAL MACHINE 0 INFORMATION
ID                  : 0
NAME                : alpine-0
USER                : oneadmin
GROUP               : oneadmin
STATE               : ACTIVE
LCM_STATE           : RUNNING

...

BACKUP CONFIGURATION
BACKUP_VOLATILE="NO"
FS_FREEZE="NONE"
INCREMENTAL_BACKUP_ID="-1"
KEEP_LAST="4"
LAST_INCREMENT_ID="-1"
MODE="INCREMENT"

To configure using the Sunstone GUI, click on the virtual machine, select the Backup tab and click on the Backup config button:

Sunstone will display the screen to update the VM Configuration.

Reference: Backup Configuration Attributes

Taking VM Backups

Backup actions may potentially take some time, leaving some resources in use for a long time. In order to make efficient use of resources, backups are planned by the OpenNebula scheduler through the schedule actions interface.

One-shot Backups

You can take backups (one-shot) using the onevm backup operation (or the equivalent Sunstone action). The backup will use the configured attributes for the VM (e.g., MODE) and two additional arguments:

• Datastore ID: The datastore where the backup will be stored
• Reset (optional): When doing incremental backups, you can close the current active chain and create a new one by passing this flag

Important, only the oneadmin account can initiate backups directly, regular users needs to schedule the operation. See example:

$ onevm backup --schedule now -d 100 0
VM 0: backup scheduled at 2022-12-01 13:28:44 +0000

Using Sunstone to take one-shot backup:

After the backup is complete you should see the backup information in the VM details, as well as the associated backup image. For example:

$ onevm show 0
VIRTUAL MACHINE 0 INFORMATION
ID                  : 0
NAME                : alpine-0
USER                : oneadmin
GROUP               : oneadmin
STATE               : ACTIVE
LCM_STATE           : RUNNING

...

SCHEDULED ACTIONS
   ID ACTION  ARGS   SCHEDULED REPEAT   END STATUS
    0 backup   100 12/01 13:28             Done on 12/01 13:28
    1 backup   100 12/01 13:36             Done on 12/01 13:36

BACKUP CONFIGURATION
BACKUP_VOLATILE="NO"
FS_FREEZE="NONE"
INCREMENTAL_BACKUP_ID="1"
KEEP_LAST="4"
LAST_INCREMENT_ID="1"
MODE="INCREMENT"

VM BACKUPS
IMAGE IDS: 1

$ oneimage show 1
IMAGE 1 INFORMATION
ID             : 1
NAME           : 0 01-Dec 13.36.56
USER           : oneadmin
GROUP          : oneadmin
LOCK           : None
DATASTORE      : RBackups
TYPE           : BACKUP
REGISTER TIME  : 12/01 13:36:56
PERSISTENT     : Yes
SOURCE         : 25f4b298
FORMAT         : raw
SIZE           : 172M
STATE          : rdy
RUNNING_VMS    : 1

PERMISSIONS
OWNER          : um-
GROUP          : ---
OTHER          : ---

IMAGE TEMPLATE

BACKUP INFORMATION
VM             : 0
TYPE           : INCREMENTAL

BACKUP INCREMENTS
 ID PID T SIZE                DATE SOURCE
  0  -1 F 172M      12/01 13:36:56 25f4b298
  1   0 I 0M        12/01 14:22:46 6968545c

The SOURCE attribute in the backup images (and increments) is an opaque reference to the backup in the backup system used by the datastore. For restic, this corresponds to the snapshot ID, for example:

$ restic snapshots
repository d5b1499c opened (repository version 2) successfully, password is correct
ID        Time                 Host                                  Tags        Paths
-----------------------------------------------------------------------------------------------------------------
25f4b298  2022-12-01 13:36:51  ubuntu2204-kvm-local-6-5-e795-2.test  one-0       /var/lib/one/datastores/0/0/backup
6968545c  2022-12-01 14:22:44  ubuntu2204-kvm-local-6-5-e795-2.test  one-0       /var/lib/one/datastores/0/0/backup
-----------------------------------------------------------------------------------------------------------------

Note: with the restic driver each snapshot is labeled with the VM id in OpenNebula.

Scheduling Backups

You can program periodic backups through the schedule actions interface. Note that in this case, you have to pass the target datastore ID as argument of the action. You can create a periodic backup with the --schedule option in the CLI, or through Sunstone in the Schedule Action dialog (to open the dialog, click the Sched Actions tab then click Add action).

Note: As with any other schedule action, you can plan for several backup operations or add a pre-set backup schedule in the VM template.

Reference: Scheduler Backup Attributes

The schedule actions are in control of OpenNebula core. You can tune the number of concurrent backup operations with the following parameters in /etc/one/oned.conf

Cancel Backup

You can cancel an ongoing backup operation by using the onevm backup-cancel. The command will try to gracefully terminate backup operation. If the command succeeds the VM will return to running (or poweroff) state. Note that not all stages of the backup operation can be canceled and some files may be left on the VM folder in the system datastore. These files will be cleaned up during a subsequent backup.

If the backup operation is not running but the VM stays in the backup state, use command onevm recover to return the VM back to running state.

Restoring Backups

There are two main methods for restoring VM backups:

• In-place restore: This involves replacing the disks of the VM with a backup copy.
• Full restore: This process creates new disk images and templates. Unlike in-place restore, this operation doesn’t require the VM to exist beforehand.

In-place Restore

In this mode, the disks of an existing VM are replaced with a copy from a backup. This operation requires that the VM is in a powered-off state. During the restoration process, you have the option to restore all disks or only the selected one.

For example, let’s consider a scenario with VM 83 in a powered-off state and an image backup (176) of this VM with three increments. It’s important to note that the VM remains powered off during this process:

$ oneimage show 176
IMAGE 176 INFORMATION
ID             : 176

...

BACKUP INFORMATION
VM             : 83
TYPE           : INCREMENTAL

BACKUP INCREMENTS
 ID PID T SIZE                DATE SOURCE
  0  -1 F 173M      05/06 08:46:08 5f33de
  1   0 I 1M        05/06 08:52:05 a0c4eb
  2   1 I 1M        05/06 08:52:46 046843

and the corresponding VM:

$ onevm show 83
VIRTUAL MACHINE 83 INFORMATION
ID                  : 83
NAME                : alpine-83

...

VM DISKS
 ID DATASTORE  TARGET IMAGE                               SIZE      TYPE SAVE
  0 default    vda    alpine                              173M/256M file   NO
  1 -          hda    CONTEXT                             1M/-      -       -

...

BACKUP CONFIGURATION
BACKUP_VOLATILE="NO"
FS_FREEZE="NONE"
INCREMENTAL_BACKUP_ID="176"
INCREMENT_MODE="CBT"
KEEP_LAST="4"
LAST_INCREMENT_ID="2"
MODE="INCREMENT"

VM BACKUPS
IMAGE IDS: 176

...

To restore all disks of the VM from the second increment (ID 1), simply execute:

$ onevm restore --increment 1 83 176

Note that all snapshots of the VM will be deleted upon restoring the backup.

Full Restore

When you perform a full restore of a VM backup, OpenNebula will create:

• A Virtual Machine Template, with an equivalent definition to that of the VM when the backup was taken (i.e., NICs, capacity…)
• A disk image for each of the disks stored in the backup.

Note that in this case the VM does not have to exist. This operation is not tied to the original VM where the backup was made.

When you restore the backup you may choose to:

• Not keep the NIC addressing (i.e., IPs, or MAC)
• Not keep any NIC definition
• In the case of incremental backups you can choose which increment to restore (or last by default)
• Pick a base name for the VM templates and disk Images that will be created
• Restore only an individual disk, without the associated VM template

After you restore the VM, we recommend reviewing the restored template to fine-tune any additional parameter. The following example shows the recovery procedure:

$ oneimage restore -d default --no_ip 1
VM Template: 1
Images: 2

The API call returns the IDs of the images (2, in the example) and the ID of the VM template (1). As you see, images are named after the VM and snapshot in the form: <VM_ID>-<SNAPSHOT_ID>-disk-<DISK_ID>.

$ oneimage show
IMAGE 2 INFORMATION
ID             : 2
NAME           : 0-6968545c-disk-0
USER           : oneadmin
GROUP          : oneadmin
LOCK           : None
DATASTORE      : default
TYPE           : OS
REGISTER TIME  : 12/01 15:03:33
PERSISTENT     : No
SOURCE         : /var/lib/one//datastores/1/d7784b595d33b757bb2593661346c51c
PATH           : restic://100/0:25f4b298,1:6968545c//var/lib/one/datastores/0/0/backup/disk.0

The complete list of attributes removed from a template are described in the table below:

VM Template attributes removed upon restore

Advanced Configurations

Quotas and Access Control

Backup Datastores follow the same datastore abstraction as the Image and System Datastore. Hence the same operations are supported for Backup Datastores. In particular you can easily set quotas to limit:

• The overall size that backups can take from the backup storage for a given group or user
• The number of backups a user can create (Important: increments counts just as a single backup)

In the same way, you can limit which backup datastore a given user or group can use by simply adjusting the permissions or, if you need a finer grain, by setting an ACL.

Multi-tier Backup Policies (Full Backups)

If you are using FULL backups you can schedule backups in different servers (i.e., different datastores) using different schedules. For example:

• Schedule a backup in the Datastore “in-house” every Friday.
• Schedule a backup in the Datastore “cloud-storage” once every month.

< Previous page Next page >