Storage Bitmaps
In storage, a bitmap is an index mapping where al content is. It is also called a bit array and it will allow you to track changes.
Dirty Bitmap
A dirty bitmap refers to a table of changes, it has just the changes that you will need for ex. a backup job, where you only want to read the changes since last backup.
In Proxmox (PVE) and Proxmox Backup Server (PBS) the backup solution keeps a bitmap of changes for the storage, for all running VMs. The dirty bitmap gets removed when the VM is stopped, unless it is a persistent dirty-bitmap. Proxmox can support persistent bitmaps, only for the qcow2 file format for disks, but do NOT use persistent bitmaps for backups.
The dirty bitmap allow Proxmox to know what data have been changed on storage since last backup, and it will only read the changes from the VM and send and write the changes to PBS. This drastically reduce the backup time, and is therefore extremely important for efficient backups. The efficiency also allows you to run many backups daily, if you need that. There will be a short interruption when the backup starts to snapshot, it is usually only milliseconds, and backups running every 5 minutes is completely viable.
Stopped VMs
As long as the VM is running, the disks are locked and nothing else can change content in disks, so Proxmox knows that all changes are in the bitmap. But if the VM is stopped, content of disks could be changed by other processes, and Proxmox no longer knows if the bitmap of changes is current or if there are more changes. This is very rare but possible, ex. if you attach the disk to another VM, do changes, then attach the disk back to original VM. In backup procedures, even if the case is rare and you think “this will never happen at our company”, years of experience will at some point teach you that you should never do it.
Backup procedures should never allow any kind of backup jobs, that you cannot be sure to include all your data. Therefore the bitmap is removed, when the VM is stopped.
Persistent bitmaps
A persistent bitmap is a bitmap that stays forever, no matter if the VM is running or stopped. For this to work the bitmap cannot reside outside of the file system, to be persistent and current, the bitmap must be part of the file system and stay inside the disk file. Proxmox supports persistent bitmaps, only for the qcow2 file format for disks, but do NOT use them for backups.
- Shutdown or Stopping a VM: Dirty bitmap will be removed. A full read of the entire disks is required on every backup, while VM is stopped. Will still only send and write changes. Only your VM storage read IO will be affected, a total read is required.
- Rebooting a VM (from Proxmox or client): Dirty bitmap will be removed. Total read required.
- Changing backup target: Bitmap will be removed, total backup read and write is required.
- Backup to several different targets or namespaces: Bitmap per namespace on PBS is coming soon, allowing the same disk to use different bitmaps for different backup targets.
Enable persistent dirty-bitmaps in Proxmox / PBS
No, Proxmox supports persistent dirty bitmaps for qcow2 file format in, and they can be used from the API, but NOT from PBS.
PBS backup jobs
- “dirty-bitmap status: OK” – Proxmox are only reading the changes since last time, from the dirty-bitmap.
- dirty-bitmap status: created new” – there is no dirty bitmap, and Proxmox created a new. Entire disk will be read again.
Notice that the dirty-bitmap status is per disk, you can have one disk with a dirty-bitmap and another on the same VM that needs to be created, in the same backup job for same VM, ex. from a backup log, with disk 0 needing a new dirty bitmap, 1 have one with no changes, 2 have one and there is 40 MB of changes to backup).
INFO: scsi0: dirty-bitmap status: created new INFO: scsi1: dirty-bitmap status: OK (drive clean) INFO: scsi2: dirty-bitmap status: OK (40.0 MiB of 32.0 GiB dirty)
Disk file
Get info about a disk file:
cd /mnt/pve/NAME/images/VMID qemu-img info vm-VMID-disk-0.qcow2
Check if a bitmap is present – MAKE SURE VM IS OFF – because of force share. Backing-chain will also check higher hierarchy, like snapshots etc:
qm shutdown VMID qemu-img info --force-share --backing-chain vm-VMID-disk-0.qcow2
If a bitmap is present, you will see it, and if no bitmap is there, there will be no bitmap info. Bitmap present shows like:
Bitmaps:
Name: "bitmap-name"
It is possible to manually use qemu-img to add and enable bitmaps, but this will not help you for Proxmox and PBS to use dirty-bitmap. For that, Proxmox needs to enable and create the bitmap itself.
Links
- Proxmox – Persistent Dirty Bitmaps