This page describes the XO 5 interface. This area is not available in XO 6 yet: use the XO 5 link in the top-right corner of XO 6. See what lives where.
In computing, load balancing distributes workloads across multiple computing resources, such as computers, a computer cluster, network links, central processing units or disk drives.
In the case of virtualization, you have multiple physical hosts, which run your virtual machines (VMs). The goal here is to distribute the VM load in the best way possible across your servers.
tip
You may have heard about VMware DRS (Distributed Resource Scheduler). That's the same principle here, but for XCP-ng.
The first goal is to adapt your VM placement in real time, without service interruption, depending on the load. Since Xen Orchestra is connected to multiple pools and XCP-ng supports live storage motion, we can perform load balancing on a whole XCP-ng infrastructure, even between remote datacenters.
tip
A load balancing policy is called a plan.
Here's a simple example, with 2 hosts running 6 VMs:
A quiet cluster: two hosts, six VMsVM
Let's say both hosts are using only 5% of all their CPUs. Suddenly, one of your VMs starts to have a very high CPU load (in red): the other VMs on this same host end up starved for CPU (in amber):
VM 5 heats up, its neighbors starveCPU-hungrystarved
Host 1 is still using 5% of its CPUs, but Host 2 is now at 95%.
The load balancer detects it, and moves the starved VMs to the other host:
The plan reacts: starved VMs move out and turn healthy, the busy VM gets the hosthealthystarvedCPU-hungry
Host 1 has a slightly higher load, but Host 2 can fully handle the "problematic" VM without disrupting the other VMs.
This way, the heavy load on a single VM doesn’t end up penalizing everything else.
However, there are more ways to look at it and optimize your resource usage:
Performance mode: You might want to spread VM workloads across as many servers as possible, to get the most out of your hardware (as in the previous example).
Density mode: You might prefer to reduce power consumption by consolidating VMs onto as few hosts as possible, then shutting down the unused ones.
Performance: VMs are placed to make the most of all available resources: the load is spread across every available host, to give the best overall performance.
To specify how the performance plan should act, see the Performance plan behavior section.
Density: This time, the objective is to use the least hosts possible, and to concentrate your VMs. In this mode, emptied hosts are shut down. A few safety rules apply: the pool master is never evacuated, and only hosts with a power-on mode configured (so XO can power them back on later) are candidates for shutdown. A host is also skipped if any of its running VMs cannot be migrated (missing guest tools, or carrying an affinity/anti-affinity tag).
Simple: This mode allows you to use VM affinity and anti-affinity without any load balancing mechanism.
tip
The performance plan also powers halted hosts back on: any halted host of the plan's pools with a power-on mode configured is started, so its resources are available to spread the load. This is the counterpart of the density plan shutting hosts down.
In a load balancing plan, you can define two critical thresholds:
CPU usage (%)
Free memory (MB)
To configure thresholds for your plan:
Go to Configuration → Plans.
Select the plan you want to update.
Go to Plan → Critical thresholds.
Check the Fill information (optional) box.
New fields will appear where you can set your thresholds:
zoomed detailSet the critical thresholds of the plan
The defaults are 90% for CPU and 1000 MB of free memory.
warning
The threshold you configure is the critical value, and the plugin deliberately acts before reaching it: a host is considered overloaded when its average CPU usage exceeds 85% of the critical value (76.5% with the default 90%), and migrations continue until it falls back under 65% of it (58.5% by default). For memory it's symmetrical: migrations start when free RAM drops below 1.2× the configured value (1200 MB by default) and stop once it's back above 1.5× (1500 MB by default).
To avoid reacting to a momentary spike, the decision is based on a weighted average: 75% of the last minute, 25% of the last 30 minutes. A host has to be busy for a while before its VMs start moving.
If you want to prevent load balancing from triggering migrations on a particular host or VM, you can exclude it from the process. This can be done with:
The Excluded hosts setting in each plan: an excluded host is never used as a source or a destination, and is never powered on or shut down by the plugin
The Ignored VM tags parameter, which is common to every plan: VMs carrying one of these tags are never migrated
note
Only running VMs are considered, and a VM needs the guest tools installed to be live migrated: VMs without them simply stay where they are.
The global situation (resource usage) is examined every minute, and each run works on smoothed metrics rather than instant values (see Critical thresholds).
Every migration decided by the plugin is visible in the XO Tasks view, as a task named "Load balancer migrates VM …" whose description tells you the source, the destination and the reason: CPU over threshold for too long, CPU significantly higher than the pool average, vCPU balancing, affinity or anti-affinity satisfaction, or a density-plan host shutdown. Nothing moves silently.
tip
During a Rolling Pool Update, the load balancer is automatically suspended, and re-enabled 30 minutes after the update completes, so the two features never fight over VM placement.
This option lets you define the maximum number of migrations that can run at the same time (default: 2, shared by all plans). Limiting concurrent migrations can help maintain the overall performance of your environment.
This option prevents the load balancer from migrating the same VM twice in quick succession, by forcing a given delay between load-balancer-related migrations. Defaults to 30 minutes, set to 0 to disable it.
note
The density plan does not apply the cooldown: when it evacuates a host to shut it down, it moves everything.
With a performance plan, you can pick a behavior that adds extra logic on top of the plan's threshold-based balancing. The default behavior always stays active: these options complement it, they don't replace it.
zoomed detailThe three behaviors of a performance plan
By default, the performance plan only migrates VMs once CPU or memory usage reaches critical levels. The Preventive option goes further by trying to reduce CPU usage imbalances between hosts in the pool. For example, it aims to avoid a situation where one host runs at 60% CPU while others sit at 10%, something the default behavior would normally allow. Concretely, it kicks in when a host uses at least 1.5× the pool's average CPU (and at least 25% CPU), and only when no host is over the critical thresholds.
When the pool load is low (under 40% CPU usage), this option proactively distributes VMs across hosts to reduce large disparities in the number of vCPUs per CPU, rather than waiting until a host is overloaded.
This way, VMs are pre-positioned to minimize the number of migrations needed later as the load increases.
VM affinity keeps VMs together: all the VMs sharing one of the plan's Affinity tags are placed on the same host. This is useful when a group of VMs talk to each other a lot and benefit from staying on the same machine (chatty application tiers, a VM and its cache, etc.).
Affinity runs in every plan mode, including Simple. The plugin gathers all VMs carrying a given tag onto the host that already holds most of them, and if that host lacks memory, it will even move other, untagged VMs out of the way to make room.
warning
Avoid giving a VM several affinity tags: the plugin then has to merge all these groups into one (if VM 1 has tags A and B, and VM 2 has tags B and C, then A, B and C must all land on the same host), and it logs a warning about it.
VM anti-affinity is the opposite: it prevents VMs sharing the same tag from running on the same host. Use it to keep redundant VMs (replicas, cluster members) apart, so losing one host never takes down both copies.
For example, imagine you have several VMs running MySQL and PostgreSQL with high availability or replication.
Naturally, you don’t want both replicated databases to be placed on the same physical host.
To prevent that, set up your plan like this:
zoomed detailA simple plan with two anti-affinity tags
Simple plan: no active load balancing mechanism is used.
Anti-affinity: we added two tags, meaning any VMs with one of these tags will not run on the same host as another VM that has the same tag (whenever possible).
You can also use the Performance plan with Anti-affinity enabled to continue migrating VMs that don’t have tags: tagged VMs are left to the affinity logic, and are never moved by the performance or density algorithms themselves.
tip
This feature is not limited by the number of VMs sharing the same tag. For example, if you have six VMs with the same anti-affinity tag and two hosts, the plugin will try to place three VMs on each host. It distributes VMs as evenly as possible and, in most cases, takes priority over the performance algorithm.
