Optimizing Placement Group (PG) Distribution with the Ceph Balancer

#ceph #ceeph_storage #network_storage #tunning #balancer

Optimizing Placement Group (PG) Distribution with the Ceph Balancer

Introduction

The Ceph balancer is designed to optimize the allocation of placement groups (PGs) across OSDs to ensure a balanced distribution. It can function in either automatic or supervised mode.

There are two available modes for the Ceph balancer:

1. crush-compact (old and risky)
2. upmap (newer and safer)

We will use upmap, as it provides greater accuracy and stability.

How Upmap Works

Starting from Luminous and later releases, the OSDMap can store explicit upmap entries. These entries override the default CRUSH placement calculations, allowing fine-grained PG control to achieve a more even distribution.

In most cases, upmap results in an almost perfect balance, where each OSD gets an equal number of PGs (with a possible difference of ±1).

Example:

• Suppose we have 11 PGs and 5 OSDs.
• The upmap balancer will attempt to distribute 2 PGs per OSD, but one OSD will receive an extra PG (3 PGs instead of 2).
• This results in a nearly balanced distribution.

In our actual scenario, we noticed an imbalance:

• The last OSD has 50 PGs, while the first OSD has only 20 PGs.
• This means the last OSD is under higher load, which can lead to performance degradation.

To resolve this issue, we need to fine-tune the balancer settings.

Checking and Configuring the Ceph Balancer

1. Checking the Balancer Status

First, check if the balancer is active:

ceph balancer status
        

Example Output:

{
    "active": false,
    "last_optimize_duration": "",
    "last_optimize_started": "",
    "mode": "upmap",
    "no_optimization_needed": false,
    "optimize_result": "",
    "plans": []
}
        

As seen above, the balancer is currently disabled.

2. Adjusting Balancer Parameters

Before enabling the balancer, we need to tweak its settings for optimal performance.

2.1. Target Misplaced Object Ratio

The balancer is allowed to misplace a limited number of objects. To check the current threshold, run:

ceph config get mgr target_max_misplaced_ratio
        

Example Output:

0.50000  # (5%)
        

A 5% threshold is too high for our setup. We will reduce it to 2%:

ceph config set mgr target_max_misplaced_ratio .02
        

2.2. Balancer Sleep Interval

The balancer pauses between optimization runs. To check the sleep interval:

ceph config get mgr mgr/balancer/sleep_interval
        

To set the sleep interval to 60 seconds:

ceph config set mgr mgr/balancer/sleep_interval 60
        

2.3. Automatic Balancing Schedule

To check the start and end times for automatic balancing:

ceph config get mgr mgr/balancer/begin_time
ceph config get mgr mgr/balancer/end_time
        

To set balancing to run 24/7:

ceph config set mgr mgr/balancer/begin_time 0000
ceph config set mgr mgr/balancer/end_time 2359
        

2.4. Restricting Balancing to Specific Days

To check the current weekday restrictions:

ceph config get mgr mgr/balancer/begin_weekday
ceph config get mgr mgr/balancer/end_weekday
        

To allow balancing every day:

ceph config set mgr mgr/balancer/begin_weekday 0
ceph config set mgr mgr/balancer/end_weekday 6
        

2.5. Restricting Balancing to Specific Pools

By default, all pools are balanced. To limit balancing to specific pools:

ceph config set mgr mgr/balancer/pool_ids 1,2,3
        

To verify the current pool settings:

ceph config get mgr mgr/balancer/pool_ids
        

3. Enabling the Balancer

Once all parameters are properly set, enable the balancer:

ceph balancer on
        

To confirm:

ceph balancer status
        

Example Output:

{
    "active": true,
    "last_optimize_duration": "0:00:00.002238",
    "last_optimize_started": "Sat Jan 18 12:24:16 2025",
    "mode": "upmap",
    "no_optimization_needed": true,
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}
        

This confirms the balancer is active and optimizing the PG distribution.

Disabling the Balancer (In Case of Issues)

If needed, you can disable the balancer at any time:

ceph balancer off
        

# Check the current status of the Ceph balancer
sudo ceph balancer status

# Set the maximum misplaced object ratio to 0.5% (default is 5%) to reduce imbalance impact
sudo ceph config set mgr target_max_misplaced_ratio .005

# List all available pools in the Ceph cluster
sudo ceph osd lspools

# Restrict automatic balancing to pools with IDs 1 and 3
sudo ceph config set mgr mgr/balancer/pool_ids 1,3

# Set the balancer to start running at 11:00 PM
sudo ceph config set mgr mgr/balancer/begin_time 2300 

# Set the balancer to stop running at 5:00 AM
sudo ceph config set mgr mgr/balancer/end_time 0500 

# Set the balancer to start running from Sunday (0 represents Sunday)
sudo ceph config set mgr mgr/balancer/begin_weekday 0

# Set the balancer to stop running on Saturday (6 represents Saturday)
sudo ceph config set mgr mgr/balancer/end_weekday 6

# Set the sleep interval between balancer runs to 600 seconds (10 minutes)
sudo ceph config set mgr mgr/balancer/sleep_interval 600

# Update the balancer start time to 7:30 PM
sudo ceph config set mgr mgr/balancer/begin_time 1930 

# Update the balancer end time to 1:30 AM
sudo ceph config set mgr mgr/balancer/end_time 0130 

# Retrieve the configured start time for automatic balancing
sudo ceph config get mgr mgr/balancer/begin_time 

# Retrieve the configured end time for automatic balancing
sudo ceph config get mgr mgr/balancer/end_time 

# Retrieve the configured start day for automatic balancing
sudo ceph config get mgr mgr/balancer/begin_weekday

# Retrieve the configured end day for automatic balancing
sudo ceph config get mgr mgr/balancer/end_weekday

# Retrieve the current sleep interval between balancer runs
sudo ceph config get mgr mgr/balancer/sleep_interval

# Retrieve the current target max misplaced object ratio
sudo ceph config get mgr target_max_misplaced_ratio

# Retrieve the current list of pool IDs assigned for automatic balancing
sudo ceph config get mgr mgr/balancer/pool_ids
        

Conclusion

The Ceph balancer is an essential tool for ensuring efficient PG distribution and avoiding overloading OSDs. By configuring it properly, we can maximize cluster performance and reliability.