A Step-by-Step Guide to Patching RoCE Switches on Oracle Exadata

Posted on by Ahmed Abdel Fattah

In the world of Oracle Exadata, performance is king. And at the heart of that performance lies the RDMA over Converged Ethernet (RoCE) Network Fabric. Whether you are running an X9M, X10M, or the latest X11M, keeping this fabric updated is critical for maintaining low-latency, high-throughput communication between your database servers and storage cells.

However, patching network infrastructure in a live production environment can be daunting. One wrong move could disrupt the entire cluster.

In this guide, we will walk through the detailed, Oracle-recommended procedure for patching RoCE switches using the patchmgr utility. We’ll cover everything from prerequisites to post-patch verification, ensuring you can perform these updates with confidence.

Why RoCE Switch Patching Matters

Starting with the Exadata X8M generation, Oracle moved from InfiniBand to RoCE. This shift brought incredible flexibility but also introduced a new layer of network complexity. The RoCE switches are not just “plug-and-play” networking gear; they are integral components of the Exadata storage stack.

Regular patching ensures:

  • Security: Protection against known vulnerabilities in the switch firmware (Cisco Nexus).
  • Stability: Fixes for bugs that might cause port flapping or fabric isolation issues.
  • Compatibility: Ensuring the switch firmware aligns with your current Exadata System Software version.

Prerequisites: Before You Begin

Before you even touch the command line, ensure you have checked these boxes. Skipping these steps is the most common cause of failed patches.

1. Identify the Correct Patch

Starting with Oracle Exadata System Software release 19.3.0, RoCE switch patches are distributed separately from the main Exadata infrastructure bundle.

  • Go to My Oracle Support.
  • Refer to Note KB153930 to find the correct “Exadata RDMA Network Fabric Switch” patch for your specific system model and software version.

2. Prepare the Driving System

You don’t need to run the patch from every server. You need one “Driving System”—an Oracle Linux machine (usually one of your DB nodes) that has network access to all RoCE switches.

  • User Privileges: If you are on a release prior to 19.3.9, you must run patchmgr as a non-root user. Even on newer releases, using a non-root user with sudo privileges is a best practice for auditability.
  • SSH Access: Ensure you have the admin password for the RoCE switches. patchmgr will use this to set up passwordless SSH keys automatically.

3. The Golden Rule: Isolate Your Updates

Never patch Storage Cells and RoCE Switches at the same time.
The RoCE switch update requires a reboot of the switch, which temporarily disrupts the RDMA fabric. Storage Cell updates require a stable fabric to migrate data and services. Perform these maintenance windows separately.

Step 1: Stage the Patch

Log in to your driving system and download the patch ZIP file. Unzip it to a dedicated directory.

$ unzip p<patch_number>_Linux-x86-64.zip
$ cd patch_switch_<version>

Inside this directory, you will find the patchmgr utility specifically built for RoCE switches.

Step 2: Create the Switch List File

This is where many admins stumble. You need to tell patchmgr exactly which switches to update and what role they play in your rack.

Create a file named switches.lst. Each line should contain the hostname or IP address of the switch, followed by a colon (:) and a configuration tag.

Common Configuration Tags:

  • leaf: Standard leaf switch in a single-rack system.
  • mspine: Spine switch (used in multi-rack configurations).
  • mleaf: Leaf switch in a multi-rack system.
  • sfleaf: Leaf switch with Secure Fabric enabled.

Example for a Single Rack (Half or Full):

exa01-sw-rocea0:leaf
exa01-sw-roceb0:leaf

Example for a Multi-Rack System:
For multi-rack systems, you must also append a unique loopback octet (the last part of the IP address) to uniquely identify each switch.

rack1-sw-rocea0:mleaf.101
rack1-sw-roceb0:mleaf.102
rack2-sw-rocea0:mleaf.103
rack2-sw-roceb0:mleaf.104

Pro Tip: Double-check your tags against the Oracle Exadata Maintenance Guide. Using the wrong tag (e.g., labeling a spine as a leaf) can lead to configuration mismatches.

Step 3: Run the Pre-Check (Do Not Skip!)

Before applying any changes, simulate the upgrade. Starting with the August 2022 patchmgr release, Oracle added robust pre-checks that analyze the health of your RoCE fabric.

Run the following command:

$ ./patchmgr --roceswitches switches.lst --upgrade --roceswitch-precheck --log_dir auto

What does this check?

  1. Connectivity: Can patchmgr reach all switches?
  2. Port Health: Are there any storage servers with only one active port connected to the switch being patched? If so, patching that switch would isolate the storage server, causing a failure. The tool will abort if this risk is detected.
  3. Configuration Validation: Does the current switch config match the expected “Golden Configuration”?

If the output says SUCCESS, you are clear to proceed. If it says FAIL, read the error log carefully. Fix the underlying network issue (e.g., replace a faulty cable or SFP) before retrying.

Step 4: Apply the Patch

Once the pre-check is clean, execute the actual upgrade. This process is rolling, meaning it will patch one switch at a time, wait for it to come back online, verify connectivity, and then move to the next.

$ ./patchmgr --roceswitches switches.lst --upgrade --log_dir auto

During this process:

  1. You may be prompted for the admin password for the switches.
  2. The switch will reboot. Expect brief interruptions in RDMA traffic for the servers connected to that specific switch.
  3. patchmgr will automatically apply the latest “Golden Configuration” template to ensure optimal settings.

Step 5: Post-Patch Verification

Don’t just walk away once the command finishes. Verify the health of your fabric.

1. Verify Configuration

You can manually trigger a configuration check to ensure all switches are aligned with the golden template:

$ ./patchmgr --roceswitches switches.lst --verify-config

2. Check Exadata Health

Run exachk or check Oracle Enterprise Manager (OEM) to ensure there are no critical alerts related to the RDMA network.

3. Validate RDMA Connectivity

On a database server, you can use ibstatus or rdma link show to ensure all links are up and operating at the expected speed (e.g., 100Gb/s).

Troubleshooting Common Issues

  • “Config validation failed”: This usually means the switch has manual changes that deviate from the Oracle standard. Use the --apply-config option to force the golden template onto the switch before retrying the upgrade.
  • Pre-check fails due to “Down Port”: Investigate the physical layer. Is the cable loose? Is the SFP module faulty? Never force an upgrade (--force) if a port is down, as you risk isolating a storage cell.
  • SSH Timeouts: Ensure your driving system has proper DNS resolution for the switch hostnames, or use IP addresses in your switches.lst file.

Conclusion

Patching RoCE switches on Exadata doesn’t have to be a stressful event. By leveraging the patchmgr utility and respecting the pre-checks, you can ensure a smooth, rolling update that keeps your fabric secure and performant.

Remember: Pre-check first, patch second, verify always.

Have you recently upgraded your Exadata RoCE fabric? Share your experiences or questions in the comments below!

Leave a Reply