Mastering Exadata Database Server Patching: A Step-by-Step Guide with patchmgr

Posted on by Ahmed Abdel Fattah

Patching an Oracle Exadata Database Machine is a critical task that requires precision, planning, and a deep understanding of the underlying architecture. Unlike traditional servers, Exadata integrates compute, storage, and networking into a cohesive system. When it comes to updating the Database Servers (Compute Nodes), Oracle provides a powerful automation tool: the DB Node Update Utility.

In this guide, we will walk through the complete lifecycle of patching Exadata Database Servers using patchmgr in orchestration mode. We’ll cover prerequisites, execution steps, precautions, and post-patching validation to ensure a smooth and successful update.

Why Use patchmgr?

Starting with Oracle Exadata System Software release 12.1.2.2.0, Oracle recommends using patchmgr for updating database nodes. While you can still use dbnodeupdate.sh in standalone mode, patchmgr offers significant advantages:

  • Orchestration: It manages the update sequence across multiple nodes from a single command.
  • Rolling Updates: It supports rolling updates, allowing you to patch nodes one by one while maintaining database availability (ideal for RAC environments).
  • Automation: It automatically handles stopping/starting Grid Infrastructure, managing VMs, taking OS backups, and relinking Oracle Homes.
  • Safety: It performs rigorous pre-checks and creates automatic backups before applying changes.

Note: patchmgr cannot patch the server it is running on. Therefore, you must execute it from a “driving node” that is not part of the target list, or patch the driving node last using dbnodeupdate.sh.

Phase 1: Preparation and Prerequisites

Before touching any production system, thorough preparation is key to avoiding downtime and unexpected errors.

1. Download the Patch Bundle

Visit My Oracle Support (MOS) and download the latest dbserver.patch.zip file. Refer to MOS Note 1553103.1 for the latest version details and compatibility matrix.

2. Stage the Files

Choose one database server as your Driving Node. Create a staging directory and unzip the patch bundle there.

mkdir -p /u01/staging/exadata_patch
unzip dbserver.patch.zip -d /u01/staging/exadata_patch

3. Create the Node List File

Create a text file (e.g., dbnode_list.txt) containing the hostnames of all database servers you intend to patch.

db01.example.com
db02.example.com
db03.example.com
db04.example.com

4. Run Exachk Health Check

Run the Exachk utility to identify any pre-existing configuration issues that might block the patching process.

exachk -nodes all -pre

Resolve any critical findings before proceeding.

5. Verify System Readiness

Ensure the following:

  • Sufficient disk space on the root filesystem.
  • No active network mounts (NFS) unless you plan to use the --allow_active_network_mounts flag.
  • All cluster services are healthy.

Phase 2: Pre-Check and Validation

Never skip the pre-check phase. This step validates prerequisites, checks for conflicts, and ensures the system is ready for the upgrade.

Run the following command from the staging directory on your driving node:

cd /u01/staging/exadata_patch
./patchmgr -dbnodes dbnode_list.txt \
  -dbnode_precheck \
  -dbnode_loc /u01/staging/exadata_patch \
  -target_version <TARGET_VERSION> \
  --rolling \
  --log_dir auto

Key Parameters:

  • -dbnodes: The file containing the list of target nodes.
  • -dbnode_precheck: Initiates the prerequisite check.
  • -target_version: The specific Exadata software version you are upgrading to.
  • --rolling: Ensures nodes are patched sequentially to maintain availability.
  • --log_dir auto: Automatically creates a unique log directory for this session.

Action: Review the output carefully. If there are errors or warnings, resolve them before moving to the upgrade phase. You can use the --modify_at_prereq flag to allow the removal of conflicting RPMs if necessary.

Phase 3: Executing the Upgrade

Once the pre-checks pass, you are ready to apply the patch.

./patchmgr -dbnodes dbnode_list.txt \
  -dbnode_upgrade \
  -dbnode_loc /u01/staging/exadata_patch \
  -target_version <TARGET_VERSION> \
  --rolling \
  --log_dir auto

What Happens During the Upgrade?

  1. Backup: patchmgr automatically creates a backup of the root filesystem for rollback purposes.
  2. Service Management: It gracefully stops databases and Grid Infrastructure on the target node.
  3. Installation: It uses yum (for Oracle Linux 7) or dnf (for Oracle Linux 8+) to install the new RPMs.
  4. Firmware Updates: Any required firmware updates are applied.
  5. Reboot: The node reboots automatically. Do not interrupt this process.
  6. Validation: After reboot, patchmgr verifies that all services come up correctly.
  7. Next Node: Once the current node is healthy, it moves to the next one in the list.

⚠️ Critical Warning: Database nodes will reboot during the update. Do not manually reboot nodes or interrupt the patchmgr session.

Phase 4: Patching the Driving Node

Since patchmgr cannot patch the node it is running on, the driving node remains on the old version after the above step. To bring it to the same level, use dbnodeupdate.sh locally on that node.

# Log in to the driving node
cd /u01/staging/exadata_patch
./dbnodeupdate.sh -u \
  -l /u01/staging/exadata_patch \
  -t <TARGET_VERSION>

This ensures your entire Exadata rack is running the same software version.

Phase 5: Post-Patching Validation

After the upgrade is complete, perform these verification steps to ensure system stability.

1. Verify Software Version

Check the Exadata image version on all nodes:

imageinfo -ver

2. Run Post-Update Exachk

exachk -nodes all -post

Address any new findings reported by Exachk.

3. Validate Cluster Health

Ensure Oracle Clusterware and ASM are functioning correctly:

crsctl check crs
crsctl stat resource -t
asmcmd lsdg
srvctl status database -d <DB_NAME>

4. Review Logs

Check the patchmgr logs for any warnings or errors that might need attention later.

ls -la /u01/staging/exadata_patch/log_*/
cat /u01/staging/exadata_patch/log_*/patchmgr.log

Rollback Procedure

If something goes wrong, don’t panic. patchmgr supports rollback using the automatic backup created during the upgrade.

./patchmgr -dbnodes dbnode_list.txt \
  -rollback \
  --rolling \
  --log_dir auto

For a single node, you can use:

./dbnodeupdate.sh -r

Best Practices & Tips

  • Maintenance Window: Always schedule patching during a defined maintenance window.
  • Backups: Ensure you have recent RMAN backups of your databases before starting.
  • Documentation: Keep MOS Note 1553103.1 open for reference during the process.
  • Testing: If possible, test the patching procedure in a non-production environment first.
  • Monitoring: Keep an eye on the logs in real-time using tail -f to catch issues early.

Conclusion

Patching Exadata Database Servers doesn’t have to be a daunting task. By leveraging the patchmgr utility, you can automate complex steps, reduce human error, and ensure a consistent update across your infrastructure. Remember, preparation is 90% of the battle—run your pre-checks, validate your environment, and let the tool do the heavy lifting.

Have you experienced any challenges while patching Exadata? Share your thoughts in the comments below!

Tags: #OracleExadata #DatabasePatching #LinuxAdmin #OracleDBA #SysAdmin #TechBlog

Leave a Reply