The Definitive Guide to Patching Oracle Exadata Storage Cells: End-to-End Procedure, Best Practices, and Troubleshooting

Introduction

Patching an Oracle Exadata Database Machine is a critical maintenance task that ensures security, stability, and performance. Unlike traditional infrastructure, Exadata patches are “Full Stack,” meaning they update the Oracle Linux operating system, firmware (Flash, Disk, RAID controller, ILOM), and the Oracle Exadata System Software simultaneously.

The good news? Most Exadata Storage Server patches can be applied while databases remain fully functional. This is achieved through Rolling Patching, where cells are updated one at a time, leveraging Oracle ASM’s redundancy to keep I/O flowing.

However, complexity brings risk. This guide merges strategic planning with deep-dive technical execution to provide you with a complete, step-by-step procedure for patching Exadata storage cells safely and efficiently.

Part 1: Arrangements & Strategic Planning

Success begins long before you run the first command. Proper planning minimizes downtime and prevents unexpected issues.

1. Choose Your Patching Strategy

Rolling Patch (Recommended): Updates one cell at a time. Databases remain online. Ideal for production environments.
Non-Rolling Patch: Updates all cells simultaneously. Requires database downtime. Only used when rolling is not feasible (e.g., specific firmware constraints).

2. Estimate Maintenance Window

Timeframe: A full rack update can take 26–40 hours, depending on data volume, resync requirements, and patch complexity.
Buffer: Always add a 20–30% buffer to your estimated time for troubleshooting.

3. Team & Communication

Coordinate with DBAs, SysAdmins, and Network teams.
Create an OEM Blackout for all cell servers to prevent false alerts.
Disable AMR (Automatic Memory Resizing) monitoring during the window.
Have Oracle Support contact details ready for Sev 1 escalation.

Part 2: Prerequisites & Pre-Checks

Skipping prerequisites is the #1 cause of patching failures. Verify every item below.

1. SSH Equivalence

Ensure passwordless SSH from the driving node (usually DB Node 1) to all Cell Servers.

			
# Create cell group file
ibhosts | sed s'/"//' | grep cel | awk '{print $6}' | sort > /root/cell_group
# Test connectivity
dcli -g ~/cell_group -l root hostname

2. Download & Stage the Patch

Download the latest Quarterly Full Stack Download bundle from My Oracle Support (MOS Doc ID 888828.1).

			
# Create directory
mkdir -p /patches/Exadata_Patch_2024
cd /patches/Exadata_Patch_2024
# Unzip all parts
for I in `ls p*_Linux-x86-64*f*.zip`
do
  unzip $I
done
# Extract tar files
cat *.tar.* | tar -xvf -

		

Ensure ownership is correct (root or oracle:dba).

3. Upgrade OPatch

Always upgrade OPatch to the latest version before starting, especially for rolling patches.

4. ASM Repair Timers (Critical)

Oracle ASM must tolerate a cell being offline for the duration of the patch + reboot.

Check Current Values:

			
SQL> select dg.name as diskgroup, a.name as attribute, a.value 
     from v$asm_diskgroup dg, v$asm_attribute a 
     where dg.group_number=a.group_number 
     and (a.name like '%repair_time' or a.name = 'compatible.asm');

Adjust if Necessary:
If disk_repair_time or failgroup_repair_time is < 8.5 hours, increase it to 24h.

			
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'disk_repair_time' = '24h';
SQL> ALTER DISKGROUP RECO SET ATTRIBUTE 'failgroup_repair_time' = '24h';

Note: Disk groups with compatible.asm >= 12.1.0.2.0 use failgroup_repair_time.

5. Health Checks

Run Exachk: Download and run the latest Exachk (MOS Doc ID KB74841). Resolve all open issues.
Check ASM Rebalances: Ensure no active rebalances are running.SQL> select * from gv$asm_operation; -- Must return no rows
ASM Power Limit: Set to at least 4 to speed up rebalancing after patching.SQL> alter system set ASM_POWER_LIMIT = 4 scope=both;

6. Review Known Issues

Check the release-specific MOS note for known issues and workarounds for your target patch version.

Part 3: Pre-Patching Preparation

1. Backup Critical Files

Back up the IMSM dracut rules file on each cell. This is your safety net if the node boots into dracut emergency mode.

			
dcli -g ~/cell_group -l root "cp /etc/dracut.conf.d/99-imsm.conf /root/99-imsm.conf.bak"

2. Collect Baseline Diagnostics

Capture the current state for comparison and troubleshooting.

			
# Image versions
dcli -l root -g cell_group 'imageinfo -version'
dcli -l root -g cell_group 'imageinfo -status'
dcli -l root -g cell_group 'uname -r'
# Clear old alerts
dcli -l root -g cell_group 'cellcli -e clear alert history'
# Generate SOS/Sundiag reports
# (Run sundiag.sh on all cells)

		

3. Validate Disk Health

Ensure all disks are healthy before starting.

			
dcli -l root -g cell_group 'cellcli -e list physicaldisk attributes name, status'
dcli -l root -g cell_group 'cellcli -e list griddisk attributes name, asmmodestatus'

Resolve any abnormal statuses before proceeding.

4. Clean Up Previous Patch Runs

			
cd <patch_directory>
# For first-time updates
./patchmgr -cells ~/cell_group –reset_force
# For subsequent runs
./patchmgr -cells ~/cell_group -cleanup

		

5. Run Prerequisite Checks

This is the final gatekeeper. Do not proceed if this fails.

			
./patchmgr –cells ~/cell_group -patch_check_prereq -rolling \
-smtp_from "admin@ahmedfattah.com" \
-smtp_to "dba-team@ahmedfattah.com"

Part 4: The Patching Process (Rolling Method)

While patchmgr -rolling automates the sequence, manual oversight per cell is recommended for maximum control.

Step 1: Prepare the Target Cell

For each cell in the cell_group:

Open ILOM Console: ssh root@<cell-ilom-ip> start /HOST/consoleKeep this terminal open. You need real-time visibility into the boot process.
Inactivate Grid Disks:
This tells ASM to gracefully stop I/O to this cell. ssh root@<target-cell> cellcli -e alter griddisk all inactive exit

Step 2: Apply the Patch

			
cd <patch_directory>
# Patch single cell
nohup ./patchmgr -cells <single_cell_file> -patch -rolling &
# Monitor logs
tail -f patchmgr.stdout

		

What patchmgr does automatically:

Stages software on the cell.
Updates Oracle Linux OS.
Updates Firmware (Flash, Disk, RAID, ILOM).
Reboots the cell.
Validates post-reboot health.

Step 3: Monitor Reboot via ILOM

Watch the ILOM console closely.

Normal: The node reboots and comes back online.
Issue: If the node hangs in dracut emergency mode:
1. Do not wait for timeout.
2. Power cycle via ILOM:# In ILOM CLI reset /SYS
3. If it fails again, restore the dracut backup:cp /root/99-imsm.conf.bak /etc/dracut.conf.d/99-imsm.conf

Step 4: Post-Reboot Verification (Per Cell)

After the cell reboots:

Reactivate Grid Disks: ssh root@<patched-cell> cellcli -e alter griddisk all active exit
Verify Status: dcli -l root -g cell_group 'cellcli -e list griddisk attributes name, asmmodestatus, asmdeactivationoutcome'All disks must be ONLINE/NORMAL.
Confirm Version: dcli -l root -g cell_group 'imageinfo -version' dcli -l root -g cell_group 'uname -r'

CRITICAL: Do not proceed to the next cell until the current cell is fully verified and stable.

Part 5: Post-Patching Steps

1. Final Verification

Verify all cells report the same new image version.
Run Exachk again to validate post-patch compliance.
Check ASM alert logs for any errors.

2. Restore ASM Parameters

If you increased disk_repair_time, set it back to its original value.

			
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'disk_repair_time' = '<original_value>';

3. Collect Post-Patch Reports

Generate new SOS/Sundiag reports.
Capture ILOM snapshots.
Save patchmgr log files.
Upload these to Oracle SR if required.

4. Functional Testing

Verify database operations.
Remove OEM blackouts.
Re-enable AMR monitoring.

Part 6: Rollback Procedure

If a critical issue arises, you can rollback to the previous image.

			
cd <patch_directory>
# 1. Cleanup
./patchmgr -cells ~/cell_group -cleanup
# 2. Pre-check for rollback
./patchmgr -cells ~/cell_group -patch_check_prereq -rolling
# 3. Execute Rollback
./patchmgr -cells ~/cell_group -rollback -rolling

		

Troubleshooting Tips

Cell Not Rebooting: STOP. Do not force the next cell. Check ILOM logs. Open MOS Sev 1.
Grid Disks Not Coming Online: Check ASM alert logs. Verify disk health. Check disk_repair_time.
Dracut Emergency Mode: Restore dracut rules backup and power cycle via ILOM.
Pre-Check Failures: Resolve all errors listed in the log before proceeding. Never ignore pre-check failures.

Conclusion

Patching Exadata Storage Servers is a structured process that, when followed meticulously, ensures minimal disruption and maximum reliability. By combining strategic planning with precise technical execution, you can keep your Exadata Machine secure, up-to-date, and performant.

Key Takeaways:

Plan: Adequate window, team coordination, and backups.
Prepare: SSH, OPatch, ASM timers, and health checks.
Execute: Rolling patches, ILOM monitoring, and grid disk management.
Verify: Comprehensive post-patch checks and reporting.

Have you patched your Exadata recently? Share your experiences or questions in the comments below!

The Definitive Guide to Patching Oracle Exadata Storage Cells: End-to-End Procedure, Best Practices, and Troubleshooting

Introduction

Part 1: Arrangements & Strategic Planning

1. Choose Your Patching Strategy

2. Estimate Maintenance Window

3. Team & Communication

Part 2: Prerequisites & Pre-Checks

1. SSH Equivalence

2. Download & Stage the Patch

3. Upgrade OPatch

4. ASM Repair Timers (Critical)

5. Health Checks

6. Review Known Issues

Part 3: Pre-Patching Preparation

1. Backup Critical Files

2. Collect Baseline Diagnostics

3. Validate Disk Health

4. Clean Up Previous Patch Runs

5. Run Prerequisite Checks

Part 4: The Patching Process (Rolling Method)

Step 1: Prepare the Target Cell

Step 2: Apply the Patch

Step 3: Monitor Reboot via ILOM

Step 4: Post-Reboot Verification (Per Cell)

Part 5: Post-Patching Steps

1. Final Verification

2. Restore ASM Parameters

3. Collect Post-Patch Reports

4. Functional Testing

Part 6: Rollback Procedure

Troubleshooting Tips

Conclusion

Like this:

Leave a Reply Cancel reply

Introduction

Part 1: Arrangements & Strategic Planning

1. Choose Your Patching Strategy

2. Estimate Maintenance Window

3. Team & Communication

Part 2: Prerequisites & Pre-Checks

1. SSH Equivalence

2. Download & Stage the Patch

3. Upgrade OPatch

4. ASM Repair Timers (Critical)

5. Health Checks

6. Review Known Issues

Part 3: Pre-Patching Preparation

1. Backup Critical Files

2. Collect Baseline Diagnostics

3. Validate Disk Health

4. Clean Up Previous Patch Runs

5. Run Prerequisite Checks

Part 4: The Patching Process (Rolling Method)

Step 1: Prepare the Target Cell

Step 2: Apply the Patch

Step 3: Monitor Reboot via ILOM

Step 4: Post-Reboot Verification (Per Cell)

Part 5: Post-Patching Steps

1. Final Verification

2. Restore ASM Parameters

3. Collect Post-Patch Reports

4. Functional Testing

Part 6: Rollback Procedure

Troubleshooting Tips

Conclusion

Share this:

Like this:

Leave a Reply Cancel reply