Linux Kernel Settings for Consolidating Databases on Exadata

Introduction

When you consolidate many databases on a single Exadata system, the default Linux kernel settings are rarely enough. Oracle’s consolidation guidance recommends the following:

• Set the number of shared memory segments (SHMMNI) greater than the number of databases.
• Set the maximum shared memory segment size (SHMMAX) to 85 percent of physical memory size.
• Set the maximum total number of system semaphores (SEMMNS) greater than the sum of all database processes.
• Set the maximum number of semaphores in a semaphore set (SEMMSL) greater than the largest number of processes in any single database.

The second recommendation raises a question I hear often: if one shared memory segment can be 85% of physical memory, what is left for the other database instances? This post answers that question and walks through all of these parameters.

1. The Recommendations at a Glance

2. The SHMMAX Question: Does One Segment Take 85% of the Memory?

No. SHMMAX does not limit the total shared memory on the system. It limits the size of a single shared memory segment. Every instance can have its own segment of up to 85% of RAM. It is a ceiling on one segment, not a slice that the first database claims.

Setting SHMMAX does not reserve or allocate anything. A segment is created only when an instance starts, and it takes only the size of that instance’s SGA.

What actually limits the total

When you consolidate databases, the total is governed by SHMALL and by physical RAM, not by SHMMAX. The kernel refuses to create a segment only if:

• the segment is larger than SHMMAX,
• the number of segments would exceed SHMMNI, or
• the total of all segments would exceed SHMALL.

A worked example

Take a server with 1 TB of RAM:

• SHMMAX at about 85% of RAM is roughly 870 GB, so no single segment can be larger than that.
• Ten databases with 50 GB SGAs each create ten 50 GB segments. That is perfectly fine, because each one is far below SHMMAX.
• SHMALL of 1 TB / 4096 = 268435456 pages (with 4 KB pages) allows up to the full RAM size in total.

3. Why 85% Is a “No Surprises” Ceiling

Setting SHMMAX high means you never fail to start an instance (for example with ORA-27123) after someone raises an SGA. It is deliberately generous because it is only an upper bound.

Your real protection against overcommitting memory is capacity planning and a few guardrails:

1. Size the sum of all SGAs. Add up the SGA and PGA of every database, plus ASM and OS needs, and keep the total well below physical memory.
2. Use HugePages. The pool must cover the sum of all SGAs. With USE_LARGE_PAGES=ONLY, an instance will not start if it cannot get enough HugePages. That is a hard guardrail against overcommitting.
3. Set SHMALL correctly. It should be at least total physical memory divided by the page size.

4. One Segment per Database? Not Always

It is natural to assume that one shared memory segment equals one database instance’s SGA, and that is often true. But an SGA can be split into several segments, for example when it is larger than SHMMAX, in NUMA-aware configurations, or depending on the Oracle version and settings.

So don’t size SHMMNI as exactly one per database. Keeping it greater than the number of databases leaves headroom, and ASM and other instances use segments too.

5. Semaphores: SEMMNS and SEMMSL

Oracle uses semaphores to coordinate its processes, and the number needed is driven by the PROCESSES setting of each instance. On a consolidated system, two limits matter most:

• SEMMSL must be greater than the largest number of processes in any single database, because one instance needs its semaphores from sets of this size.
• SEMMNS must be greater than the sum of the processes of all databases on the server, because it caps the semaphores system-wide.

A simple way to get the numbers is to sum the PROCESSES parameter across all instances on the node, and note the largest single value:

-- Run in each database on the node
SELECT value AS processes FROM v$parameter WHERE name = 'processes';

Remember that SEMMNS cannot usefully exceed SEMMSL multiplied by SEMMNI (the number of semaphore sets), so check all three values together.

6. Example Settings

The values below are illustrations for a hypothetical 1 TB server. Calculate your own from the formulas above, and validate them against the Oracle documentation and Exadata guidance for your software versions.

# /etc/sysctl.conf (illustrative values for a 1 TB server)
# One segment may be up to ~85% of RAM (bytes)
kernel.shmmax = 934584883609
# Total shared memory in pages (1 TB / 4096)
kernel.shmall = 268435456
# More segments than databases (leave headroom)
kernel.shmmni = 4096
# semmsl  semmns    semopm  semmni
kernel.sem = 2048 1000000 100 4096

Apply the changes without a reboot:

sysctl -p

7. Quick Checks

ipcs -m                          # current segments and their sizes
ipcs -lm                         # shmmax / shmall / shmmni limits in effect
ipcs -ls                         # semaphore limits in effect
sysctl kernel.shmmax kernel.shmall kernel.shmmni kernel.sem
grep -i huge /proc/meminfo       # HugePages total, free, reserved

If you add up the segment sizes from ipcs -m and compare the sum to SHMALL (converted to bytes) and to physical RAM, you can see how much consolidation headroom you have left.

8. Best Practices Checklist

1. Set SHMMAX to about 85% of physical memory, and remember it limits one segment, not the total.
2. Set SHMALL to at least physical memory divided by the page size.
3. Keep SHMMNI above the number of databases, with headroom for multi-segment SGAs, ASM and other instances.
4. Set SEMMSL above the largest PROCESSES value of any single database.
5. Set SEMMNS above the sum of PROCESSES across all databases, and check it against SEMMSL multiplied by SEMMNI.
6. Configure HugePages for the sum of all SGAs, and consider USE_LARGE_PAGES=ONLY as a guardrail.
7. Recheck these values every time you add a database or resize an SGA.

Conclusion

The 85% recommendation for SHMMAX looks alarming until you remember what it means: a per-segment ceiling, not a per-system allocation. On a consolidated Exadata node, the parameters that really enforce your memory budget are SHMALL, HugePages and your own capacity planning. How many databases are you consolidating per node? Share your experience in the comments.

Disclaimer: Always test kernel parameter changes in a non-production environment first, and verify the settings against the Oracle documentation for your specific Exadata System Software and database versions.