Introduction

If there is one memory setting that Oracle repeatedly stresses on Exadata, it is HugePages. On current Exadata software, it is no longer just a tuning tip: large pages are mandatory for the SGA. This post is split in two parts. Part 1 explains the concept, so you understand why HugePages matter. Part 2 walks through the implementation, so you can calculate, configure and verify them on your own system.

Part 1: The Concept

1. How Linux Organizes Memory

Linux manages physical memory in blocks called pages. On x86 systems the default page size is 4 KB. Every page needs an entry in the page tables, and the CPU’s memory management unit (MMU) caches recent translations in a small structure called the TLB (translation lookaside buffer).

That works well for small systems, but it scales badly. A server with 1 TB of RAM has more than 268 million 4 KB pages. Managing that many page table entries adds overhead, and the TLB can only hold a tiny fraction of the translations, so misses become frequent.

2. What HugePages Change

HugePages are a Linux kernel feature that lets the operating system use much larger pages. On x86 Linux for Oracle, the recommended HugePage size is 2 MB. With 2 MB pages, the same 1 TB of RAM needs only about 524 thousand pages.

The practical benefits for the Oracle SGA are:

• Fewer page table entries, so less memory is spent on page tables and less work is done by the MMU.
• Fewer TLB misses, because each TLB entry now covers 2 MB instead of 4 KB.
• Memory that is locked and not swappable, so the SGA stays in RAM.
• Lower and more predictable overhead as the number of processes attached to the SGA grows, which matters on consolidated systems.

Making all memory use large pages would waste memory for applications that work fine with 4 KB pages. The solution is to reserve a pool of HugePages for the big consumer, the SGA, and leave regular 4 KB pages for everything else.

3. HugePages vs Transparent HugePages (THP)

These two similar names describe different mechanisms, and confusing them is a common mistake.

4. Oracle’s Recommendations for Exadata

Oracle’s guidance for Exadata can be summarized in a few points:

1. Always use large pages for the SGA, and configure the HugePages in advance, before the databases start.
2. Use a 2 MB HugePage size for the SGA.
3. Set USE_LARGE_PAGES=ONLY for every database. This is the Maximum Availability Architecture (MAA) recommended setting for Exadata.
4. Size the HugePages pool for the aggregate of all SGAs on the server, not for a single database.
5. Disable Transparent HugePages.
6. Do not use Automatic Memory Management (AMM), because it is not compatible with HugePages. Use SGA_TARGET and PGA_AGGREGATE_TARGET instead.

Oracle’s automation follows the same rules. Databases created with the Oracle Exadata Deployment Assistant (OEDA), and databases created on Exadata cloud services, use USE_LARGE_PAGES=ONLY, and the ExaChk utility flags any database that does not.

An important change: small pages are no longer an option

Starting with Oracle Database Release Updates 19.27 and 23.8, Oracle states that small pages are not available on Exadata systems, and large pages must be used. In practice, the USE_LARGE_PAGES values that permit 4 KB pages are no longer available there. See MOS Doc ID 3081878.1 for the details of your release.

5. The USE_LARGE_PAGES Parameter

The USE_LARGE_PAGES initialization parameter controls how an instance uses HugePages:

Why would you want an instance to fail to start? Because a database that starts with a sub-optimal memory configuration can be far worse: you get unpredictable performance that is hard to diagnose. A clear failure at startup, with a message in the alert log, is the better trade.

Part 2: The Implementation

Step 1: Inventory the memory you need

The number of HugePages is driven by the sum of all SGAs that will run on the server, including the ASM instance and any databases you plan to add. Collect the SGA sizes of every instance on each database server:

-- Run in each database on the node
SELECT name, value/1024/1024 AS size_mb
FROM   v$parameter
WHERE  name IN ('sga_target','sga_max_size','memory_target');

If MEMORY_TARGET is set, the database is using AMM. Move it to SGA_TARGET and PGA_AGGREGATE_TARGET before you proceed.

Step 2: Do the arithmetic

With 2 MB pages, the formula is:

HugePages needed = (sum of all SGAs in MB) / 2  + a small safety margin

For example, a single 50 GB SGA (51200 MB) needs 51200 / 2 = 25600 HugePages. Oracle advises adding a few extra pages to cover byte offsets and page alignment.

For a consolidated node running ten databases with 50 GB SGAs each, plus ASM:

• Ten SGAs: 500 GB = 512000 MB, which is 256000 pages.
• Add a margin of 1 to 2 percent as a starting point for ASM, alignment and growth (a rule of thumb, not an Oracle figure), which gives roughly 261000 pages.
• That reserves about 510 GB of RAM for HugePages (261000 x 2 MB).

Step 3: Let Oracle’s script confirm the number

Oracle provides a script, hugepages_settings.sh, in My Oracle Support Doc ID 401749.1. It reads the shared memory segments of the running instances and recommends a vm.nr_hugepages value. Start all instances, including ASM, before you run it:

./hugepages_settings.sh
# Recommended setting: vm.nr_hugepages = 261200

Keep in mind that the script only sees the instances that are running at that moment, so check the result against your own inventory from Step 1.

Step 4: Disable Transparent HugePages

Check the current THP state:

cat /sys/kernel/mm/transparent_hugepage/enabled
grep AnonHugePages /proc/meminfo

If THP is enabled (the value in brackets is not never) or AnonHugePages shows a value greater than 0 kB, disable it with the kernel boot parameter transparent_hugepage=never. The Unbreakable Enterprise Kernel (UEK) typically has THP disabled already, but verify it rather than assume.

Step 5: Set vm.nr_hugepages

As root, add the value to /etc/sysctl.conf (or a file under /etc/sysctl.d/ on Oracle Linux 7 and later) and apply it:

# /etc/sysctl.conf
vm.nr_hugepages = 261200

sysctl -p

Do this on every database server of the cluster, sized for the instances that run on each one. If the kernel cannot allocate all the pages because memory is fragmented, the value will not fully take effect and you will need to reboot the server. A reboot is also the most reliable way to get a clean allocation on a busy system. On Exadata, perform it as a rolling change across the RAC nodes.

If you run Exadata with virtual machines, plan the HugePages per VM, against the memory assigned to that VM.

Step 6: Check the memory locking limits

The Oracle software owner must be allowed to lock enough memory. Verify the memlock limits in /etc/security/limits.conf (or your Exadata-managed equivalent). Oracle’s installation guides suggest a value of around 90 percent of RAM, and in any case it must be at least as large as the HugePages pool. Exadata deployments normally configure this for you, so confirm it instead of overwriting it.

Step 7: Set USE_LARGE_PAGES=ONLY

Set the parameter in the spfile and restart the instance:

ALTER SYSTEM SET use_large_pages = ONLY SCOPE = SPFILE SID = '*';
-- then restart the database, for example:
-- srvctl stop database -d MYDB
-- srvctl start database -d MYDB

Step 8: Verify

First check the system-wide picture:

grep -i huge /proc/meminfo
# HugePages_Total:  total pages configured
# HugePages_Free:   pages not yet allocated (includes reserved ones)
# HugePages_Rsvd:   pages reserved by instances but not yet touched
# Hugepagesize:     2048 kB
# AnonHugePages:    should be 0 kB when THP is disabled

Then check each instance. At startup, the database writes a Large Pages Information section to its alert log. It shows the value of use_large_pages, how much of the shared global area is in large pages, and how many pages the instance used. You want to see 100 percent of the SGA in large pages.

You can also compare the shared memory segments with the HugePages pool:

ipcs -m

Step 9: Keep it in step with change

HugePages are not a set-and-forget setting. Revisit them whenever you:

• add a database or an instance to a server,
• increase an SGA,
• move instances between nodes, which changes what each node must hold after a failover.

For failover, remember that a node must have enough HugePages for the instances it may have to host, not just the ones running on it today.

Troubleshooting

Be careful not to over-allocate. Each HugePage is 2 MB, so you can multiply vm.nr_hugepages by 2 MB to see how much memory is taken out of the general pool. Allocating far too many is as harmful as allocating too few.

How This Relates to the Shared Memory Parameters

HugePages do not replace the kernel shared memory limits discussed in the previous post on consolidating databases. SHMMAX, SHMALL and SHMMNI still apply to the SGA segments, whether they sit in HugePages or not. Think of it this way: the shared memory parameters decide whether a segment may be created, and HugePages decide what kind of memory backs it.

Checklist

1. Inventory all SGAs on each server, including ASM and planned databases.
2. Move any database using AMM to SGA_TARGET and PGA_AGGREGATE_TARGET.
3. Calculate: sum of SGAs in MB divided by 2, plus a margin, and confirm with hugepages_settings.sh (MOS 401749.1).
4. Disable Transparent HugePages and confirm AnonHugePages is 0 kB.
5. Set vm.nr_hugepages on every database server and reboot (rolling) if the allocation is incomplete.
6. Confirm the memlock limits cover the pool.
7. Set USE_LARGE_PAGES=ONLY on all databases.
8. Check the alert log for 100 percent of the SGA in large pages.
9. Recalculate whenever SGAs, databases or node placement change.

Conclusion

On Exadata, HugePages are the foundation of a stable, high-performance SGA, and with recent release updates they are mandatory. The concept is simple: fewer, larger pages mean less memory-management overhead and a locked SGA. The implementation is mostly careful arithmetic plus discipline: size the pool for the sum of all SGAs, disable Transparent HugePages, use USE_LARGE_PAGES=ONLY, and recheck whenever the workload changes. How do you track HugePages in your consolidated environments? Let me know in the comments.

Disclaimer: Always test memory configuration changes in a non-production environment first, and verify the settings against the Oracle documentation and My Oracle Support notes for your specific Exadata System Software and database versions.