OnCallReadyStart learning

Memory & OOM: glossary

72 terms, in plain words, with where each one is taught.

/proc/meminfo (meminfo)
The kernel's memory report: one "Name: value kB" line per figure. free, top and vmstat all read it. Lesson 5.3
128 + signal (exit 143, exit 130, 128 + N)
How a death by signal becomes an exit code: 128 plus the signal number. 137 = 128 + 9 (SIGKILL), 143 = 128 + 15 (SIGTERM), 130 = 128 + 2 (Ctrl+C). Lesson 5.11
allocation order (order=0)
How many pages in a row the failed request wanted, as a power of two: order=0 is one page, so memory was truly exhausted, not just fragmented. Lesson 5.8
anonymous memory (RssAnon, anon, AnonPages)
Memory with no file behind it: the heap, stacks, anything allocated with malloc. Its only copy is in RAM, so the kernel can reclaim it only by swapping it out or killing the process. grep RssAnon /proc/<pid>/status for one process; AnonPages in /proc/meminfo for all of them. Lesson 5.1
badness (oom_badness)
The kernel's name for a process's OOM score: resident pages + swapped pages + page tables, plus oom_score_adj scaled to total memory. Highest badness is killed. Lesson 5.8
buff/cache (Buffers, Cached)
The free column that holds the page cache (cache) plus the disk's own bookkeeping (buffers). Counted as used, but mostly given back the moment programs need memory. Lesson 5.3
cgroup OOM (memcg OOM, CONSTRAINT_MEMCG, oom_memcg)
An OOM where one cgroup hit its own memory limit while the machine may have had plenty free. Only processes inside that cgroup can be killed. The log says CONSTRAINT_MEMCG. Lesson 5.7
CommitLimit (Committed_AS)
In /proc/meminfo: Committed_AS is how much memory has been promised in total; CommitLimit is the ceiling, enforced only in strict overcommit mode 2. Lesson 5.3
committed memory (reserved memory)
In JVM reports, reserved is address space set aside (like VSZ) and committed is memory actually backed and usable. Committed is what counts against a limit. Lesson 5.13
dentry (dentries)
The kernel's cached entry for a file name it has looked up, so the next lookup of that path is fast. Kept in the slab. Lesson 5.3
direct buffers (direct memory, MaxDirectMemorySize, NIO buffers)
Memory a Java program asks for outside the heap, usually to pass data to the network or disk quickly. By default it may grow as large as the heap. Lesson 5.13
drop_caches (/proc/sys/vm/drop_caches)
A kernel file: writing 1, 2 or 3 into it throws away the page cache, the dentry and inode caches, or both. A demonstration, never a fix. Run sync first to write dirty pages out, and use echo 3 | sudo tee ... because the redirection must run as root. Lesson 5.3
ExitOnOutOfMemoryError (-XX:+ExitOnOutOfMemoryError)
A JVM option that makes it exit on the first OutOfMemoryError instead of limping on, so systemd's Restart= can start a clean copy. Lesson 5.13
file-backed memory (RssFile, file-backed, rss_file)
Pages that are a copy of a file on disk, such as the program itself and its libraries. The kernel can drop them and read them back from the file later. Lesson 5.1
free memory (MemFree)
RAM doing nothing at all. Low is normal and good on Linux, because idle RAM is used as page cache. Lesson 5.3
garbage collector (GC, garbage collection)
The part of the JVM that walks the heap from time to time and frees objects nothing uses any more. Java programs never free memory themselves. Lesson 5.13
gfp_mask (GFP_KERNEL, GFP_HIGHUSER_MOVABLE)
In an OOM report, flags describing the failed memory request. GFP_HIGHUSER_MOVABLE is ordinary program memory; GFP_KERNEL is the kernel allocating for itself. Lesson 5.8
glibc
The GNU C library: the implementation of libc on Ubuntu. It provides malloc and most of what programs call into. Lesson 5.1
HeapDumpOnOutOfMemoryError (-XX:+HeapDumpOnOutOfMemoryError)
A file with the entire contents of the heap, written for later analysis. As big as the heap itself. Lesson 5.13
host OOM (global OOM, global_oom, CONSTRAINT_NONE)
An OOM where the whole machine ran out of memory. Every process is a candidate. The kernel log says constraint=CONSTRAINT_NONE and global_oom. Lesson 5.7
InitialRAMPercentage (-Xms)
The JVM's starting heap, as a percentage of its memory. -Xms is the same as a fixed size. Setting it equal to the maximum makes the full footprint appear at start-up. Lesson 5.13
Java class
The blueprint for one kind of object in Java code. The JVM keeps the description of every loaded class in metaspace. Lesson 5.13
jcmd
A JDK tool that sends commands to a running JVM, for example jcmd <pid> VM.native_memory summary. Lesson 5.13
JDK (JDK 10, Java version)
The Java Development Kit: a Java release, and the full install that includes tools like jcmd. "JDK 10" means Java version 10. Lesson 5.13
JIT (code cache, ReservedCodeCacheSize, just in time)
The JVM compiling busy Java code into machine code while it runs ("just in time"). The result lives in the code cache, outside the heap. Lesson 5.13
kernel ring buffer (ring buffer)
A fixed-size log inside the kernel's memory where the kernel writes its messages; the newest overwrite the oldest. dmesg and journalctl -k read it. Lesson 5.7
KiB (kB)
A kibibyte: 1024 bytes. Linux tools often write "kB" but mean KiB. ps shows RSS and VSZ in KiB. Lesson 5.1
malloc (malloc arena)
The standard "give me memory" call programs use. glibc keeps a pool of memory per thread (an arena) to serve these calls quickly, which inflates VSZ. Lesson 5.1
MaxRAMPercentage (-XX:MaxRAMPercentage)
A JVM option: with no -Xmx, the maximum heap is this percentage of the memory the JVM sees (its cgroup limit). Default 25; 70-80 is typical for a service. Lesson 5.13
MemAvailable
The kernel's estimate of how much memory a new program could get right now, counting the cache it would throw away to make room. The number to judge and alert on. It is MemAvailable in /proc/meminfo; free and top just print it. Lesson 5.3
memcg
Short for memory cgroup: a cgroup seen from its memory side, with its limit and usage. Lesson 5.7
memhog
A simulator tool that allocates as much memory as you ask for. On a real box, stress-ng --vm 1 --vm-bytes 400M does the same job. Lesson 5.11
memory page (4 KiB page, pages RAM)
The unit the kernel hands out memory in: a fixed-size chunk, 4 KiB on this box. Memory numbers are counts of pages converted to KiB or MiB. The OOM report's task table is in pages: multiply by 4 for KiB, divide by 256 for MiB. Lesson 5.1
memory.events (oom_kill)
A cgroup file of counters: how often it went over memory.high, hit memory.max, and how many processes were OOM-killed (oom_kill). Proof that does not depend on any log. Lesson 5.11
memory.high
The cgroup file behind MemoryHigh=: a soft limit. Above it the kernel slows the cgroup and reclaims its memory, but never kills. Lesson 5.11
memory.peak
A cgroup file: the highest memory use, in bytes, since the cgroup was created. Lesson 5.11
memory.stat
A cgroup file breaking its memory into kinds: anon, file, kernel, shmem and more, in bytes. Lesson 5.11
metaspace (MaxMetaspaceSize)
The JVM's memory for the descriptions of loaded classes. Outside the heap and unlimited by default; more code means more metaspace. Lesson 5.13
MiB (GiB)
A mebibyte: 1024 KiB. A GiB is 1024 MiB. free -m prints MiB, free -h picks Mi or Gi for you. Lesson 5.1
Native Memory Tracking (NMT, NativeMemoryTracking)
A JVM option that makes the JVM account for its own memory by region (heap, class, thread, code, GC...), which jcmd can then print. Lesson 5.13
non-heap memory (non-heap, off-heap)
Everything a JVM uses besides the heap: metaspace, thread stacks, code cache, garbage collector structures, direct buffers and native libraries. Typically a few hundred MB. Lesson 5.13
OOM invoker (invoked oom-killer)
The process whose memory request failed and so started the OOM killer. Named in the report's first line; often not the one killed, and not necessarily the one leaking. Lesson 5.8
oom_reaper (reaped process)
A kernel helper that frees a killed process's memory immediately, even if the process is stuck and cannot finish exiting. Lesson 5.8
oom_score (/proc/<pid>/oom_score)
A process's current score for the OOM killer, 0-1000 plus its adjustment, roughly its share of RAM and swap. The highest is killed first. Lesson 5.7
oom_score_adj (OOMScoreAdjust, oom adj)
A per-process adjustment from -1000 to +1000 added to its OOM score. -1000 means never kill it, +1000 means kill it first. OOMScoreAdjust= sets it in a systemd unit. Use -1000 for sshd so you can still log in; about -500 for important services, never -1000 on something large that might leak. Lesson 5.7
oom-kill result (Result: oom-kill, result 'oom-kill')
What systemd reports when the OOM killer killed a process of a unit: "Failed with result 'oom-kill'", with status=9/KILL. Lesson 5.11
OutOfMemoryError (java.lang.OutOfMemoryError, Java heap space)
The JVM's own error when one of its regions (heap, metaspace, direct memory) is full. It is written to the application log with a stack trace, unlike a kernel OOM kill. Lesson 5.13
overcommit (vm.overcommit_memory, overcommit_memory)
Linux promising programs more memory than it has, betting they will not touch all of it. malloc succeeds; if the bet fails, the OOM killer comes later instead of an error. Lesson 5.3
page tables (pgtables, pgtables_bytes)
The kernel's map from each process's addresses to real RAM. They take memory too, which belongs to no process's RSS. Lesson 5.1
Private_Dirty (Shared_Clean)
In smaps_rollup: memory that belongs to this process alone and was modified. It can only go to swap or disappear with the process. Shared_Clean is shared and unmodified. Lesson 5.1
PSI (pressure stall information, /proc/pressure/memory, memory pressure)
Pressure stall information: the kernel's report of how much of the time processes were stuck waiting for memory (or CPU, or I/O), in /proc/pressure/. Lesson 5.7
PSS (proportional set size, Pss)
Proportional set size: RSS with every shared page divided by the number of processes sharing it. The only per-process memory number you can add up across processes. Read it from /proc/<pid>/smaps_rollup, or per process with the smem package. Lesson 5.1
resident memory (mapped memory)
Memory a process has actually touched, so it sits in physical RAM right now. Mapped memory is only reserved: the address exists, but nothing is in RAM until the process uses it. Lesson 5.1
root cgroup
The top of the cgroup tree, /sys/fs/cgroup itself: the whole machine. It has no memory limit, so memory.max does not exist there. Lesson 5.11
shared library (libc, shared libraries)
Code many programs use, loaded into RAM once and mapped into every process that needs it. libc, the C standard library, is used by almost every program. Because each process counts a shared library in its own RSS, adding up RSS over-counts. Lesson 5.1
shared memory (shmem, RssShmem, shmem-rss)
Memory several processes use together on purpose, plus files in tmpfs. It counts under shared and buff/cache in free, but it cannot be dropped like cache. Lesson 5.1
SHR (shared (RES))
In top, the part of a process's resident memory that is shared with other processes, such as libraries every program uses. Lesson 5.1
si/so (swap-in, swap-out, swpd)
vmstat columns: KiB per second swapped in from disk and out to disk. Sustained non-zero values together mean thrashing. swpd is swap used, which is harmless on its own. Lesson 5.3
slab (SReclaimable, SUnreclaim, slab cache)
The kernel's own caches of small objects, such as file names it has looked up. SReclaimable can be freed under pressure; a huge SUnreclaim means a leak inside the kernel. Lesson 5.1
smaps_rollup (smaps)
A /proc/<pid>/ file with a process's memory totals, including PSS and Private_Dirty. smaps is the long version with one block per mapping. Lesson 5.1
smem
A small tool (a package) that prints PSS per process in a table, so shared memory is counted fairly. Lesson 5.1
stack trace
The list of functions a program was in when an error happened, printed with the error. A Java OutOfMemoryError leaves one; a kernel OOM kill leaves none. Lesson 5.13
swapents
In the OOM report's task table: how many of the process's pages were in swap at that moment. Lesson 5.8
swappiness (vm.swappiness)
A kernel setting (0-200, default 60) for how eagerly the kernel swaps process memory out instead of dropping page cache. Lower keeps programs in RAM. It does not turn swap on or off. Lesson 5.3
systemd-oomd
A daemon that watches memory pressure and kills a whole cgroup before the kernel has to. It logs to its own unit, not to the kernel log. Lesson 5.7
task_memcg
In the oom-kill line: the cgroup the killed process lived in, such as /system.slice/cron.service. It maps a kill back to a service. Lesson 5.8
Tasks state table (Tasks state)
The part of an OOM report listing every candidate process with its sizes in pages, its swap use and its oom_score_adj. Lesson 5.8
thrashing (swap churn)
When what the processes need does not fit in RAM, so the kernel keeps moving pages to swap and straight back. Everything crawls; vmstat shows si and so non-zero all the time. Lesson 5.3
thread stack (stack per thread, -Xss)
A thread's scratch space for the functions it is in the middle of. Every thread has one; a server with 200 threads has 200 stacks. Lesson 5.1
transient scope (run-*.scope)
A temporary cgroup systemd creates for a command run with systemd-run --scope. It disappears when the command exits. Lesson 5.11
UseContainerSupport
A JVM setting, on by default since JDK 10: the JVM reads its cgroup memory limit and treats it as the machine's RAM when sizing itself. Lesson 5.13
VmHWM (high water mark, VmPeak)
The "high water mark" line in /proc/<pid>/status: the largest RSS the process has had since it started. VmPeak is the same for VSZ. Useful after an incident: it still shows how high memory went after it has come back down. Lesson 5.1