MAP_PRIVATE vs MAP_SHARED: The mmap Flag That Decides Everything

There is exactly one argument in the mmap call that decides the fate of your writes, and it is not the one people spend time on. Not the length, not the protection bits, not the offset. It is the flag that says MAP_PRIVATE or MAP_SHARED. Get it right and a hundred microVMs restore from one memory file while sharing almost all of their RAM. Get it wrong and either your writes vanish silently, or one tenant scribbles on a template every other tenant is reading. MAP_PRIVATE is the kernel's way of letting you scribble on a library book without anyone finding out; MAP_SHARED is writing in the margin and handing it back. This post is about the difference, precisely — and about the traps that hide behind it, which is where the real bugs live.

The base model: a mapping, not a copy

mmap does not read a file. It installs a virtual memory area — a VMA — that says: this range of addresses is backed by this object, starting at this offset. The object is either a file (file-backed) or nothing at all (anonymous, backed by zero pages on demand). At the moment mmap returns, almost nothing has happened. No bytes were read, no physical frames were allocated, and typically no leaf page-table entries were installed. You have an address range and a promise. The first time you touch a page in that range you take a page fault, the kernel resolves the promise — reading the page into the page cache if it is not already there, or handing you a zero page for anonymous memory — installs a page-table entry, and lets your instruction re-execute. That is true for both flags. Reads are boring and identical: both flags fault a page in from the page cache and both, for a file everyone is reading, land on the same physical frame. What differs — the whole substance of the choice — is what happens on a WRITE.

MAP_SHARED on a file: writes go to the shared page

With MAP_SHARED, your page-table entry points at the page-cache page for that file offset, writable. When you store a byte, you are writing the page cache itself. That page is now dirty, and three consequences follow immediately. Every other process that has the same file region mapped MAP_SHARED sees your byte — not a copy of it, the same physical memory, no synchronization, no syscall. Every process that read()s the file through the ordinary I/O path also sees it, because read() serves out of the same page cache. And the kernel's writeback machinery will, at some point of its own choosing, flush that dirty page to the actual file on disk. So MAP_SHARED is two features wearing one flag: it is the IPC primitive (shared memory between unrelated processes, coordinated only by the file they both opened) and it is the persistence primitive (a memory-speed way to modify a file without write() calls). Databases lean on it. So do shared ring buffers, shared metric counters, and every "map the index file and treat it as a struct" design. Note one hard requirement: MAP_SHARED with PROT_WRITE requires the file descriptor to be opened O_RDWR. The kernel will not let you set up a writeback path through a read-only fd.

MAP_PRIVATE on a file: reads share, the first write forks the page

MAP_PRIVATE gives you a private view of the same object. Reads still ride the shared page cache: your page-table entry points straight at the file's page-cache frame, and so does everybody else's, so a hundred processes reading the same MAP_PRIVATE region occupy one copy of it in physical memory. But those entries are installed read-only, deliberately. The moment you store a byte, the MMU refuses the write and traps. The kernel's fault handler recognizes a copy-on-write fault, allocates one fresh anonymous page, copies the 4 KiB in, repoints your page-table entry at the new page with the writable bit set, and re-runs your instruction. From then on that one page is yours alone. The file is untouched. Other mappers are untouched. There is no writeback path at all — with MAP_PRIVATE, PROT_WRITE only needs a read-only fd, because your writes are never going anywhere near the file.

The consequence that matters for anyone building density is the granularity. Pages you have not written are still shared. Your memory cost is proportional to what you touched, not to what you mapped. Map a 4 GiB file MAP_PRIVATE, read all of it and write 40 MiB of it, and you have added roughly 40 MiB of private physical memory to the host's bill — the other 4 GiB is page cache that would be there anyway, shared with everyone else who mapped the same file. That single property is the reason MAP_PRIVATE is the load-bearing flag underneath fast VM restore, and the reason a copy-on-write fork of a running guest is cheap.

• Who sees your writes — MAP_PRIVATE: only you, ever; the write lands in a private anonymous copy. MAP_SHARED: every process mapping the same file region MAP_SHARED, plus anyone read()ing the file, immediately and without a syscall.
• Effect on the file — MAP_PRIVATE: none. The file is never modified through the mapping, and no writeback is scheduled. MAP_SHARED: dirty page-cache pages are written back to the file eventually, or on msync/fsync if you demand it.
• Required file access — MAP_PRIVATE: PROT_WRITE works on an O_RDONLY fd, since nothing is written back. MAP_SHARED: PROT_WRITE requires O_RDWR.
• Memory cost of a write — MAP_PRIVATE: one fresh anonymous page per page written (a copy-on-write fault), charged to your process. MAP_SHARED: zero extra pages; you dirty a page-cache page that already existed.
• Behaviour under memory pressure — MAP_PRIVATE: unwritten pages are clean shared page cache and can be dropped for free; your CoW copies are anonymous and dirty, so they can only be swapped, never dropped. MAP_SHARED: dirty pages are written back and become clean, then evictable.
• Across fork() — MAP_PRIVATE: parent and child share the mapping copy-on-write; each diverges privately on write. MAP_SHARED: the mapping stays genuinely shared; the child's writes are visible to the parent and vice versa.
• What it is for — MAP_PRIVATE: loading executables and shared libraries, snapshot restore, scratch views over immutable data, anything where the backing object must stay pristine. MAP_SHARED: IPC, shared ring buffers, memory-speed file mutation, anything where visibility or persistence IS the point.

Fifty lines of C that settle the argument

The quickest way to internalize this is to map the same file both ways in one process and watch the two views diverge. Below, a private write is invisible to the shared mapping and to the file; a shared write is visible to the file immediately.

// cc -O2 -Wall -o mmapdemo mmapdemo.c   (Linux)
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void) {
    const size_t len = 4096;
    int fd = open("/tmp/mmap-demo.bin", O_RDWR | O_CREAT | O_TRUNC, 0644);
    if (fd < 0) { perror("open"); return 1; }
    if (ftruncate(fd, len) < 0) { perror("ftruncate"); return 1; }
    if (pwrite(fd, "ORIGINAL", 8, 0) != 8) { perror("pwrite"); return 1; }

    /* Two mappings of the SAME 4 KiB of the SAME file, differing in one flag. */
    char *priv = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
    char *shrd = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED,  fd, 0);
    if (priv == MAP_FAILED || shrd == MAP_FAILED) { perror("mmap"); return 1; }

    char file[9] = {0};
    pread(fd, file, 8, 0);
    printf("start    priv=%.8s  shrd=%.8s  file=%s\n", priv, shrd, file);
    /* start    priv=ORIGINAL  shrd=ORIGINAL  file=ORIGINAL
       Both mappings are reading the very same page-cache page right now. */

    memcpy(priv, "PRIVATE!", 8);   /* write-protection fault -> private copy */
    pread(fd, file, 8, 0);
    printf("after P  priv=%.8s  shrd=%.8s  file=%s\n", priv, shrd, file);
    /* after P  priv=PRIVATE!  shrd=ORIGINAL  file=ORIGINAL
       The private mapping now points at an anonymous page of its own. */

    memcpy(shrd, "SHARED!!", 8);   /* dirties the page-cache page itself */
    pread(fd, file, 8, 0);
    printf("after S  priv=%.8s  shrd=%.8s  file=%s\n", priv, shrd, file);
    /* after S  priv=PRIVATE!  shrd=SHARED!!  file=SHARED!!
       read() sees it immediately: same page cache, no syscall needed. */

    msync(shrd, len, MS_SYNC);     /* durability for the SHARED mapping */
    msync(priv, len, MS_SYNC);     /* succeeds; writes nothing anywhere */

    munmap(priv, len);
    munmap(shrd, len);
    close(fd);
    return 0;
}

Run it and the third line is the punchline: the private view still says PRIVATE! even though the file now says SHARED!!. That page had already been copied on write, so it stopped tracking the file the instant it diverged. Now change the order — do the shared write first, before touching the private mapping — and on a typical Linux kernel the private mapping will show you SHARED!!, because it is still pointing at the shared page-cache page it never copied. Which brings us to the part that burns people: POSIX does not specify whether modifications another process makes to the underlying file become visible through your MAP_PRIVATE mapping's untouched pages. It is genuinely unspecified. In practice on Linux they generally do, until the moment you write to that page and take the copy-on-write fault, at which point your view freezes forever. So a MAP_PRIVATE mapping over a file somebody else is mutating gives you a per-page mixture of a live view and a frozen snapshot, with the split determined by which pages you happen to have written. Nobody wants that; plenty of code accidentally depends on it.

The anonymous combinations

Add MAP_ANONYMOUS and the backing object disappears — there is no file, just zero-filled pages materialized on first touch. The same private-versus-shared axis still applies, and the four combinations cover most of what memory management is:

• MAP_PRIVATE | MAP_ANONYMOUS — ordinary process memory. This is what malloc calls for large allocations and what a thread stack is. Zero pages on demand, private to you, gone at exit. Across fork() it becomes copy-on-write between parent and child.
• MAP_SHARED | MAP_ANONYMOUS — shared memory with no file behind it. Nothing else can open it by name, but it survives fork(), so a parent maps it before forking and every child sees the same physical pages. This is the classic zero-setup way to share a struct between a process and its children. (The named equivalent is a shm_open or memfd_create fd mapped MAP_SHARED.)
• MAP_PRIVATE on a file — the copy-on-write view. Executables and shared libraries are mapped this way: every process running /usr/bin/python shares its text pages physically, and a relocation write on a data page gets a private copy.
• MAP_SHARED on a file — the IPC and persistence view. Writes are visible to everyone mapping it and are eventually written back to the file.

Why this one flag is the whole trick behind fast VM restore

A Firecracker snapshot writes the guest's entire physical RAM to a file, vm.mem. Restoring it naively means reading gigabytes off disk before the guest can run a single instruction. Restoring it correctly means mapping the file MAP_PRIVATE, handing that address range to the VMM as guest physical memory, and resuming the vCPUs. The guest believes all of its RAM is present. None of it is loaded. Each first touch is a minor fault that either wires up the shared page-cache frame (a read) or mints a private copy (a write). N guests restoring from the same template file share every page they only read, and each pays real physical memory only for the pages it dirties. MAP_SHARED here would be a catastrophe rather than an optimization: the first guest to write to a page would be writing into the template's page cache, corrupting the memory image for every other guest restoring from it and eventually persisting that corruption back to the file on disk. The flag is doing isolation work, not just accounting work.

package main

import (
	"fmt"
	"os"

	"golang.org/x/sys/unix"
)

// mapGuestRAM maps a Firecracker memory snapshot (vm.mem) to serve as this
// guest's physical RAM. MAP_PRIVATE is the entire trick: every restore from
// this template shares the pages it only reads, and a guest costs real memory
// only for the pages it dirties. MAP_SHARED here would let one guest's write
// corrupt the template for every other guest on the host.
func mapGuestRAM(path string) ([]byte, error) {
	// O_RDONLY is sufficient. MAP_PRIVATE|PROT_WRITE never writes back, so the
	// kernel only demands read access; MAP_SHARED|PROT_WRITE would need O_RDWR.
	f, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	// The mapping keeps its own reference to the file, so closing the fd here
	// is safe and keeps the descriptor table small.
	defer f.Close()

	st, err := f.Stat()
	if err != nil {
		return nil, err
	}
	size := int(st.Size())

	mem, err := unix.Mmap(int(f.Fd()), 0, size,
		unix.PROT_READ|unix.PROT_WRITE,
		unix.MAP_PRIVATE|unix.MAP_NORESERVE)
	if err != nil {
		return nil, fmt.Errorf("mmap %s (%d bytes): %w", path, size, err)
	}

	// A booting guest touches its RAM in scattered order; readahead would drag
	// in pages it never looks at and inflate the page cache for nothing.
	if err := unix.Madvise(mem, unix.MADV_RANDOM); err != nil {
		fmt.Fprintln(os.Stderr, "madvise:", err) // advisory only, never fatal
	}
	return mem, nil
}

func main() {
	mem, err := mapGuestRAM("/var/lib/pandastack/seeds/base/vm.mem")
	if err != nil {
		fmt.Fprintln(os.Stderr, "map guest RAM:", err)
		os.Exit(1)
	}
	defer unix.Munmap(mem)

	// At this instant: gigabytes of address space, ~0 resident pages. Hand the
	// region to the VMM as guest physical memory and resume the vCPUs; the
	// guest faults in what it needs and nothing else.
	fmt.Printf("mapped %d bytes of guest RAM; residency follows the guest\n", len(mem))
}

This is why a PandaStack create that restores a baked snapshot lands around 179ms at p50 (p99 about 203ms), with the restore step itself roughly 49ms, against about 3s for a first cold boot: the guest resumes against a mapping, not against a completed read. It is the same reason a same-host fork runs in 400-750ms — the parent's pages are already resident, so the child's page tables point at warm physical memory — while a cross-host fork costs 1.2-3.5s, because the memory image has to reach the destination host before any page-table entry can point at it. The wall-clock is dominated by whether the bytes are already in the host's page cache, which is exactly the question MAP_PRIVATE lets you answer with "usually."

MAP_SHARED is a promise that your writes matter to somebody else. MAP_PRIVATE is a promise that they don't. Almost every memory bug I have chased at the VMM layer was somebody making the wrong promise and the kernel keeping it faithfully.

The traps, in the order they bite

1. Private dirty pages are anonymous, and anonymous pages cannot be dropped

This is the trap with the biggest operational blast radius. An unwritten MAP_PRIVATE page is clean file-backed page cache: under memory pressure the kernel can evict it instantly and for free, because a copy exists on disk and it can always be re-read. The moment you write to it, you own an anonymous dirty page. It has no home on disk. The kernel cannot drop it. Its only options are to keep it resident or to push it to swap, and if there is no swap, its only option is to keep it resident — or to invoke the OOM killer. So a host packed with guests restored from a shared template looks wonderfully dense right up until the guests start dirtying memory, at which point free-to-evict shared pages quietly convert into un-droppable private ones. Capacity planning has to be done against the dirty rate, not against the mapped size.

2. msync on a MAP_PRIVATE mapping does not do what you hope

People reach for msync expecting "flush my changes to the file." On a MAP_PRIVATE mapping there are no changes to flush — your writes live in anonymous copies that have no relationship to the file, so the call has nothing to persist. It generally returns success, which is the unhelpful part: no error tells you that your data went nowhere. If you want a private working copy that you can later persist, you must write it out explicitly with write() or pwrite(); the mapping will not do it for you. And on a MAP_SHARED mapping, remember msync is what forces the dirty page-cache pages out; MS_SYNC blocks until the write is issued, MS_ASYNC merely schedules it, and neither is a substitute for understanding your filesystem's fsync semantics.

3. Truncating a mapped file gives you SIGBUS, not an error code

A mapping is not a lock on the file's size. If another process truncates the file so that pages you have mapped now lie beyond the end of it, accessing those addresses raises SIGBUS and, by default, kills your process. There is no return value to check, because a memory access has nowhere to return an error to — that is the fundamental cost of trading syscalls for loads and stores. The same applies if you map past EOF in the first place: mapping succeeds, the pages within the final partial page read as zero, and pages entirely beyond EOF fault with SIGBUS on touch. Both flags are affected. If a file you map can be truncated by anyone else, either take a lock, or copy the file, or install a SIGBUS handler and accept that you are now writing the kind of code that needs sigsetjmp.

4. MAP_SHARED writeback timing is not a durability contract

Storing into a MAP_SHARED mapping makes your data visible to other processes immediately, and durable at some unspecified later moment — when the kernel's writeback threads get to it, when dirty ratios are exceeded, when someone else calls sync. "Visible" and "durable" are different properties and only one of them is instant. If your data must survive a power cut, you need msync(MS_SYNC) on the region, and depending on the filesystem, an fsync on the containing directory for the metadata that makes the file findable. The failure mode is nasty precisely because it is invisible in testing: everything looks persisted because the page cache serves reads correctly, and only a hard crash reveals that the bytes never left RAM.

5. fork() treats the two flags in opposite ways

A MAP_SHARED mapping is inherited by the child as a genuinely shared mapping: parent and child write the same physical pages, and each sees the other's stores. A MAP_PRIVATE mapping is inherited copy-on-write: the child gets its own view, both sides are write-protected, and the first write on either side mints a private page for that side. This is one of the sharpest edges in multi-process code, because the same buffer variable means "we can coordinate through this" in one case and "we each got a snapshot" in the other, with no syntactic difference at the point of use. Worth stating explicitly in a comment at every mmap call in a program that forks. Note also that mappings survive execve only in the sense that they do not — an exec replaces the address space entirely; if you need a region across exec, you need a file descriptor and a re-map.

6. MAP_NORESERVE, overcommit, and why mapping 4 GiB is not allocating 4 GiB

Mapping reserves address space. Allocation of physical memory happens on fault, page by page. Whether the kernel pretends to guarantee that those future faults can be satisfied depends on the overcommit policy in /proc/sys/vm/overcommit_memory: the default heuristic mode accepts most requests, mode 2 refuses to commit beyond a computed limit, and MAP_NORESERVE asks the kernel not to account swap space for this mapping at all. For a VMM mapping a large snapshot file MAP_PRIVATE, MAP_NORESERVE is usually right — you know the guest will not dirty the whole image — but it converts a clean, early mmap failure into a possible OOM kill much later. That trade is fine on a host you control and terrible on one you do not. Either way, judging your memory footprint by the size you passed to mmap will make every number you report wrong.

7. Accounting: RSS lies by design, PSS is the honest number

Resident set size counts every physical page currently mapped into the process, whether or not it is shared with anyone. That is a perfectly correct definition and a completely misleading total. Forty guests each mapping the same 1 GiB of shared clean template pages will each report about 1 GiB of RSS from that mapping, and summing across the processes will report 40 GiB of memory in use on a host that has spent one. Proportional set size divides each shared page by the number of processes sharing it, so those forty guests report about 25 MiB each and the sum lands back near the truth. When you are reasoning about copy-on-write density, PSS and Private_Dirty are the fields that mean something; RSS is a number you show people who are not going to ask follow-up questions.

Measuring it: smaps, smaps_rollup, and the fields that matter

None of this needs to be taken on faith. The kernel exposes per-mapping page accounting in /proc/<pid>/smaps — one stanza per VMA — and a pre-summed whole-process view in /proc/<pid>/smaps_rollup, which is dramatically cheaper to read on a process with thousands of mappings. The fields that answer the MAP_PRIVATE question are Shared_Clean (mapped, unwritten, still sharing the backing object's pages), Private_Dirty (the copy-on-write copies you actually paid for), and Pss.

# 1. Whole-process totals. smaps_rollup does the summing in the kernel.
$ pid=$(pgrep -n firecracker)
$ grep -E '^(Rss|Pss|Shared_Clean|Shared_Dirty|Private_Clean|Private_Dirty):' \
    /proc/$pid/smaps_rollup

# (illustrative shape of the output, not a benchmark)
Rss:             1441792 kB   # everything resident, shared or not -- flatters you
Pss:              212992 kB   # fair share: each shared page divided by its sharers
Shared_Clean:    1327104 kB   # mapped from vm.mem, never written: free to evict
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    114688 kB   # the CoW copies -- what this guest REALLY costs

# 2. Per-mapping detail: which VMA is doing it. One stanza per mapping.
$ grep -A 20 'vm.mem' /proc/$pid/smaps | grep -E 'kB$'

# 3. Watch a page migrate from shared to private. Dirty a little inside the
#    guest, then re-read: Shared_Clean falls and Private_Dirty rises by roughly
#    the same amount -- one copy-on-write fault per 4 KiB page written.
$ grep -E '^(Shared_Clean|Private_Dirty):' /proc/$pid/smaps_rollup

# 4. The double-counting demo. Sum RSS across every guest and you will "find"
#    far more memory in use than the host owns, because each shared template
#    page is counted once per process. Sum PSS instead and it reconciles.
$ for p in $(pgrep firecracker); do
>   awk '/^Rss:/{print $2}' /proc/$p/smaps_rollup
> done | paste -sd+ | bc      # inflated: shared pages counted N times
$ for p in $(pgrep firecracker); do
>   awk '/^Pss:/{print $2}' /proc/$p/smaps_rollup
> done | paste -sd+ | bc      # honest: shared pages split across sharers

The experiment worth running once, on your own machine, is the migration in step 3. Restore or fork something, note Shared_Clean, dirty a known amount of memory inside it, and read the fields again. Watching a precise number of kilobytes move from the shared column to the private column is what turns copy-on-write from a slogan into a thing you can plan capacity against. It is also how you catch the opposite failure: a workload you assumed was read-mostly quietly touching every page it maps, at which point MAP_PRIVATE has bought you nothing but a slower first write.

The mental model that holds up

mmap gives you a view of a backing object, and the flag decides who else lives in that view. MAP_SHARED means your writes belong to the object: other mappers see them, read() sees them, and the file will eventually see them, which makes it the right choice for IPC and for treating a file as memory — with msync as the durability step, not an optimization. MAP_PRIVATE means your writes belong to you: reads share physical pages with everyone else, the first write to each page takes a copy-on-write fault and hands you a private anonymous copy, and the object stays pristine. That asymmetry is what lets many microVMs restore from one memory file and pay only for what they dirty, and it is also what makes the traps sharp — copies that cannot be evicted, an msync that persists nothing, a SIGBUS where you expected an errno, and RSS totals that add up to more memory than the host has. PandaStack's agent is open source under Apache-2.0, so if you want to watch Private_Dirty climb by exactly the pages a guest touched, you can run it on your own KVM host and look. For the layer below this one, see /blog/copy-on-write-page-tables-explained; for the Firecracker-specific restore path, /blog/firecracker-memory-file-mmap-explained.