The basic idea was a linux kernel module which allowed you to get the underlying page id behind each VMA of any process.
Putting this info together (and hoping it wasn't changing too much as you queried different processes) allowed me to calculate "effective resident" and "effective vm size", where I used the rule that a page shared across N processes contributed pagesize/N to each one.
I could also give those same figures for each ELF section and ELF symbol (via some very fragile code which looked at the process mapping and read the ELF details to work out which bit of memory belonged to each section/symbol).
The code has long since rotted and the architecture sucked (I tied the data gathering code into the UI).
Would anyone be interested in reviving it/helping me revive it? I consider doing so from time to time, but I feel that codebase sucks now so avoid touching it.
I was a big user of exmap a few years ago at a previous employer (THALES), where exmap was used to do a lot of things to systems being tested/certified for CENELEC SIL-4 (safety critical) usage. I wrote portions of the online testing systems, and the full system tests for these components used exmap in our test systems to corrupt text segments, copy stuff where it shouldn't be, etc. exmap was essential to testing.
PM me privately your details and we can talk about what you want to do with exmap; at the very least I can explain to you how its being used in an industrial testing context, and there may be people still working there, maintaining exmap code, that would be interested in your situation, who nevertheless do not read n.yc.
Nice! /proc/<pid>/pagemap is basically an alternate to my kernel module. So I could do a pure userspace solution now.
Can anyone recommend visualisation tools using pagemap? I'd like to get the exmap functionality (per-process 'effective usage' and also ELF-section/symbol resolution.
If that already exists great - if not maybe I can enhance an existing pagemap tool.
Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.
Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.
"mem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.
smem has many features:
system overview listing
listings by process, mapping, user
filtering by process, mapping, or user
configurable columns from multiple data sources
configurable output units and percentages
configurable headers and totals
reading live data from /proc
reading data snapshots from directory mirrors or compressed tarballs
lightweight capture tool for embedded systems
built-in chart generation"
This is great. One of the reasons I've always liked Phusion Passenger is because of its toolchain. They use a similar method (inspecting smaps) to report Private Dirty RSS for Apache processes and Passenger processes:
The Passenger utility is difficult to extracate from their package though, because it has several separate class dependencies. It's good software writing technique for a library, but it doesn't make it terribly useful for a shell script.
KDE's KSysGuard tries to do a detailed calculation much like this if you ask for it (right-click on a process in ksysguard's process list and click "Detailed Memory Status", IIRC). This process list can be pulled directly from KRunner (Alt-F2) as well.
The basic idea was a linux kernel module which allowed you to get the underlying page id behind each VMA of any process.
Putting this info together (and hoping it wasn't changing too much as you queried different processes) allowed me to calculate "effective resident" and "effective vm size", where I used the rule that a page shared across N processes contributed pagesize/N to each one.
I could also give those same figures for each ELF section and ELF symbol (via some very fragile code which looked at the process mapping and read the ELF details to work out which bit of memory belonged to each section/symbol).
The code has long since rotted and the architecture sucked (I tied the data gathering code into the UI).
Would anyone be interested in reviving it/helping me revive it? I consider doing so from time to time, but I feel that codebase sucks now so avoid touching it.