How much memory does my program need?
Knowing when your program needs too much memory is relatively easy: either it gets terminated by the OS, or the whole machine slows to an unbearable crawl as pages start to get swapped to disk. But how to know how much memory does it needs without pushing these limits?
From the command line / operating system
Simplest way is to use the /usr/bin/time -v
command which will
report maximum memory usage (“Maximum resident set size”) :
$ /usr/bin/time -v echo 1
1
Command being timed: "echo 1"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 50%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1932
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 93
Voluntary context switches: 3
Involuntary context switches: 0
Swaps: 0
File system inputs: 72
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Reducing the apparent available memory
Reduce the amount of memory avaiable to the process through the ulimit shell command, .e.g., :
$ ulimit -v 10000 # Reduce to 10Mb
$ ls
. ..
$ ulimit -v 1000 # Reduce to 1MB
ls: error while loading shared libraries: libc.so.6: failed to map segment from shared object
Note that this reduces the total virtual memory available to each child process. Other possibilities are tools using the Linux cgroups capabilities, e.g., docker or firejail.
The advantage of this approach is that you can see how the program responds to low memory situations.
Monitor memory sys-calls
It is possible to monitor memory-related system calls using the
perf
tools, e.g., to see the memory calls when an ls
command is run can do the following:
sudo perf stat -e syscalls:sys_enter_mmap -e syscalls:sys_enter_brk ls
It does not info about sizes without work, but it can give a picture of the memory allocation/dellacation dynamics. The advantage of this approach is that it can be done while a process is running and even on a whole-system basis, giving insingt into a complex running systems.
Track the memory allocation / de-allocation
The final approach is do automatic instrumentation of the executable code so that memory allocations and de-allocations are tracket. Two popular tools for this heaptrack and valgrind. This is a powerful approach which gives a detailed picture of where and how much in the program memory is being used.