Address Sanitizier invokes OOM-killer

I am trying to use Address Sanitizer, but the kernel keeps killing my process due to excessive memory usage. Without Address Sanitizer the process runs just fine.

The program is compiled for arm-v7a using gcc-8.2.1 with

-fno-omit-frame-pointer
-fsanitize=address
-fsanitize-recover=all

-fdata-sections
-ffunction-sections
-fPIC

I am starting the process as follows:

ASAN_OPTIONS=debug=1:verbosity=0:detect_leaks=0:abort_on_error=0:halt_on_error=0:check_initialization_order=1:allocator_may_return_null=1 ./Launcher

Is there a way to reduce the memory footprint of the Address Sanitizer? Unfortunately, enabling swap is not an option.

This is the kernel log as printed by dmesg:

[512792.413376] Launcher invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
[512792.424695] CPU: 3 PID: 7786 Comm: Launcher Tainted: G        W         5.4.1 #1
[512792.432821] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[512792.439455] Backtrace:
[512792.442039] [<8010eb1c>] (dump_backtrace) from [<8010eee0>] (show_stack+0x20/0x24)
[512792.449721]  r7:811d32ec r6:00000000 r5:60070113 r4:811d32ec
[512792.455500] [<8010eec0>] (show_stack) from [<80ba06e8>] (dump_stack+0xbc/0xe8)
[512792.462840] [<80ba062c>] (dump_stack) from [<80257360>] (dump_header+0x64/0x440)
[512792.470343]  r10:00000a24 r9:a9a4ce00 r8:00016f9c r7:80e82aac r6:a749fce0 r5:a9a4ce00
[512792.478275]  r4:a749fce0 r3:6f25b167
[512792.481958] [<802572fc>] (dump_header) from [<80256364>] (oom_kill_process+0x494/0x4ac)
[512792.490066]  r10:00000a24 r9:a9a4c100 r8:00016f9c r7:80e82aac r6:a749fce0 r5:a9a4ce00
[512792.497996]  r4:a9a4d264
[512792.500636] [<80255ed0>] (oom_kill_process) from [<80256e8c>] (out_of_memory+0xf8/0x4ec)
[512792.508830]  r10:00000a24 r9:a9a4c100 r8:00016f9c r7:8110b640 r6:8110b640 r5:811d8860
[512792.516760]  r4:a749fce0
[512792.519405] [<80256d94>] (out_of_memory) from [<802a0910>] (__alloc_pages_nodemask+0xf7c/0x13a4)
[512792.528295]  r9:00000000 r8:81107d30 r7:811d5588 r6:0000233c r5:00000000 r4:00000000
[512792.536153] [<8029f994>] (__alloc_pages_nodemask) from [<80285d10>] (__pte_alloc+0x34/0x1ac)
[512792.544697]  r10:74b94000 r9:00000000 r8:00000000 r7:a8b9e580 r6:a8b9e580 r5:a7445d28
[512792.552628]  r4:a7445d28
[512792.555271] [<80285cdc>] (__pte_alloc) from [<802869c8>] (copy_page_range+0x4ec/0x650)
[512792.563295]  r9:00000000 r8:00000000 r7:a8b9e580 r6:a7174f4c r5:a8b9e580 r4:a7445d28
[512792.571148] [<802864dc>] (copy_page_range) from [<801241b8>] (dup_mm+0x470/0x4e0)
[512792.578736]  r10:a7174f14 r9:a7174f10 r8:a8b9d680 r7:a7c36420 r6:a7174f4c r5:a8b9e580
[512792.586667]  r4:a7835d20
[512792.589307] [<80123d48>] (dup_mm) from [<801255e0>] (copy_process+0x10bc/0x1888)
[512792.596807]  r10:a749ff60 r9:ffffffff r8:00000000 r7:a749e000 r6:9d283400 r5:a825c300
[512792.604738]  r4:00100000
[512792.607378] [<80124524>] (copy_process) from [<80125fb8>] (_do_fork+0x90/0x750)
[512792.614792]  r10:00100000 r9:a749e000 r8:801011c4 r7:a749e000 r6:a749ff60 r5:6f25b167
[512792.622722]  r4:00000001
[512792.625362] [<80125f28>] (_do_fork) from [<80126954>] (sys_clone+0x80/0x9c)
[512792.632428]  r10:00000078 r9:a749e000 r8:801011c4 r7:00000078 r6:7649e000 r5:6f25b167
[512792.640358]  r4:a749e000
[512792.643001] [<801268d4>] (sys_clone) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[512792.650671] Exception stack(0xa749ffa8 to 0xa749fff0)
[512792.655828] ffa0:                   54ad00fc 76ffe964 00100011 00000000 54ad00fc 00000000
[512792.664112] ffc0: 54ad00fc 76ffe964 7649e000 00000078 54ad0100 54ad0120 00000001 54ad0280
[512792.672391] ffe0: 00000078 54ad00e8 763d590b 763bf746
[512792.677546]  r5:76ffe964 r4:54ad00fc
[512792.681484] Mem-Info:
[512792.683936] active_anon:158884 inactive_anon:15315 isolated_anon:0
active_file:1041 inactive_file:1140 isolated_file:0
unevictable:2224 dirty:8 writeback:1 unstable:0
slab_reclaimable:4553 slab_unreclaimable:4490
mapped:5064 shmem:17635 pagetables:1579 bounce:0
free:56987 free_pcp:173 free_cma:53962
[512792.718450] Node 0 active_anon:635536kB inactive_anon:61260kB active_file:4264kB inactive_file:5460kB unevictable:8896kB isolated(anon):0kB isolated(file):0kB mapped:21056kB dirty:32kB writeback:4kB shmem:70540kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[512792.742142] Normal free:226708kB min:3312kB low:4140kB high:4968kB active_anon:635436kB inactive_anon:61260kB active_file:4584kB inactive_file:5652kB unevictable:8896kB writepending:36kB present:1048576kB managed:1015668kB mlocked:0kB kernel_stack:1216kB pagetables:6316kB bounce:0kB free_pcp:192kB local_pcp:0kB free_cma:215848kB
[512792.771461] lowmem_reserve[]: 0 0 0
[512792.775161] Normal: 1651*4kB (UMEC) 839*8kB (UMEC) 495*16kB (UMEC) 221*32kB (UMEC) 78*64kB (UEC) 29*128kB (MC) 1*256kB (U) 40*512kB (C) 35*1024kB (C) 21*2048kB (C) 10*4096kB (C) 2*8192kB (C) 0*16384kB 1*32768kB (C) = 226708kB
[512792.795442] 20243 total pagecache pages
[512792.799391] 0 pages in swap cache
[512792.802816] Swap cache stats: add 0, delete 0, find 0/0
[512792.808232] Free swap  = 0kB
[512792.811225] Total swap = 0kB
[512792.814296] 262144 pages RAM
[512792.817288] 0 pages HighMem/MovableOnly
[512792.821232] 8227 pages reserved
[512792.824558] 81920 pages cma reserved
[512792.828247] Tasks state (memory values in pages):
[512792.833057] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[512792.841890] [    211]     0   211     9965     1608    67584        0             0 systemd-journal
[512792.851149] [    224]     0   224     3848      249    16384        0         -1000 systemd-udevd
[512792.860222] [    317]     0   317     1559      339    12288        0             0 dhclient
[512792.868867] [    316]     0   316     1559      348    14336        0             0 dhclient
[512792.877508] [    333]     0   333     1810      856    14336        0             0 haveged
[512792.886061] [    334]   101   334     4985      261    22528        0             0 systemd-timesyn
[512792.895309] [    336]   104   336     1342      167    12288        0             0 rpcbind
[512792.903866] [    368]   106   368     1333      218    12288        0          -900 dbus-daemon
[512792.912684] [    369]     0   369     6193      356    22528        0             0 rsyslogd
[512792.921327] [    370]     0   370     2681      178    18432        0             0 systemd-logind
[512792.930490] [    372]     0   372     1625      158    14336        0             0 cron
[512792.938784] [    431]     0   431      428      122    10240        0             0 motion_sensor
[512792.947870] [    560]     0   560     8756      207    18432        0             0 automount
[512792.956597] [    564]     0   564     1190      172    12288        0             0 login
[512792.964988] [    566]     0   566     1338       98    12288        0             0 agetty
[512792.973372] [    572]     0   572     2218      276    16384        0         -1000 sshd
[512792.981664] [    574]     0   574      946       33    12288        0             0 inputattach
[512792.990569] [    637]     0   637     3017      379    18432        0             0 systemd
[512792.999122] [    640]     0   640     3504      402    20480        0             0 (sd-pam)
[512793.007768] [    653]     0   653     1760      329    12288        0             0 bash
[512793.016057] [    671]     0   671     2599     1116    18432        0             0 Server.
[512793.025310] [    732]     0   732     1300      132    12288        0             0 dbus-daemon
[512793.034212] [  31836]     0 31836     3173      980    22528        0             0 sshd
[512793.042428] [  31847]     0 31847      422      154     8192        0             0 sftp-server
[512793.051332] [   5350]     0  5350     2555      351    16384        0             0 sshd
[512793.059631] [   5452]     0  5452     1793      379    16384        0             0 bash
[512793.067924] [   5823]     0  5823     2555      350    16384        0             0 sshd
[512793.076216] [   5833]     0  5833     1760      326    14336        0             0 bash
[512793.084509] [   6822]     0  6822      792       31    10240        0             0 xinit
[512793.092813] [   6823]     0  6823    29526     5386   112640        0             0 Xorg
[512793.101103] [   6827]     0  6827     3655      866    22528        0             0 xterm
[512793.109488] [   6829]     0  6829     1620      114    14336        0             0 bash
[512793.117784] [   7256]     0  7256     1549      322    12288        0             0 watch
[512793.126169] [   7363]     0  7363   127832    56725   520192        0             0 gdb
[512793.134370] [   7368]     0  7368   281561    93707  1046528        0             0 Launcher
[512793.143613] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=Launcher,pid=7368,uid=0
[512793.152974] Out of memory: Killed process 7368 (Launcher) total-vm:1126244kB, anon-rss:365128kB, file-rss:5700kB, shmem-rss:4000kB, UID:0 pgtables:1046528kB oom_score_adj:0
[512793.387824] oom_reaper: reaped process 7368 (Launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:4000kB

Answer

You could reduce some Asan features (or enable them one by one in separate runs):

# Disable UAR error detection (reduces code and heap size)
CFLAGS+='-fsanitize-address-use-after-return=never -fno-sanitize-address-use-after-scope'
export ASAN_OPTIONS="$ASAN_OPTIONS:detect_stack_use_after_return=1"
# Disable inline instrumentation (slower but saves code size)
CFLAGS+='-fsanitize-address-outline-instrumentation'
# Reduce heap quarantine (reduces heap consumption but also lowers chance of UAF detection)
export ASAN_OPTIONS="$ASAN_OPTIONS:quarantine_size_mb=16"
# Do not keep full backtrace of malloc origin (slightly complicates debugging but reduces heap size)
export ASAN_OPTIONS="$ASAN_OPTIONS:malloc_context_size=5"

Compiler options are for Clang but GCC also has similar switches.

As for the swap, we had good experience with enabling compressed swap in RAM.