Originally published 23 April 2012, updated 10 September 2019 (add link to troubleshooting tool), updated 18 October 2019 (moved to Jekyll), updated 3 April 2023 with links to other tools

Development of large systems using many shared (dynamically) loaded libraries can sometimes lead to some frustrating bugs that are difficult to diagnose. These bugs often arise because there a few different versions of libraries on the system and the “wrong” version gets loaded instead of the one the developer used wanted.

Note You may also find of use my online tool for diagnosing linker problems on Linux implemented as an expert system. It is being expanded at the time of writing this but probably useful already.

I used to often debug these problems reasonably efficiently using the strace command (man strace) and checking which libraries are getting accessed. However, there is a much more efficient but perhaps not very well known way of debugging shared library loading problems: the LD_DEBUG environment variable.

If the LD_DEBUG variable is set then the Linux dynamic linker will dump debug information which can be used to resolve most loading problems very quickly. To see the available options just run any program with the variable set to help, i.e.:

LD_DEBUG=help cat
Valid options for the LD_DEBUG environment variable are:

  libs        display library search paths
  reloc       display relocation processing
  files       display progress for input file
  symbols     display symbol table processing
  bindings    display information about symbol binding
  versions    display version dependencies
  all         all previous options combined
  statistics  display relocation statistics
  unused      determined unused DSOs
  help        display this help message and exit

To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment
variable.

Note There are many other useful tools to deal with linking issues:

  1. The strace program provides insight into all system calls, including searching, opening dynamic libraries
  2. The ldd program resolves dynamic library dependencies
  3. The objdump -x YOURFILE | grep NEEDED command lists the records in the program or library showing which other libraries are needed
  4. The patchelf program makes it easy to change the rpath of an ELF executable, changing the built-in search order
  5. The LD_PRELOAD environment allows easy substitution of dynamic libraries

Need further advice? With over 20 years of experience we are uniquely placed to help – contact us at webs@bnikolic.co.uk

Note This variable is available Linux only. On MS Windows similar information can be obtained by enabling “Show Loader Snaps” (https://abitofscotland.wordpress.com/2019/12/20/dll-load-failure/) using the gflags.exe program to enable this and then using windbg to view the log. For example:

  1. Install Windows SDK for gflags and windbg winget install Microsoft.WindowsSDK.10.0.19041 (or download from Microsoft website )
  2. Enable ShowLoaderSnaps for an executable. Example: to enable for notepad.exe for example do in Powershell &"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /i notepad.exe +sls
  3. Execute notepad.exe under WinDbg app. Detailed DLL loading log will be displayed

And here is an example run on my laptop:

 LD_DEBUG=all cat
 28504:	
 28504:	file=libc.so.6 [0];  needed by cat [0]
 28504:	find library=libc.so.6 [0]; searching
 28504:	 search path=/home/bnikolic/s/lib/tls/x86_64:/home/bnikolic/s/lib/tls:/home/bnikolic/s/lib/x86_64:/home/bnikolic/s/lib:tls/x86_64:tls:x86_64::/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/tls/x86_64:/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/tls:/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/x86_64:/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib		(LD_LIBRARY_PATH)
 28504:	  trying file=/home/bnikolic/s/lib/tls/x86_64/libc.so.6
 28504:	  trying file=/home/bnikolic/s/lib/tls/libc.so.6
 28504:	  trying file=/home/bnikolic/s/lib/x86_64/libc.so.6
 28504:	  trying file=/home/bnikolic/s/lib/libc.so.6
 28504:	  trying file=tls/x86_64/libc.so.6
 28504:	  trying file=tls/libc.so.6
 28504:	  trying file=x86_64/libc.so.6
 28504:	  trying file=libc.so.6
 28504:	  trying file=/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/tls/x86_64/libc.so.6
 28504:	  trying file=/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/tls/libc.so.6
 28504:	  trying file=/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/x86_64/libc.so.6
 28504:	  trying file=/home/bnikolic/d/p/R-2.14.1bin/lib/R/lib/libc.so.6
 28504:	 search cache=/etc/ld.so.cache
 28504:	  trying file=/lib/x86_64-linux-gnu/libc.so.6
 28504:	
 28504:	file=libc.so.6 [0];  generating link map
 28504:	  dynamic: 0x00007fc38e525b40  base: 0x00007fc38e18c000   size: 0x00000000003a0368
 28504:	    entry: 0x00007fc38e1ad420  phdr: 0x00007fc38e18c040  phnum:                 10
 28504:	
 28504:	checking for version `GLIBC_2.4' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file cat [0]
 28504:	checking for version `GLIBC_2.3' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file cat [0]
 28504:	checking for version `GLIBC_2.3.4' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file cat [0]
 28504:	checking for version `GLIBC_2.2.5' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file cat [0]
 28504:	checking for version `GLIBC_2.3' in file /lib64/ld-linux-x86-64.so.2 [0] required by file /lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	checking for version `GLIBC_PRIVATE' in file /lib64/ld-linux-x86-64.so.2 [0] required by file /lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	
 28504:	relocation processing: /lib/x86_64-linux-gnu/libc.so.6 (lazy)
 28504:	symbol=_res;  lookup in file=cat [0]
 28504:	symbol=_res;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_res' [GLIBC_2.2.5]
 28504:	symbol=_IO_file_close;  lookup in file=cat [0]
 28504:	symbol=_IO_file_close;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_IO_file_close' [GLIBC_2.2.5]
 28504:	symbol=stderr;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `stderr' [GLIBC_2.2.5]
 28504:	symbol=error_one_per_line;  lookup in file=cat [0]
 28504:	symbol=error_one_per_line;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `error_one_per_line' [GLIBC_2.2.5]
 28504:	symbol=__malloc_initialize_hook;  lookup in file=cat [0]
 28504:	symbol=__malloc_initialize_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__malloc_initialize_hook' [GLIBC_2.2.5]
 28504:	symbol=__morecore;  lookup in file=cat [0]
 28504:	symbol=__morecore;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__morecore' [GLIBC_2.2.5]
 28504:	symbol=__key_encryptsession_pk_LOCAL;  lookup in file=cat [0]
 28504:	symbol=__key_encryptsession_pk_LOCAL;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__key_encryptsession_pk_LOCAL' [GLIBC_2.2.5]
 28504:	symbol=__progname_full;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `__progname_full' [GLIBC_2.2.5]
 28504:	symbol=__ctype32_tolower;  lookup in file=cat [0]
 28504:	symbol=__ctype32_tolower;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype32_tolower' [GLIBC_2.2.5]
 28504:	symbol=__key_gendes_LOCAL;  lookup in file=cat [0]
 28504:	symbol=__key_gendes_LOCAL;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__key_gendes_LOCAL' [GLIBC_2.2.5]
 28504:	symbol=_environ;  lookup in file=cat [0]
 28504:	symbol=_environ;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_environ' [GLIBC_2.2.5]
 28504:	symbol=_rtld_global;  lookup in file=cat [0]
 28504:	symbol=_rtld_global;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=_rtld_global;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_rtld_global' [GLIBC_PRIVATE]
 28504:	symbol=__progname;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `__progname' [GLIBC_2.2.5]
 28504:	symbol=argp_err_exit_status;  lookup in file=cat [0]
 28504:	symbol=argp_err_exit_status;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `argp_err_exit_status' [GLIBC_2.2.5]
 28504:	symbol=mallwatch;  lookup in file=cat [0]
 28504:	symbol=mallwatch;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `mallwatch' [GLIBC_2.2.5]
 28504:	symbol=__rcmd_errstr;  lookup in file=cat [0]
 28504:	symbol=__rcmd_errstr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__rcmd_errstr' [GLIBC_2.2.5]
 28504:	symbol=svcauthdes_stats;  lookup in file=cat [0]
 28504:	symbol=svcauthdes_stats;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `svcauthdes_stats' [GLIBC_2.2.5]
 28504:	symbol=__libc_enable_secure;  lookup in file=cat [0]
 28504:	symbol=__libc_enable_secure;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=__libc_enable_secure;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `__libc_enable_secure' [GLIBC_PRIVATE]
 28504:	symbol=_res_hconf;  lookup in file=cat [0]
 28504:	symbol=_res_hconf;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_res_hconf' [GLIBC_2.2.5]
 28504:	symbol=malloc;  lookup in file=cat [0]
 28504:	symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc' [GLIBC_2.2.5]
 28504:	symbol=getdate_err;  lookup in file=cat [0]
 28504:	symbol=getdate_err;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `getdate_err' [GLIBC_2.2.5]
 28504:	symbol=__tzname;  lookup in file=cat [0]
 28504:	symbol=__tzname;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__tzname' [GLIBC_2.2.5]
 28504:	symbol=__timezone;  lookup in file=cat [0]
 28504:	symbol=__timezone;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__timezone' [GLIBC_2.2.5]
 28504:	symbol=_rtld_global_ro;  lookup in file=cat [0]
 28504:	symbol=_rtld_global_ro;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=_rtld_global_ro;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_rtld_global_ro' [GLIBC_PRIVATE]
 28504:	symbol=optarg;  lookup in file=cat [0]
 28504:	symbol=optarg;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `optarg' [GLIBC_2.2.5]
 28504:	symbol=__ctype_tolower;  lookup in file=cat [0]
 28504:	symbol=__ctype_tolower;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype_tolower' [GLIBC_2.2.5]
 28504:	symbol=__ctype_toupper;  lookup in file=cat [0]
 28504:	symbol=__ctype_toupper;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype_toupper' [GLIBC_2.2.5]
 28504:	symbol=svc_max_pollfd;  lookup in file=cat [0]
 28504:	symbol=svc_max_pollfd;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `svc_max_pollfd' [GLIBC_2.2.5]
 28504:	symbol=__ctype_b;  lookup in file=cat [0]
 28504:	symbol=__ctype_b;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype_b' [GLIBC_2.2.5]
 28504:	symbol=argp_program_version_hook;  lookup in file=cat [0]
 28504:	symbol=argp_program_version_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `argp_program_version_hook' [GLIBC_2.2.5]
 28504:	symbol=__after_morecore_hook;  lookup in file=cat [0]
 28504:	symbol=__after_morecore_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__after_morecore_hook' [GLIBC_2.2.5]
 28504:	symbol=__environ;  lookup in file=cat [0]
 28504:	symbol=__environ;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__environ' [GLIBC_2.2.5]
 28504:	symbol=__ctype32_b;  lookup in file=cat [0]
 28504:	symbol=__ctype32_b;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype32_b' [GLIBC_2.2.5]
 28504:	symbol=__curbrk;  lookup in file=cat [0]
 28504:	symbol=__curbrk;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__curbrk' [GLIBC_2.2.5]
 28504:	symbol=argp_program_version;  lookup in file=cat [0]
 28504:	symbol=argp_program_version;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `argp_program_version' [GLIBC_2.2.5]
 28504:	symbol=__daylight;  lookup in file=cat [0]
 28504:	symbol=__daylight;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__daylight' [GLIBC_2.2.5]
 28504:	symbol=__memalign_hook;  lookup in file=cat [0]
 28504:	symbol=__memalign_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__memalign_hook' [GLIBC_2.2.5]
 28504:	symbol=__malloc_hook;  lookup in file=cat [0]
 28504:	symbol=__malloc_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__malloc_hook' [GLIBC_2.2.5]
 28504:	symbol=__free_hook;  lookup in file=cat [0]
 28504:	symbol=__free_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__free_hook' [GLIBC_2.2.5]
 28504:	symbol=svc_pollfd;  lookup in file=cat [0]
 28504:	symbol=svc_pollfd;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `svc_pollfd' [GLIBC_2.2.5]
 28504:	symbol=_nl_domain_bindings;  lookup in file=cat [0]
 28504:	symbol=_nl_domain_bindings;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_nl_domain_bindings' [GLIBC_2.2.5]
 28504:	symbol=_nl_msg_cat_cntr;  lookup in file=cat [0]
 28504:	symbol=_nl_msg_cat_cntr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_nl_msg_cat_cntr' [GLIBC_2.2.5]
 28504:	symbol=argp_program_bug_address;  lookup in file=cat [0]
 28504:	symbol=argp_program_bug_address;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `argp_program_bug_address' [GLIBC_2.2.5]
 28504:	symbol=__key_decryptsession_pk_LOCAL;  lookup in file=cat [0]
 28504:	symbol=__key_decryptsession_pk_LOCAL;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__key_decryptsession_pk_LOCAL' [GLIBC_2.2.5]
 28504:	symbol=h_errlist;  lookup in file=cat [0]
 28504:	symbol=h_errlist;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `h_errlist' [GLIBC_2.2.5]
 28504:	symbol=program_invocation_short_name;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `program_invocation_short_name' [GLIBC_2.2.5]
 28504:	symbol=optind;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `optind' [GLIBC_2.2.5]
 28504:	symbol=_dl_starting_up;  lookup in file=cat [0]
 28504:	symbol=_dl_starting_up;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=_dl_starting_up;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	symbol=stdout;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `stdout' [GLIBC_2.2.5]
 28504:	symbol=obstack_alloc_failed_handler;  lookup in file=cat [0]
 28504:	symbol=obstack_alloc_failed_handler;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `obstack_alloc_failed_handler' [GLIBC_2.2.5]
 28504:	symbol=error_print_progname;  lookup in file=cat [0]
 28504:	symbol=error_print_progname;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `error_print_progname' [GLIBC_2.2.5]
 28504:	symbol=optopt;  lookup in file=cat [0]
 28504:	symbol=optopt;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `optopt' [GLIBC_2.2.5]
 28504:	symbol=_IO_funlockfile;  lookup in file=cat [0]
 28504:	symbol=_IO_funlockfile;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_IO_funlockfile' [GLIBC_2.2.5]
 28504:	symbol=obstack_exit_failure;  lookup in file=cat [0]
 28504:	symbol=obstack_exit_failure;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `obstack_exit_failure' [GLIBC_2.2.5]
 28504:	symbol=error_message_count;  lookup in file=cat [0]
 28504:	symbol=error_message_count;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `error_message_count' [GLIBC_2.2.5]
 28504:	symbol=svc_fdset;  lookup in file=cat [0]
 28504:	symbol=svc_fdset;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `svc_fdset' [GLIBC_2.2.5]
 28504:	symbol=program_invocation_name;  lookup in file=cat [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to cat [0]: normal symbol `program_invocation_name' [GLIBC_2.2.5]
 28504:	symbol=loc1;  lookup in file=cat [0]
 28504:	symbol=loc1;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `loc1' [GLIBC_2.2.5]
 28504:	symbol=free;  lookup in file=cat [0]
 28504:	symbol=free;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
 28504:	symbol=loc2;  lookup in file=cat [0]
 28504:	symbol=loc2;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `loc2' [GLIBC_2.2.5]
 28504:	symbol=__fpu_control;  lookup in file=cat [0]
 28504:	symbol=__fpu_control;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__fpu_control' [GLIBC_2.2.5]
 28504:	symbol=re_syntax_options;  lookup in file=cat [0]
 28504:	symbol=re_syntax_options;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `re_syntax_options' [GLIBC_2.2.5]
 28504:	symbol=stdin;  lookup in file=cat [0]
 28504:	symbol=stdin;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `stdin' [GLIBC_2.2.5]
 28504:	symbol=__check_rhosts_file;  lookup in file=cat [0]
 28504:	symbol=__check_rhosts_file;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__check_rhosts_file' [GLIBC_2.2.5]
 28504:	symbol=opterr;  lookup in file=cat [0]
 28504:	symbol=opterr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `opterr' [GLIBC_2.2.5]
 28504:	symbol=__ctype32_toupper;  lookup in file=cat [0]
 28504:	symbol=__ctype32_toupper;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__ctype32_toupper' [GLIBC_2.2.5]
 28504:	symbol=__realloc_hook;  lookup in file=cat [0]
 28504:	symbol=__realloc_hook;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__realloc_hook' [GLIBC_2.2.5]
 28504:	symbol=_dl_argv;  lookup in file=cat [0]
 28504:	symbol=_dl_argv;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=_dl_argv;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_dl_argv' [GLIBC_PRIVATE]
 28504:	symbol=rpc_createerr;  lookup in file=cat [0]
 28504:	symbol=rpc_createerr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `rpc_createerr' [GLIBC_2.2.5]
 28504:	symbol=_IO_2_1_stderr_;  lookup in file=cat [0]
 28504:	symbol=_IO_2_1_stderr_;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_IO_2_1_stderr_' [GLIBC_2.2.5]
 28504:	symbol=_IO_2_1_stdout_;  lookup in file=cat [0]
 28504:	symbol=_IO_2_1_stdout_;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_IO_2_1_stdout_' [GLIBC_2.2.5]
 28504:	symbol=_IO_2_1_stdin_;  lookup in file=cat [0]
 28504:	symbol=_IO_2_1_stdin_;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_IO_2_1_stdin_' [GLIBC_2.2.5]
 28504:	
 28504:	relocation processing: cat (lazy)
 28504:	symbol=__gmon_start__;  lookup in file=cat [0]
 28504:	symbol=__gmon_start__;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	symbol=__progname;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__progname' [GLIBC_2.2.5]
 28504:	symbol=__progname_full;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__progname_full' [GLIBC_2.2.5]
 28504:	symbol=optind;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `optind' [GLIBC_2.2.5]
 28504:	symbol=stderr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `stderr' [GLIBC_2.2.5]
 28504:	symbol=stdout;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `stdout' [GLIBC_2.2.5]
 28504:	
 28504:	relocation processing: /lib64/ld-linux-x86-64.so.2
 28504:	symbol=_r_debug;  lookup in file=cat [0]
 28504:	symbol=_r_debug;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=_r_debug;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_r_debug' [GLIBC_2.2.5]
 28504:	symbol=free;  lookup in file=cat [0]
 28504:	symbol=free;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
 28504:	symbol=__libc_memalign;  lookup in file=cat [0]
 28504:	symbol=__libc_memalign;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__libc_memalign' [GLIBC_2.2.5]
 28504:	symbol=malloc;  lookup in file=cat [0]
 28504:	symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc' [GLIBC_2.2.5]
 28504:	symbol=__tls_get_addr;  lookup in file=cat [0]
 28504:	symbol=__tls_get_addr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	symbol=__tls_get_addr;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `__tls_get_addr' [GLIBC_2.3]
 28504:	symbol=calloc;  lookup in file=cat [0]
 28504:	symbol=calloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `calloc' [GLIBC_2.2.5]
 28504:	symbol=realloc;  lookup in file=cat [0]
 28504:	symbol=realloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `realloc' [GLIBC_2.2.5]
 28504:	symbol=free;  lookup in file=cat [0]
 28504:	symbol=free;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
 28504:	
 28504:	calling init: /lib64/ld-linux-x86-64.so.2
 28504:	
 28504:	
 28504:	calling init: /lib/x86_64-linux-gnu/libc.so.6
 28504:	
 28504:	symbol=gettimeofday;  lookup in file=cat [0]
 28504:	binding file cat [0] to cat [0]: normal symbol `gettimeofday' [LINUX_2.6]
 28504:	symbol=clock_gettime;  lookup in file=cat [0]
 28504:	binding file cat [0] to cat [0]: normal symbol `clock_gettime' [LINUX_2.6]
 28504:	symbol=__libc_start_main;  lookup in file=cat [0]
 28504:	symbol=__libc_start_main;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__libc_start_main' [GLIBC_2.2.5]
 28504:	
 28504:	initialize program: cat
 28504:	
 28504:	
 28504:	transferring control: cat
 28504:	
 28504:	symbol=getpagesize;  lookup in file=cat [0]
 28504:	symbol=getpagesize;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `getpagesize' [GLIBC_2.2.5]
 28504:	symbol=strrchr;  lookup in file=cat [0]
 28504:	symbol=strrchr;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `strrchr' [GLIBC_2.2.5]
 28504:	symbol=setlocale;  lookup in file=cat [0]
 28504:	symbol=setlocale;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `setlocale' [GLIBC_2.2.5]
 28504:	symbol=malloc;  lookup in file=cat [0]
 28504:	symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc' [GLIBC_2.2.5]
 28504:	symbol=free;  lookup in file=cat [0]
 28504:	symbol=free;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
 28504:	symbol=bindtextdomain;  lookup in file=cat [0]
 28504:	symbol=bindtextdomain;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `bindtextdomain' [GLIBC_2.2.5]
 28504:	symbol=textdomain;  lookup in file=cat [0]
 28504:	symbol=textdomain;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `textdomain' [GLIBC_2.2.5]
 28504:	symbol=__cxa_atexit;  lookup in file=cat [0]
 28504:	symbol=__cxa_atexit;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__cxa_atexit' [GLIBC_2.2.5]
 28504:	symbol=getopt_long;  lookup in file=cat [0]
 28504:	symbol=getopt_long;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `getopt_long' [GLIBC_2.2.5]
 28504:	symbol=__fxstat;  lookup in file=cat [0]
 28504:	symbol=__fxstat;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `__fxstat' [GLIBC_2.2.5]
 28504:	symbol=malloc;  lookup in file=cat [0]
 28504:	symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc' [GLIBC_2.2.5]
 28504:	symbol=read;  lookup in file=cat [0]
 28504:	symbol=read;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
 28504:	binding file cat [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `read' [GLIBC_2.2.5]

The output of the command is also very useful for tracking performance problems for excessive dynamic loading etc. So it is a very simple but extremely useful tool for debugging linux ld problems!