All 32-bit binaries crash with glibc 2.27

[Lekensteyn]

All 32-bit applications currently crash with glibc 2.27 as demonstrated by:

$ echo 'int main(){}' > empty.c
$ clang -g -fsanitize=address -m32 empty.c -o empty && ./empty
=================================================================
==27867==ERROR: AddressSanitizer: SEGV on unknown address 0xf7f70020 (pc 0xf7f70034 bp 0xffa9ac1c sp 0xffa9abdc T16777215)
==27867==The signal is caused by a WRITE memory access.
    #0 0xf7f70033 in _dl_get_tls_static_info (/lib/ld-linux.so.2+0x12033)
    #1 0x567081cb in __sanitizer::InitTlsSize() (main+0x11b1cb)
    #2 0x566e7cd2 in __asan::AsanInitInternal() [clone .part.0] (main+0xfacd2)
    #3 0xf7f6da57 in _dl_init (/lib/ld-linux.so.2+0xfa57)
    #4 0xf7f5ec99  (/lib/ld-linux.so.2+0xc99)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/ld-linux.so.2+0x12033) in _dl_get_tls_static_info
==27867==ABORTING

For some reason the calling convention has changed from regparm(2) to cdecl. It has likely something to do with https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=commit;h=17e00cc69eac2ec10ac69a3f85db0dffc9d73845

Environment: Arch Linux x86_64 with lib32-glibc 2.27-3. Working version is lib32-glibc 2.26-11.
Originally found while running check-all LLVM tests, all 781 i386 tests fail, plus:

DataFlowSanitizer-x86_64 :: custom.cc
LeakSanitizer-AddressSanitizer-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
MemorySanitizer-X86_64 :: Linux/sunrpc.cc
MemorySanitizer-X86_64 :: Linux/sunrpc_bytes.cc
MemorySanitizer-X86_64 :: Linux/sunrpc_string.cc
MemorySanitizer-X86_64 :: dtls_test.c
SanitizerCommon-lsan-x86_64-Linux :: Posix/sanitizer_set_death_callback_test.cc
ThreadSanitizer-x86_64 :: sunrpc.cc

Related topic: #914

[kcc]

The discussion with the glibc folks died out. I will ping, but I don't have high hopes.
My team does not have resources to chaise all glibc versions especially if the glibc team does not cooperate.

[fweimer]

Jakub fix this in the GCC copy of the code: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84761

[kcc]

.. and looks like the same change has never been applied to clang
https://reviews.llvm.org/D44623

As we agreed many years ago, GCC variant of run-time should be a verbatim copy of one in LLVM
and the changes should go to LLVM first.
It's sad that we don't follow this agreement any more.

[krytarowski]

I've fixed ASan for NetBSD/i386 https://reviews.llvm.org/D46585

[kcc]

This is not submitted, nor even posted properly (llvm-commits was not CC-ed, I also didn't know about this patch).
Whoever wants this bug to be fixed, please ensure that LLVM and GCC implementations match again.

[Lekensteyn]

@krytarowski is that related to this issue or the other? I am not sure if NetBSD uses glibc.

Is it OK to fix LLVM and then retroactively change GCC? I would prefer making progress in LLVM and not depend on GCC for fixing this. D44623 seems a good start.

There is another version check slightly below in InitTlsSize, it parses confstr(_CS_GNU_LIBC_VERSION). Would it be worth pulling that into a separate function and reuse it? Then the dlvsym call could be dropped. Oh, Jakub also notes this in the review comments of D446223:

glibc 2.27 has added the glob@@GLIBC_2.27 symbols approx. one month after these internal_function changes, so it is closer to that than e.g. trying to parse confstr for glibc 2.27 and later, and is also smaller than trying to parse the confstr string.

Regarding the synchronization between GCC/LLVM wrt compiler-rt, I have no idea about that. Perhaps @fweimer could comment on that?

[kcc]

Our contract between LLVM and GCC is (has been?) that all changes to sanitizer run-times are first made to the LLVM tree. Then GCC pulls them whenever they need to.
If this contract is not respected by the GCC team any more (is it?) -- I am sorry, but I have no control over it.

[krytarowski]

@Lekensteyn I've noted issues with TLS and i386 and though it will be worth to share for your interest. We can use GLIBC, but it's non-orthodox :)

If it's totally unrelated, please disregard.

[Lekensteyn]

Perhaps it is a misunderstanding and not an intentional contract violation. Let's see what GCC/@jakubjelinek has to say about the comment in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84761#c12

[Lekensteyn]

This should be fixed now by "[ASAN] Fix crash on i?86-linux (32-bit) against glibc 2.27 and later" as proposed by Jakub. It requires compiler-rt to match the glibc version in use (and otherwise it would crash again).

For more test results, see https://reviews.llvm.org/D44623#1126736 (independent of this patch, there were already some test failures).

---

Buildbot failures post-merge

All 2 failures are flaky and a rebuild was successful.

sanitizer-x86_64-linux-bootstrap-msan

failed annotate failed check-lld msan http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap-msan/builds/5208
Verdict: flaky test? Rebuild passed without errors: http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap-msan/builds/5209

Output (frames 3 and 4 were manually symbolized based on libc6-dbg 2.24-11+deb9u3)

FAIL: lld :: ELF/strip-all.s (1494 of 1697)
==47305==MemorySanitizer CHECK failed: /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc:180 "((kBlockMagic)) == ((((u64*)addr)[0]))" (0x6a6cb03abcebc041, 0x0)
    #0 0x58c13b in MsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/projects/compiler-rt/lib/msan/msan.cc:387
    #1 0x57c5b5 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79
    #2 0x57361f in __sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc:180
    #3 0x7f35023b593f in __run_exit_handlers /build/glibc-yWQXbR/glibc-2.24/stdlib/exit.c:83
    #4 0x7f35023b5999 in exit /build/glibc-yWQXbR/glibc-2.24/stdlib/exit.c:105
    #5 0x7f35023a02e7 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e7)
    #6 0x572039 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/lld+0x572039)

https://sources.debian.org/src/glibc/2.24-11+deb9u3/stdlib/exit.c/#L83

clang-cmake-aarch64-full

failed ninja check 2 http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/builds/5244
Verdict: flaky test, successful rebuild: http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/builds/5245

# ninja check-all
FAIL: DataFlowSanitizer-aarch64 :: custom.cc (20884 of 42190)
--
/home/buildslave/buildslave/clang-cmake-aarch64-full/llvm/projects/compiler-rt/test/dfsan/custom.cc:817:39: warning: data argument not used by format string [-Wformat-extra-args]
  assert(sprintf(buf, "Hello world!", 42, "hello") == 12);
                      ~~~~~~~~~~~~~~  ^
/usr/include/assert.h:89:5: note: expanded from macro 'assert'
  ((expr)                                                               \
    ^~~~
1 warning generated.
==3067==WARNING: DataFlowSanitizer: call to uninstrumented function htonl
==3067==WARNING: DataFlowSanitizer: call to uninstrumented function signal
==3067==WARNING: DataFlowSanitizer: call to uninstrumented function pthread_join
==3067==WARNING: DataFlowSanitizer: call to uninstrumented function strcat
[..]
==3067==WARNING: DataFlowSanitizer: call to uninstrumented function strcat
/home/buildslave/buildslave/clang-cmake-aarch64-full/llvm/projects/compiler-rt/test/dfsan/custom.cc:817:39: warning: data argument not used by format string [-Wformat-extra-args]
  assert(sprintf(buf, "Hello world!", 42, "hello") == 12);
                      ~~~~~~~~~~~~~~  ^
/usr/include/assert.h:89:5: note: expanded from macro 'assert'
  ((expr)                                                               \
    ^~~~
1 warning generated.
/home/buildslave/buildslave/clang-cmake-aarch64-full/stage2/projects/compiler-rt/test/dfsan/AARCH64Config/Output/custom.cc.script: line 4:  4631 Segmentation fault      (core dumped) DFSAN_OPTIONS="strict_data_dependencies=0" /home/buildslave/buildslave/clang-cmake-aarch64-full/stage2/projects/compiler-rt/test/dfsan/AARCH64Config/Output/custom.cc.tmp