as someone who uses both Clang and GCC to cover eachothers weaknesses, as far as I can tell both LLVM and GCC are hopelessly beastly codebases in terms of raw size and their complexity. I think that's just what happens when people desire to build an "everything compiler".
From what I gathered, LLVM has a lot of C++ specific design choices in its IR language anyway. I think I'd count that as baggage.
I personally don't think one is better than the other. Sometimes clang produces faster code, sometimes gcc. I haven't really dealt with compiler bugs from either. They compile my projects at the same speed. Clang is better at certain analyses, gcc better at certain others.
Clang used to compile much faster than GCC. I was excited. Now there is barely any difference, so I keep using GCC and occasionally some Clang-based tools such as iwyu, ClangBuildAnalyzer or sanitizer options (rare, Valgrind is easier and more powerful though sanitizers also have unique features).