At what point does "knows how to use a calculator" equate to knowing how to do math? Feels pretty close to me...
Well, LLMs are bad at math but they're ok at detecting math and delegating it to a calculator program.
It's kind of like humans.