It will get you the correct answer, not a solution. Once again it’s a terrible example, I don’t know why you used it. It’s certainly not a gotcha
The reason I used it is that the correct answer to the actual problem is unknown and nobody has any idea how to solve it. No amount of sampling an LLM will give you a correct answer. It will give you the best known answer today, but it won't give you a correct answer. This is an example where LLMs all give correlated answers that do not solve the problem.
If you want to scale back, many programming problems are going to be like this, too. Failure points of different models are correlated as much as failure points during sampling are correlated. You only gain information from repeated trials when those trials are uncorrelated, and sampling multiple LLMs is still correlated.
the correct answer is "the solution is unknown"
That's not what I asked the LLM for. I asked it for a counterexample, not whether a counterexample is currently known to humans.
Is that the correct answer to "write a lock-free MPMC queue"? That is a coding problem that literally every LLM gets wrong, but has several well-known solutions.
There's merit to "I don't know" as a solution, but a lot of the knowledge encoded in LLMs is correlated with other LLMs, so more sampling isn't going to get rid of all the "I don't knows."