Yes. Ask it to do it 10 times and pick the right answer
That only works if you assume the fail cases are uncorrected. Spoiler alert: they are not.
Ask 10 different models then
Same problem: The models are also correlated on what they can and can't solve.
To give you an extreme example, I can ask 1000000 different models for a counterexample to the 3n + 1 problem, and all will get it wrong.
No. What a bizarre example to choose. This is so easy to demonstrate. They will all come back with the exact same correct answer
If it's so easy, go do it. You can publish the result in any math journal you like with just a title and a number, because this is one of the hardest problems in mathematics.
For reference: https://en.wikipedia.org/wiki/Collatz_conjecture
My guy, every LLM has read Wikipedia
I don't know if you're purposely being dense. The first sentence of Wikipedia is that this is a famous unsolved problem.
So no, sampling 1000000 LLMs will not get you a solution to it. I guarantee you that.
It will get you the correct answer, not a solution. Once again it’s a terrible example, I don’t know why you used it. It’s certainly not a gotcha
The reason I used it is that the correct answer to the actual problem is unknown and nobody has any idea how to solve it. No amount of sampling an LLM will give you a correct answer. It will give you the best known answer today, but it won't give you a correct answer. This is an example where LLMs all give correlated answers that do not solve the problem.
If you want to scale back, many programming problems are going to be like this, too. Failure points of different models are correlated as much as failure points during sampling are correlated. You only gain information from repeated trials when those trials are uncorrelated, and sampling multiple LLMs is still correlated.
the correct answer is "the solution is unknown"
That's not what I asked the LLM for. I asked it for a counterexample, not whether a counterexample is currently known to humans.
Is that the correct answer to "write a lock-free MPMC queue"? That is a coding problem that literally every LLM gets wrong, but has several well-known solutions.
There's merit to "I don't know" as a solution, but a lot of the knowledge encoded in LLMs is correlated with other LLMs, so more sampling isn't going to get rid of all the "I don't knows."