An LLM is essentially playing blindfold chess if it just gets the moves and not the position. You have to be fairly good to never make illegal moves in blindfold.
A chat conversation where every single move is written down and accessible at any time is not the same as blindfold chess.
OK, but the LLM is still playing without a board to look at, except what's "in its head". How often would 1800 ELO chess players make illegal moves when playing only using chess notation over chat, with no board to look at?
What might be interesting is to see if there was some sort of prompt the LLM could use to help itself; e.g., "After repeating the entire game up until this point, describe relevant strategic and tactical aspects of the current board state, and then choose a move."
Another thing that's interesting is the 1800 ELO cut-off of the training data. If the cut-off were 2000, or 2200, would that improve the results?
Or, if you included training data but labeled with the player's ELO, could you request play at a specific ELO? Being able to play against a 1400 ELO computer that made the kind of mistakes a 1400 ELO human would make would be amazing.
MaiaChess [1] supposedly plays at a specific ELO, making similar mistakes a human would make at those levels.
It looks like they have 3 public bots on lichess.org: 1100, 1500, and 1900
You can make it available to the player and I suspect it wouldn’t change the outcomes.
Sure but I'm better than 99% of people at chess and if I was playing under those conditions there is a high chance I would make an illegal move.
The LLM can't refer to notes, it is just relying on its memory of what input tokens it had.
Does it not always have a list of all the moves in the game always at hand in the prompt?
You have to give this human the same log of the game to refer to.
I think even then it would still be blindfold chess, because humans do a lot of "pattern matching" on the actual board state in front of them. If you only have the moves, you have to reconstruct this board state in your head.