diggan 3 days ago

Has those results been reproduced elsewhere with other benchmarks than what Google seems to use?

Hard to trust their own benchmarks at this point, and Im not home at the moment so cant try it myself either.

1
llm_nerd 3 days ago

They are testing for a very straightforward needle retrieval, as LLMs traditionally were terrible for this in longer contexts.

There are some more advanced tests where it's far less impressive. Just a couple of days ago Adobe released one such test- https://github.com/adobe-research/NoLiMa