May be it's my luck but I found a glaring issue with Gemini 2.5 Pro in AI Studio.
I asked it whether a language feature in Zig was available. It answered yes and proceeded to generate a whole working sample code. I compiled it and got an error. Reported the error and it said the error showed I typed it wrong and asked me to make sure it's typed correctly. Eh?! It's a copy-and-paste. I confirmed again it's wrong. It then said it must be my compiler version was too old. Nope, using the latest. It then said very convincingly that based on its extensively research into the language official documentation, official examples, and release notes, the feature must exist. I asked it to show me the reference materials it used to draw the conclusion. None of links it gave were valid. I told it they were wrong. It gave back another set of links and claimed it had checked the links to make sure they are alive. The links were alive but didn't contain any mention of the feature. I let it know again. It admitted couldn't find the mentioned feature. But it insisted the feature had been merged in a PR. The PR link it gave was unrelated. I let it know. It gave me another 3 PR's and said one mentioned something related so the feature must be in. At the point I gave up.
The issue was that it sounded very convincing and stated "facts" very confidently, with backings to documents and other resources even if they were wrong or irrelevant. Even when told it gave the wrong info, it would double down and made up some BS reference material to back up its claim.
Generative AI makes things up so I'm surprised that you seem surprised. For some situations checking the documentation is still the best option.
I know LLM’s hallucinate. I’m surprised how convincing and how stubborn it insisted it’s right. Other LLM’s would have given up and admit they don’t know.
The other day I wanted to use a function in an unfamiliar library. Gemini kept putting an argument into the call that wasn't supposed to be there even after I explicitly told it not to. This was after I trusted Gemini, got an error message and looked at the docs. The argument it was adding was required for other functions in other libraries the same subject area so I suppose that's why it did it.
Wrong library version is also a classic.
At some point it would be nice if someone could come up with a way of grounding/adding package docs and/or version as part of the context automatically
Gemini told someone I know that golang have manual memory management and was very adamant about it.