This is really brilliant stuff! Somehow I didn't realize that logprobs were being returned as part of the OAI requests, and I really like this application of it.
Any interest in seeing this sort of thing being added to llama.cpp?
Looking at llama.cpp, it already supports the logprob field in its OAI API emulation, so it shouldn't be too difficult to use this library with it.
It feels like this would be useful enough to build around -- I especially like the idea of asking the API to return the top K results for each field, and denoting their likelyhood -- almost like a dropdown box with percentages attached for each possible result.
I believe mikupad[0] supports showing logprobs from a llama.cpp backend.
Thank you for this link -- I had not seen this before. That is an absolutely gorgeous and intuitive interface!