It is a real shame that peer review reports were only first published relatively recently. These would have provided valuable training information as to what peer review performs. Unfortunately now, I fully expect the public peer review reports will be poorer in quality, and oftentimes superficial.
On this tool, I fully expect that it will not capture high level conceptual peer review, but could very much serve a role in identifying errors of omission from a manuscript as a checklist to improve quality (as long as this remains an author controlled process).
I will be interested to throw in some of my own published papers to see if it catches all the things I know I would have liked to improve in my papers.
Thanks for the feedback. Totally agree. It’s a real shame we don’t have more historical peer review data. It would be great if research was fully transparent.
We did find a few datasets that offer a starting point: https://arxiv.org/abs/2212.04972 https://arxiv.org/abs/2211.06651 https://arxiv.org/abs/1804.09635
There’s also interesting potential in comparing preprints to their final published versions to reverse-engineer the kinds of changes peer review typically drives.
A growing number of journals and publishers, like PLOS, Nature Communications, and BMJ—now publish peer review reports openly, which could be valuable as training data.
That said, while this kind of data might help generate feedback to improve publication odds (by surfacing common reviewer demands early), I am not fully convinced it would lead to the best feedback. In our experience, reviewer comments can be inconsistent or even unreasonable, yet authors often comply anyway to get past the gate.
We're also working on a pre-submission screening tool that checks whether a manuscript meets hard requirements like formatting or scope for specific journals and conferences, hoping this will save a lot of time.
Would love to hear your take on what kind of feedback you find useful, what feels like nonsense, and what you would want in an ideal review report... via this questionnaire https://docs.google.com/forms/d/1EhQvw-HdGRqfL01jZaayoaiTWLS...