So a Chinese Room?
Tell me you've read Blindsight... and if you haven't, go read it, I'll wait.
... and then read The Freeze-Frame Revolution.
Thanks, I've been looking for some recommendations actually.
(It's been a few months since the last time I rambled aimlessly through my house muttering consciousness is a parasite under my breath.)
There's a sequence of stories by different authors that allude to one another that I find to be an interesting read in that order.
First, there's the BLIT stories by David Langford. Several of these are online.
https://www.infinityplus.co.uk/stories/blit.htm
https://www.nature.com/articles/44964 (did you know that Nature did science fiction short stories? https://www.nature.com/nature/articles?type=futures )
https://www.lightspeedmagazine.com/fiction/different-kinds-o...
Then, you go to Accelerando
https://www.antipope.org/charlie/blog-static/fiction/acceler...
> Luckily, infowar turns out to be more survivable than nuclear war – especially once it is discovered that a simple anti-aliasing filter stops nine out of ten neural-wetware-crashing Langford fractals from causing anything worse than a mild headache.
This is followed by its sequel-ish Glasshouse https://www.goodreads.com/book/show/17866.Glasshouse
It's not technically a sequel, but one can see the universe of Glasshouse following from the ending of Accelerando.
A quick diversion to Vernor Vinge with Peace War and Marooned in Realtime https://www.goodreads.com/series/57273-across-realtime (there's a short story in there titled The Ungoverned)
Implied spaces Walter Jon Williams https://www.goodreads.com/book/show/2059573.Implied_Spaces which takes another approach to the unexplored events leading into Glasshouse and a possible path of escalation. The reference passage in this book is:
> “I and my confederates,” Aristide said, “did our best to prevent that degree of autonomy among artificial intelligences. We made the decision to turn away from the Vingean Singularity before most people even knew what it was. But—” He made a gesture with his hands as if dropping a ball. “—I claim no more than the average share of wisdom. We could have made mistakes.”
This then ends with... {spoilers}.