Doubtful.
OpenStep used Display Postscript and was written in Objective-C; WebKit is written in C++.
Rendering text on the web is a different animal all together.
I was under the impression that when we got new css in Safari, in the next software cycle those same features ended up in Books. It wouldn't make sense to give it a different rendering engine... but then I've never been able to find much in the way of which epub readers used which rendering engines anywhere.