I solve most of those issues with a Docker Bakefile; I'm confident I could solve the rest with Bakefiles if I had to. Reasonable developer experience.
The last time I tried something more than Buildx that Docker itself put out the experience was bad - something about not properly caching. I'll have to give this another shot sometime, though.