Where exactly do you mine "dirt" from on a planet that hasn't had any sort of biosphere to create it? Dirt isn't just "stuff you're standing on" - it contains a lot of stuff that's been decomposing.

Plus, Mars is full of chemical compounds that we suspect are pretty terrible for you, and the Moon's regolith is full of jagged, sharp bits that aren't good for you or the plants they're growing in.

Mining "dirt" from most of the places we can get to would do as good of a job of growing food as grinding up your windows & plumbing.