The short answer is yes, I do mind.
I've taken some steps to mitigate this. I made a design brief (https://gist.github.com/aleshh/7435682311b6cb944bf18ecc3751f...) for LLMs to follow to kind of steer them in a particular direction. And I do spend time tweaking design with them.
But I agree it's not enough. The problem is, there are as I see it really two options here
(1) Guide the LLM and just spend more time on the visual stuff. The trouble is those refactors take as much time as adding major new features, sometimes more, because the LLM doesn't get it right and you gotta go around and around. So it takes a big time investment to go from something that's pretty good to something that's merely somewhat better.
(2) The other option is to do it all by hand. I sometimes do some of this, but the further along you get the harder it is, because the LLMS write lots of CSS, and (at least the last time I tried) refactoring their stuff is time-consuming and kind of a prerequisite to tweaking things.
Probably what I should do is expand on the design spec, and add it to AGENTS.md (instead of just throwing it in on the initial prompt) and generally try to get good on this aspect.
I should note that on my "real" project (https://flipper.fm), I spend a lot more time steering the design, creating a component library, Storybook, creating CSS classes for the LLM to use, etc etc. Again, this thing is a toy.
Sorry for the long answer, but you're really hitting on something real here.