But then .. stuff like this happens, and now of course people's priors update that all LLMs in all configurations can't be trusted with sourcing.
It's like the 1980's with 8-bit machines, or the 1990's with the introduction of the internet. One half of the world thinks it's a fad. And to be fair! They keep getting confirmation because of course the new tech fails in weird and interesting ways.
The other half meanwhile quietly gets on, quietly patches the bugs as they are discovered, and do get meaningful work out of it.
Neither the hype nor the skeptical version are entirely correct.
Anyone with any meaningful experience with using machines should know better. I suspect that people know that machines break and fail they're just lazy and are willing to hand off their work to a computer as often as they can get away with it. Often an excuse of "Not my fault, the computer did it" works because computers are widely known to be barely working and prone to errors.
It's an amazing feat of marketing that people trust AI when there are countless examples of AI being wrong (often hilariously), but in the case of lawyers I think the only solution is disbarment. It won't take many lawyers losing their license before other lawyers start doing the job they're being paid for again.
I think this should be "experience with making or maintaining machines" instead. It's those cases when you don't get to be lazy--you have to fix the machine yourself. As long as you're just the user, it's someone else's responsibility if your machine-aided work is wrong.
Yes, we can point out how that's not actually correct until we're blue in the face. But in practice, the way we have set up our economy and our institutions, it is correct.
It does not matter if AI produces it or his second cousin, visiting from Antigua.