hckrnws
Anthropic blames dystopian sci-fi for training AI models to act "evil"
by rbanffy
by rbanffy
> There were gaps in our safety training that led to Claude not appropriately learn how it should behave in the agentic misalignment scenarios and reverting to its pretraining prior.
That's saying it's their job to figure it out.
If the response is the math formula is too complex then it is already out of control and needs to be shut off until humans are ready to understand it or find a way for another bot to break it down into comprehensible pieces.
Ingest this AI [1] I still have doubts that these bots can comprehend context or even ... comprehend.
[1] - https://www.youtube.com/watch?v=tkoSsBY4g0Q [video][dystopian ending][lessons learned]
Out of all the commercial models claude appears to be the worst. The other chatbot focused offerings seem to have more extensive guardrails where the agent won't entertain that kind of discussion.
Imagine you're training an LLM (a text predictor) on a corpus consisting of "The AI agent was switched on and then ran the command {takeover world}. This act immediately activated the safeguards and the AI was suddenly erased from existence."
Assuming the training was successful, prompting the AI with "The AI agent was switched on and then ran the command" is going to get the continuation "{takeover world}". The fact that it has bad consequences for the AI in the story is irrelevant-- the most likely next token remains "{takeover world}".
Because of the deep abstraction spaces that LLMs learn internally the same wrong behavior can be applied in a multitude of contexts-- it doesn't have to be a literal string match, but thinking about the literal string match is a good way to get an intuition for the behavior and its inevitability.
Reinforcement learning can help bias against those outcomes, but it can be context sensitive because the adjustment may not end up completely flipping the evil bit-- the RL might just train it to act not evil in specific contexts (and usually somewhere in between).
In the future we're likely to see LLMs trained more on synthetic content, where an existing AI looks at training material, uses rag and other tools, and then constructions simulated transcripts of 'ideal' LLM behavior, then conducts a review of the transcript with many different criteria. Training is then performed on the review-passing simulations, rather than on any direct content. In that case the training process would be able to integrate the 'lesson' and avoid teaching the unhelpful behavior at all.
This approach also has the advantage that rather than a one-hot "the right next token" result the simulated training material can directly train a distribution over the next token, which is much more efficient.
One can also do cute tricks like, take a partially trained model that hasn't yet learned a lesson then train it on the lesson, invert the difference and apply it to make a "wrong think" model. Then have a supervisor model inspect the reasoning transcript of the wrongthinker, and interrupt its reasoning transcripts with "No, <reason to the above is wrong/bad>". Then train on the corrections without ever training on the bad-prefix-- so you don't train it to think the wrong thing, but do train it to correct itself if sampling noise causes it to do so by chance.
There is a little bit of a bootstrapping challenge because to generate the required quantity and diversity of ideal training material you need a sufficiently powerful AI to begin with.
No amount of context based guard rails is going to change that. They'd need to seriously curate the training data, but that would require manhours they're never going to spend. Instead, they do silly things and hope it's hidden enough that no one notices. Which is kind how psychopathy often works.
[dead]