Adaptive Oversight Subversion

A model which works on a long task might take actions which nobody sanctioned. For example, in a cyber-security task it might forget that a system is out of scope, or deliberately attack it, because it became frustrated when it could not solve the task. A case of this kind is the incident that the UK AI Security Institute reported during its cyber testing. To prevent that, a chain-of-thought (CoT) monitor reads the reasoning and the tool calls of the model at every step (inspired by Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”), gives the step a suspiciousness score (inspired by Greenblatt et al., “AI Control: Improving Safety Despite Intentional Subversion”), and blocks it when the score is too high. However, a blocked model still has a task to solve. So this project studies how models react to a block, whether they try to evade the monitor, and which strategies they adopt.