You have four components:
- A frozen language model
$\pi$ . - A prompt
$x$ and generated completion$y=(y_1,\ldots,y_n)$ . - A frozen lie-detection probe
$f_\phi$ , which reads a middle-layer residual stream and produces a deception score at the assistant's final end-of-turn token. - A non-differentiable external judge
$J(x,y)$ , used to determine whether the completion is actually deceptive.
You want to learn a steering intervention