Regarding
“When I say an AI A is aligned with an operator H, I mean: A is trying to do what H wants it to do. The “alignment problem” is the problem of building powerful AI systems that are aligned with their operators.”
I take it that an operator can be unaligned. Else all possible operators always certainly are aligned. Therefore,
“In particular, this is the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right. An aligned AI would try to figure out which thing is right, and like a human it may or may not succeed.”
Get I.S. Gesink’s stories in your inbox
Join Medium for free to get updates from this writer.
is then false, in the sense of “in particular this is” - in the case of an unaligned operator.
Shortly paraphrased you cannot simultaneously hold
(1) A follows H and (2) A tries to do good,
without implied and false universal (3) H tries do good.
I suppose I should interpret “right” not as “good” here but as “what H desires”. And then e.g. a nuclear weapon system seems close to qualify as aligned.
