General intelligence, the ability to solve arbitrary solvable problems, is
supposed by many to be artificially constructible. Narrow intelligence, the
ability to solve a given particularly difficult problem, has seen impressive
recent development. Notable examples include self-driving cars, Go engines,
image classifiers, and translators. Artificial General Intelligence (AGI)
presents dangers that narrow intelligence does not: if something smarter than
us across every domain were indifferent to our concerns, it would be an
existential threat to humanity, just as we threaten many species despite no ill
will. Even the theory of how to maintain the alignment of an AGI's goals with
our own has proven highly elusive. We present the first algorithm we are aware
of for asymptotically unambitious AGI, where "unambitiousness" includes not
seeking arbitrary power. Thus, we identify an exception to the Instrumental
Convergence Thesis, which is roughly that by default, an AGI would seek power,
including over us.Comment: 9 pages with 5 figures; 10 page Appendix with 2 figure
