There is much to learn from what Turing hastily dismissed as Lady Lovelace s
objection. Digital computers can indeed surprise us. Just like a piece of art,
algorithms can be designed in such a way as to lead us to question our
understanding of the world, or our place within it. Some humans do lose the
capacity to be surprised in that way. It might be fear, or it might be the
comfort of ideological certainties. As lazy normative animals, we do need to be
able to rely on authorities to simplify our reasoning: that is ok. Yet the
growing sophistication of systems designed to free us from the constraints of
normative engagement may take us past a point of no-return. What if, through
lack of normative exercise, our moral muscles became so atrophied as to leave
us unable to question our social practices? This paper makes two distinct
normative claims:
1. Decision-support systems should be designed with a view to regularly
jolting us out of our moral torpor.
2. Without the depth of habit to somatically anchor model certainty, a
computer s experience of something new is very different from that which in
humans gives rise to non-trivial surprises. This asymmetry has key
repercussions when it comes to the shape of ethical agency in artificial moral
agents. The worry is not just that they would be likely to leap morally ahead
of us, unencumbered by habits. The main reason to doubt that the moral
trajectories of humans v. autonomous systems might remain compatible stems from
the asymmetry in the mechanisms underlying moral change. Whereas in humans
surprises will continue to play an important role in waking us to the need for
moral change, cognitive processes will rule when it comes to machines. This
asymmetry will translate into increasingly different moral outlooks, to the
point of likely unintelligibility. The latter prospect is enough to doubt the
desirability of autonomous moral agents