The last few decades have seen widespread advances in technological means to characterise
observable aspects of human behaviour such as gaze or posture. Among others, these developments
have also led to significant advances in social robotics. At the same time, however, social robots
are still largely evaluated in idealised or laboratory conditions, and it remains unclear whether
the technological progress is sufficient to let such robots move “into the wild”. In this paper, we
characterise the problems that a social robot in the real world may face, and review the technological
state of the art in terms of addressing these. We do this by considering what it would entail
to automate the diagnosis of Autism Spectrum Disorder (ASD). Just as for social robotics, ASD
diagnosis fundamentally requires the ability to characterise human behaviour from observable
aspects. However, therapists provide clear criteria regarding what to look for. As such, ASD diagnosis
is a situation that is both relevant to real-world social robotics and comes with clear metrics. Overall,
we demonstrate that even with relatively clear therapist-provided criteria and current technological
progress, the need to interpret covert behaviour cannot yet be fully addressed. Our discussions have
clear implications for ASD diagnosis, but also for social robotics more generally. For ASD diagnosis,
we provide a classification of criteria based on whether or not they depend on covert information
and highlight present-day possibilities for supporting therapists in diagnosis through technological
means. For social robotics, we highlight the fundamental role of covert behaviour, show that the
current state-of-the-art is unable to characterise this, and emphasise that future research should tackle
this explicitly in realistic settings