Performing controlled user experiments on small devices in naturalistic
mobile settings has always proved to be a difficult undertaking for many Human
Factors researchers. Difficulties exist, not least, because mimicking natural
small device usage suffers from a lack of unobtrusive data to guide
experimental design, and then validate that the experiment is proceeding
naturally.Here we use observational data to derive a set of protocols and a
simple checklist of validations which can be built into the design of any
controlled experiment focused on the user interface of a small device. These,
have been used within a series of experimental designs to measure the utility
and application of experimental software. The key-point is the validation
checks -- based on the observed behaviour of 400 mobile users -- to ratify that
a controlled experiment is being perceived as natural by the user. While the
design of the experimental route which the user follows is a major factor in
the experimental setup, without check validations based on unobtrusive observed
data there can be no certainty that an experiment designed to be natural is
actually progressing as the design implies.Comment: 12 pages, 3 table
