Field Experiments To Understand Mobility

This paper recommends "in the field" usability testing. They indicate that some variables cannot and should not be controlled, and that in-situ testing is needed for mobile devices. It offers advice on how to run mobile usability tests and encourages the use of a number of different metrics to track behavior.

"It is possible to attempt to keep the levels of such variables consistent, for example, by carefully scheduling and re-scheduling experiments and removing data from runs that did not stay within appropriate control levels. However, this can consume much time and resources and may prove irritating to participants as well as experimenters. In addition, we believe that removing all variation would produce unrealistic results which may not mean much for real-world usage. Therefore, a more effective method is to let these variables vary across conditions, as they would in real world use."

Goodman, J., Brewster, S. and Gray, P. Using Field Experiments to Evaluate Mobile Guides. in Schmidt-Belz, B.a.C., K. ed. HCI in Mobile Guides, workshop at Mobile HCI 2004, Glasgow, UK, 2004.
pdf