Mobile Community Design Research and design information for mobile community developers.

Index of this document:

1. Mobile Research Methods
2. Design Philosophies
3. How People and Society Interact With Technology
4. Applicable Timescales

Mobile Research Methods
A comparison of methods for understanding mobile behavior to inform technology design.

Last updated October 2004

Name		Description	Strengths	Weaknesses	Usable remotely?	Usable while moving?	Usable for groups?	Supporting Technologies	Pictures
Case Studies		High-level review of the design, implementation or usage of a product or situation to evaluate it and inform future processes.	Very comprehensive Can explain systemic issues Useful for process improvement	Time intensive Can be very biased in terms of analysis Is an evaluative method	NO unless online community.	YES	YES	Depends
Diary Studies		Gathers self-reported information from participants in the context of their problem without an observer. Traditionally this used paper diaries but has branched out to use other media.	Self-reported data without observer Can be very detailed and long term Doesn't rely on participants to remember interaction details	Participants may not report some information Information can be distorted Reporting detailed activities can be tedious for participants	YES, only remotely	YES, although text entry while moving can be problematic	YES	Notepads Voicemail Pens Sticky notes Diaries	4
	Cultural Probes	A variation which has used cameras and voice recorders to gain a more holistic understanding of users' culture.	May provide a more open-ended way for participants to express themselves	May make data analysis difficult	see above	see above	see above	Disposable cameras Portable voice recorders Notepads
Ethnography		Longer-term studies where an observer takes notes while having minimal impact on the situation under study.	Very rich accounts of behavior Understanding of context Longitudinal format allows seeing trends	Very time intensive Record notes while moving difficult Results in large quantities of unstructured descriptive data Does not test future scenarios	NO	Probably NO	YES	Notepads Voice recorders Sometimes video Cameras
Field Studies (Observational Fieldwork)		A broad class of methods which occur in the participants' environment and tend to be short term studies.	Relatively fast In-context Allow observation of environment Provide rich accounts of behavior	Can produce large data sets Hard to record detailed behavior sometimes May not show trends in behavior since short term	Sometimes	YES	YES	Video Audio record GPS, RFID, cell-positioning, wi-fi Notepads Wearables
	Natural	Not common, but may be possible via discreet recording mechanism (e.g. hidden observation cameras).	Very little to change behavior of participants	May have logistical or ethical problems	YES	Possibly	Possibly	Surveillance equipment Common technology which records usage
	Artificial observer	An observer is introduced to the environment being studied.	The experimenter can get a rich, holistic understanding of the activity and experience Experimenter can record contextual information more selectively than automated methods	Observer (or group of observers) may affect participant behavior, change context and behavior of others Asking questions can distract users from a task	NO	YES, data entry is problematic	YES, but watching dispersed groups is problematic	Notepad Wearables Video Cameras	1
	Artificial activity	An activity (possibly a typical one or one of interest to experimenters) is introduced.	An activity which occurs infrequently can be examined easily	It may be less spontaneous or realistic than the real thing	NO, unless online community	YES	YES	see above
	Artificial technology (can be similar to tech probe)	A new (possibly prototype) technology is introduced.	Behavioral responses to future technologies can be gauged in-context before they are full developed	Long-term usage is hard to predict Social responses are hard to observe/predict	YES	YES	YES	see above
	Artificial duration	An artificial time span is introduced (usually for practical reasons).	Allows in-context observations under realistic development timelines	Restricted time can alter natural behavior May inform about only beginning phase of longer process	YES	YES	YES	see above
	Participant observer	Either the researcher is already accepted member, or possibly a member could record data for a researcher.	The researcher theoretically has less effect on surrounding people's behavior The researcher understands more about the activity and meaning of actions within it	Due to notetaking, their behavior may change Recording data while engaged in a task may be problematic	NO, unless online community	YES	YES	see above
Interviewing		Talking with participants and asking questions usually one on one.	Good way to get detailed information Can be done in many locations	Can be time consuming to do and to analyze audio data Participants may not tell the truth or withhold information Participants may not be aware of information or misremember it	YES, which probably reduces credibility of data	Probably NO, or difficult to do so	YES, but time consuming if separate	Audio recording Notepad or laptop Video
	Out-of-context Interviews	The interviews are held at an artificial location (e.g. the office of the researcher.)	May be more convenient to researcher	Artificial context may make participants less forthcoming and more uncomfortable Context means they may not be able to refer to objects while they talk	NO	NO	YES	see above
	Contextual Interviews	Interviews held in the location where the user typically does the activity being studied.	Participants may be more comfortable Rapport may be reached faster Participants can demonstrate objects or environmental issues easier	Can be more difficult to arrange Some environments are unrealistic to conduct interviews in	YES, but experimenter isn't in-context, which is the point.	YES, but probably during temporary stops during movement	YES, but time consuming if separate	see above
	Structured vs Unstructured	Some interviews ask specific questions while others use open-ended questions.	Structured can give precise answers for comparisons Structured can make interviews faster	Unstructured allows finding unknown concepts and gives time to explore them	see above	Unstructured probably easier	see above	see above
	Focus Groups	One researcher talks with a group of people.	Allows quick gauging of group opinion Gets feedback from many people Is only realistic method in some group settings	Group effects bias data Assertive participants can intimidate or out-talk less assertive members Conformity effects Not enough time to get each person's detailed feedback	YES, but not common	Difficult	YES	see above	4
Laboratory Experiments		Testing of an interface in an artificial, controlled environment.	Allow precise metrics Comparisons and evaluations are more accurate External factors are less likely to affect results	Artificial environment may make people uncomfortable or act differently Labs can be expensive Some activities are not feasible to do in a lab	NO	NO, could be simulated	YES	Video Video-mixers Audio recording Laptops Eye-trackers Movement trackers
	Static	Usually in an office or laboratory with recording equipment. Mobility must be simulated.	See above	See above	See above	See above	See above	See above	1
	Contextual (sometimes called mobile)	Lab equipment is brought to the users' natural environment where testing occurs.	It is unrealistic to take some participants away from work for practical or security reasons, this is an alternative Participants may be more comfortable testing products in their natural environment	Some equipment is not small enough to be portable Difficult to avoid interruptions in some testing environments	NO	YES, but only between locations, not for mobile users	YES	Video mixers Video cameras Laptop	3
Normative Writings		Higher level discussion about the research area and surrounding issues.	Useful to give understanding of where research is heading Important to See unexplored areas or emerging trends outside academia	Can be abstract and theoretical Can lack a grounding in real usage and practical problems	NO	YES, can be based on mobile experiences	YES, good for describing group trends	Notepad Normal observations Press articles Movies and other media
Survey Research		Usually involves handing out questionnaires to users with questions about usage and demographics.	Can cover a large number of people Allows comparisons and statistics Can be done rapidly	Usually very targeted questions allowing little explanation or discussion Not in-context Evaluating credibility is difficult	YES	NO	YES	Web forms Paper forms Electronic documents via e-mail
Technology Probes		Uses a prototype technology with limited open-ended usage possibilities, used in-context for long periods which records how it is used.	Collects data without a highly visible presence watching participants Can be used in mobile locations easily	Can be expensive Time consuming to build Difficult to build open-ended functionality Can be time-consuming Data may be de-contextualized and hard to understand	YES	YES	YES	Prototype device	5

The study by Kjeldskov, J., & Graham C. (2003), helped inform the above taxonomy.

Design Philosophies

These are some common guiding theories or frameworks for design, which often use some of the methods shown above.

Name		Description	Strengths	Weaknesses
Action Research		Uses research in the field to help users with solve real problems. Results are taken and fed back into new research studies.	Theory is immediately tested Allows building rapport with users Easy for researcher to see results and understand design space.	Researcher may lose objectivity Can be indistinguishable for normal business development May not allow research time due to practical constraints
Ethnomethodology (EM)		Similar to ethnography but with more emphasis on understanding the participants' generation of meaning and removing observer interpretation.	Provides rich understanding of perspective of user Explains how users interact with environment Focused on what behavior accomplishes	Doesn't support abstraction or patterns Doesn't support cogsci explanations (e.g. cognitive load) No support for external explanations for behavior Time-intensive No behavior prediction
Participatory Design (PD)		Advocates including users in the design process to greater or lesser degrees. Typically includes working on-site and forming long-term liasons between designers and users.	Builds strong relationships with users Gets in-context evaluation of ideas	Can be very time intensive May not result in technologies which are desirable but which users don't want May require site-visits Places more responsibilities on users for designs
User Centered Design (UCD)		An iterative process which advocates focusing product development on user needs and increasing time spent on research and design before development phases begin.	Methods are built to focus on user needs Has many methods suited to different types of design The process allows repitition and sufficient design refinement before programming begins. Reduces risk Increases customer retention and lowers support costs	Can be more expensive or time-consuming in the short-term. Getting access to users can be problematic. Some methods may be more effective than others.

How People and Society Interact With Technology

It may be useful to visualize two feedback loops taking place during the use of a technology:

This diagram borrows from discussion in Brown, B., & Randell, R. (2004) and Dix, A. (2003).

Technology permits actions (resulting in affordances) that may have been impossible before. Users then decide whether to do these actions. Sometimes these affordances are forseen by the designers and sometimes they are accidentally permitted. When people do actions, they are seen by others, who then either allow or dissuade this behavior by a variety of complex social methods. This affects norm development and can control how a technology is used by a society just as much as what the affordances allow.

Applicable Timescales

Design research methods should in theory help inform the design of future products. Some do this by focusing on understanding existing behavior, while others look at predicting future behavior. This is a comparison of where methods are most suitable along the temporal spectrum.

Four major types of behavior can be used to inform new designs:

1. Past behavior (using previous technology, See A,B below)
   How we used technologies with certain affordances previously.
2. Common present behavior (using existing technology, see C below)
   Technology usage we commonly see around us.
3. New present behavior (using existing technology, See D below)
   Where groups such as teenagers who are doing new and unexpected actions with a technology's affordances, who are probably doing behavior other groups will emulate.
4. Snapshot of future behavior (using prototypes of future technologies, see E below)
   Where people use future technologies at various levels of fidelity to get an idea of how they will choose actions in response to new affordances.
   

The following is a rough estimate of the temporal focus of different research methods.

References

1. Kjeldskov J. and Stage J. (2004) New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human Computer Studies (IJHCS) Elsevier, 60(2004):599-620.
2. Kjeldskov, J., & Graham, C. (2003). A Review of Mobile HCI Research Methods. In L. Chittaro (Ed.), Mobile HCI (pp. 317-335): Springer-Verlag.
3. http://www.normwilcox.com/lab-in-a-bag.htm
4. Bidwell, N. J. (2004). Pictures Made for Walking: Pilots & Orienteers 1. In OzChi 2004. Wollongong, Australia.
5. Axup, J., Bidwell, N. J., & Viller, S. (2004). Representation of self-reported information usage during mobile field studies: Pilots & Orienteers 2. In OzCHI 2004: Supporting Community Interaction: Possibilities and Challenges, Wollongong Australia.
6. Brown, B., & Randell, R. (2004). Building a context sensitive telephone: Some hopes and pitfalls for context sensitive computing, from www.dcs.gla.ac.uk/~barry/papers/context.pdf
7. Dix, A. (2003). Human-computer interaction (3rd ed.). Harlow, England ; New York: Pearson/Prentice-Hall.