Evaluating HCI research beyond usability

Evaluating research artefacts is an important step to showcase the validity of a chosen approach. The CHI community has developed and agreed upon a large variety of evaluation methods for HCI research; however, sometimes those methods are not applicable or not sufficient. This is especially the case when the contribution lies within the context of the application area, such as for research in sustainable HCI, HCI for development, or design fiction and futures studies. In this SIG, we invite the CHI community to share their insights from projects that encountered problems in evaluating research and aim to discuss solutions for this difficult topic. We invite researchers from all areas of HCI research who are interested to engage in a debate of issues in the process of validating research artefacts

The Revolution of Mobile Phone-Enabled Services for Agricultural Development (m-Agri Services) in Africa: The Challenges for Sustainability

The provision of information through mobile phone-enabled agricultural information services (m-Agri services) has the potential to revolutionise agriculture and significantly improve smallholder farmers’ livelihoods in Africa. Globally, the benefits of m-Agri services include facilitating farmers’ access to financial services and sourcing agricultural information about input use, practices, and market prices. There are very few published literature sources that focus on the potential benefits of m-Agri services in Africa and none of which explore their sustainability. This study, therefore, explores the evolution, provision, and sustainability of these m-Agri services in Africa. An overview of the current landscape of m-Agri services in Africa is provided and this illustrates how varied these services are in design, content, and quality. Key findings from the exploratory literature review reveal that services are highly likely to fail to achieve their intended purpose or be abandoned when implementers ignore the literacy, skills, culture, and demands of the target users. This study recommends that, to enhance the sustainability of m-Agri services, the implementers need to design the services with the users involved, carefully analyse, and understand the target environment, and design for scale and a long-term purpose. While privacy and security of users need to be ensured, the reuse or improvement of existing initiatives should be explored, and projects need to be data-driven and maintained as open source. Thus, the study concludes that policymakers can support the long-term benefit of m-Agri services by ensuring favourable policies for both users and implementers

Evaluation Beyond Usability

The evaluation of research artefacts is an important step to validate research contributions. Sub-disciplines of HCI often pursue primary goals other than usability, such as Sustainable HCI (SHCI), HCI for development, or health and wellbeing. For such disciplines, established evaluation methods are not always appropriate or sufficient, and new conventions for identifying, discussing, and justifying suitable evaluation methods need to be established. In this paper, we revisit the purpose and goals of evaluation in HCI and SHCI, and elicit five key elements that can provide guidance to identifying evaluation methods for SHCI research. Our essay is meant as a starting point for discussing current and improving future evaluation practice in SHCI; we also believe it holds value for other subdisciplines in HCI that encounter similar challenges while evaluating their research