Between analysis and transformation: technology, methodology and evaluation on the SPLICE project

This paper concerns the ways in which technological change may entail methodological development in e-learning research. The focus of our argument centres on the subject of evaluation in e-learning and how technology can contribute to consensus-building on the value of project outcomes, and the identification of mechanisms behind those outcomes. We argue that a critical approach to the methodology of evaluation which harnesses technology in this way is vital to agile and effective policy and strategy-making in institutions as the challenges of transformation in a rapidly changing educational and technological environment are grappled with. With its focus on mechanisms and multiple stakeholder perspectives, we identify Pawson and Tilley’s ‘Realistic Evaluation’ as an appropriate methodological approach for this purpose, and we report on its use within a JISC-funded project on social software, SPLICE (Social Practices, Learning and Interoperability in Connected Environments). The project created new tools to assist the identification of mechanisms responsible for change to personal and institutional technological practice. These tools included collaborative mind-mapping and focused questioning, and tools for the animated modelling of complex mechanisms. By using these tools, large numbers of project stakeholders could engage in a process where they were encouraged to articulate and share their theories and ideas as to why project outcomes occurred. Using the technology, this process led towards the identification and agreement of common mechanisms which had explanatory power for all stakeholders. In conclusion, we argue that SPLICE has shown the potential of technologically-mediated Realistic Evaluation. Given the technologies we now have, a methodology based on the mass cumulation of stakeholder theories and ideas about mechanisms is feasible. Furthermore, the summative outcomes of such a process are rich in explanatory and predictive power, and therefore useful to the immediate and strategic problems of the sector. Finally, we argue that as well as generating better explanations for phenomena, the evaluation process can itself become transformative for stakeholders

General characteristics of anticipated user experience (AUX) with interactive products

Providing a positive user experience (UX) has become the key differentiator for products to win a competition in mature markets. To ensure that a product will support enjoyable experiences for its users, assessment of UX should be conducted early during the design and development process. However, most UX frameworks and evaluation techniques focus on understanding and assessing user’s experience with functional prototypes or existing products. This situation delays UX assessment until the late phases of product development which may result in costly design modifications and less desirable products. A qualitative study was conducted to investigate anticipated user experience (AUX) to address this issue. Twenty pairs of participants were asked to imagine an interactive product, draw their product concept, and anticipate their interactions and experiences with it. The data was analyzed to identify general characteristics of AUX. We found that while positive AUX was mostly related to an imagined/desired product, negative AUX was mainly associated with existing products. It was evident that the pragmatic quality of product was fundamental, and significantly influenced user’s anticipated experiences. Furthermore, the hedonic quality of product received more focus in positive than negative AUX. The results also showed that context, user profile, experiential knowledge, and anticipated emotion could be reflected in AUX. The understanding of AUX will help product designers to better foresee the users’ underlying needs and to focus on the most important aspects of their positive experiences, which in turn facilitates the designers to ensure pleasurable UX from the start of the design process

Trends in LN-embedding practices at Waikato Institute of Technology (Wintec) in 2019

In this report, we describe the trends in literacy-embedding practices of level-2 and level-3 tutors who worked in vocational contexts at Waikato Institute of Technology (Wintec), and who completed the New Zealand Certificate in Adult Literacy and Numeracy Education (NZCALNE[Voc]) in 2019. We analysed 19 observations, following constructivist grounded theory methodology (Charmaz, 2014), to produce 1302 descriptive labels that highlight literacy and numeracy practices integrated into tutors’ teaching intentionally pursued in a collaborative and mentored training process. Of the initial 12 categories, we conflated the mapping of LN course demands and identifying learners’ LN needs to arrive at a final 11. We then used these categories in an axial analysis (Saldaňa, 2013), categorising the 1302 labels as binaries (i.e. if the label was related to the category, 1 was coded; if not 0 [zero]). The matrix of 14322 ratings of 1s and 0s was then analysed. We calculated the frequency of 1s by category. We argued that the axial analysis allowed us to develop a more holistic perspective which showed how the 1302 labels were configured in relation to the 11 categories of analysis. We concluded that the 11 categories represented key aspects of vocational teaching and training emphasising that LN-embedding practices have to be seamlessly integrated into general pedagogical approaches. A key construct for new tutors is to shape their understanding of seamlessly integrated versus bolted-on LN practices. Our recommendations remain within the whole-of-organisation perspective proposed in the 2017-2018 report (Greyling, 2019)

The accessibility of administrative processes: Assessing the impacts on students in higher education

Administrative processes that need to be completed to maintain a basic standard of living, to study, or to attain employment, are perceived to create burdens for disabled people. The navigation of information, forms, communications, and assessments to achieve a particular goal raises diverse accessibility issues. In this paper we explore the different types of impacts these processes have on disabled university students. We begin by surveying literature that highlights the systemic characteristics of administrative burdens and barriers for disabled people. We then describe how a participatory research exercise with students led to the development of a survey on these issues. This was completed by 104 respondents with a diverse range of declared disabilities. This provides evidence for a range of impacts, and understanding of the perceived level of challenge of commonly experienced processes. The most common negative impact reported was on stress levels. Other commonly reported impacts include exacerbation of existing conditions, time lost from study, and instances where support was not available in a timely fashion. Processes to apply for disability-related support were more commonly challenging than other types of processes. We use this research to suggest directions for improving accessibility and empowerment in this space

Structured evaluation of virtual environments for special-needs education

This paper describes the development of a structured approach to evaluate experiential and communication virtual learning environments (VLEs) designed specifically for use in the education of children with severe learning difficulties at the Shepherd special needs school in Nottingham, UK. Constructivist learning theory was used as a basis for the production of an evaluation framework, used to evaluate the design of three VLEs and how they were used by students with respect to this learning theory. From an observational field study of student-teacher pairs using the VLEs, 18 behaviour categories were identified as relevant to five of the seven constructivist principles defined by Jonassen (1994). Analysis of student-teacher behaviour was used to provide support for, or against, the constructivist principles. The results show that the three VLEs meet the constructivist principles in very different ways and recommendations for design modifications are put forward

Satisfiers and dissatisfiers for international vocational education students: A case study using narrative frames

While the satisfaction of international students is frequently surveyed, much of this research is based on a very limited range of closed-item data collection methods, producing findings that partially reflect the researchers’ assumptions in designing the survey items. Recognizing the potential value in using methods that are more open-ended and qualitative analyses, the present study employed narrative frames and follow-up interviews to explore the satisfiers and dissatisfiers for international vocational education students at one institution in New Zealand. Reporting on perceptions of the class, institution, and community, the findings identify the participants’ top-of-mind (dis)satisfiers and complement the existing literature by identifying seldom-reported themes. Discussion of issues in analyzing narrative frames is also provided

Consequences of Unhappiness While Developing Software

The growing literature on affect among software developers mostly reports on the linkage between happiness, software quality, and developer productivity. Understanding the positive side of happiness -- positive emotions and moods -- is an attractive and important endeavor. Scholars in industrial and organizational psychology have suggested that also studying the negative side -- unhappiness -- could lead to cost-effective ways of enhancing working conditions, job performance, and to limiting the occurrence of psychological disorders. Our comprehension of the consequences of (un)happiness among developers is still too shallow, and is mainly expressed in terms of development productivity and software quality. In this paper, we attempt to uncover the experienced consequences of unhappiness among software developers. Using qualitative data analysis of the responses given by 181 questionnaire participants, we identified 49 consequences of unhappiness while doing software development. We found detrimental consequences on developers' mental well-being, the software development process, and the produced artifacts. Our classification scheme, available as open data, will spawn new happiness research opportunities of cause-effect type, and it can act as a guideline for practitioners for identifying damaging effects of unhappiness and for fostering happiness on the job.Comment: 6 pages. To be presented at the Second International Workshop on Emotion Awareness in Software Engineering, colocated with the 39th International Conference on Software Engineering (ICSE'17). Extended version of arXiv:1701.02952v2 [cs.SE

An Analysis of Cooperating Teacher Feedback: A Qualitative Inquiry

Cooperating teachers are a key component to the success of student teaching internships, serving an integral part in “raising” a teacher. To effectively facilitate the student teaching internship, teacher preparation programs must identify cooperating teachers who align philosophically with the pedagogical training delivered by university programs (Korthagen & Kessels, 1999; Tom, 1997), specifically, cooperating teachers who can reinforce the theoretical framework underpinning the professional coursework pre-service teachers experience in university teacher preparation programs. This qualitative study sought to better understand the feedback provided to future school-based agricultural education (SBAE) teachers during their student teaching experience. Through initial and secondary coding, the research team identified themes among the feedback provided to student teachers by their cooperating teachers. The study revealed cooperating teacher feedback reflects the pedagogical training provided via the teacher preparation program, specifically, around effective teaching behaviors. Understanding the cooperating teacher feedback provides insight for teacher preparation programs

Understanding the information experiences of parents involved in negotiating post-separation parenting arrangements

The paper presents findings from a study into the information experiences of people needing to make post-separation parenting arrangements. Data was collected from 20 participants, through in-depth, semi-structured, telephone interviews. Thematic analysis identified five major themes: Following, Immersion, Interpersonal, History and Context which depict the information experiences of the participants. The findings can be used as an evidence base to inform the design and delivery of support and services provided by government agencies and other community groups supporting the legal information needs of individuals and families. The work extends current understandings of information experience as an object of study in the information science discipline