THERMOS: providing feedback for students via a Learning Analytics Dashboard
The Thermos project developed a feedback instrument (THERMOS dashboard) that provides students with insight and actionable feedback on their study behavior. The developed dashboard was piloted in six study programs in 20/21, across different faculties of Utrecht University (UU), including those located at University Medical Center Utrecht (UMCU). The follow-up project ‘Thermos for UU’ starts in 21/22 and will further focus on the implementation process in collaboration with more study programs across UU and the UMCU.
Go to the project website for more videos
Background
Students learn best when they have insight in their own study behavior and motivation and know how to self-regulate. The Thermos project gives students insight in their study behavior and motivation by means of Learning Analytics. Learning Analytics revolves around gathering data about learners and their learning processes to better understanding and where possible improve them. It is often defined as “the measurement, collection, analysis and reporting of data, about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & Baker, 2012). Presenting students with data about their study motivation and study behavior may help them improve on aspects they choose. However, simply gathering and visualizing data may not necessarily work, as interpreting and acting upon data (visualizations) is not always easy (Jivet et al., 2020). Implementing a Learning Analytics Dashboard (LAD) into a meaningful educational context can increase the value for learners (Wise et al., 2016).
Project description
From academic year 18/19 to 20/21, the Thermos project developed a Learning Analytics Dashboard (LAD) to support students in their studies. This dashboard gives insight in several aspects of study behavior (e.g. motivation and group work skills) and study progress (e.g. ECTS). It also provides users with actionable feedback to engage with these aspects such as individual exercises, and additional referrals to (for example) Skills lab workshops or an online studycoach.
Figure 1. Screen shot of THERMOS dashboard
In academic year 20/21, the dashboard was piloted in 6 study programs from different faculties of UU and the UMCU. In this pilot we explored the dashboard’s usability and usefulness for students, and ways to implement it in a study program.
From 21/22, the follow-up project ‘Thermos for UU’ starts, which focusses more on the process of implementing the dashboard into tutorprograms. Within this follow-up project, the dashboard will be implemented in more study programs across both UU and UMCU.
Goals, Results and Further Suggestions
Goals
The overall goal is to support students in improving aspects of study behavior by: helping them to assess important aspects of study behavior, to communicate students’ study behavior profile to provide them with insight, and to help them develop through actionable feedback.
Results ‘Thermos’
The dashboard was implemented in the tutorprogram of six studyprograms across UU and UMCU in academic year 20/21. This pilot evaluated the dashboard’s usability, usefulness, and the way in which it was implemented across study programs. Overall, the dashboard’s usability was already good, students understood and were able to interpret the different parts. As it was suggested that usability could be improved upon by developing a Dutch version of the dashboard, this is will be available at the start of the Thermos for UU project. The dashboard’s experienced usefulness varied, with both positive (e.g. remarks such as “I learned more about my study behavior and some useful tips”) and less positive evaluations (“At this moment I’m not sure in which way the dashboard could help my study behavior”).
Goals
The Thermos for UU project aims to develop implementation standards and guidelines, as a suitable implementation can enhance the value of the dashboard. The development and evaluation of standards contributes to its success. The other objective of the implementation project is to integrate the dashboard into the existing UU organization.
Results ‘Thermos for UU’
Thermos has been implemented in 12 educational programs across 5 faculties. In total, approximately 2500 unique students have used the system. The findings and further development have been generally positive, with high user satisfaction. The project has successfully integrated educational scientific research. The project team is cohesive and collaborative.
An important goal of this project was to transfer ownership from the Thermos project to the standing organization within the UU (Utrecht University). This process has been considered since the start of the Thermos project in 2018 and intensified during the academic year 19/20. However, it did not lead to the desired outcome at that time. As a result, numerous discussions were held with various parties that could potentially play a role in this integration.
Regarding process ownership, these parties included SkillsLab, Educate-It, and the Online Coaching Centre. For system ownership, ITS (Information Technology Services) is involved. Preparations for data ownership have been made with the relevant privacy officer.
Together with Hjalmar Eijkelenkamp, a business case was developed in the spring of 2023 for Student Development, which is part of SO&O (Student Affairs & Operations) and oversees SkillsLab, Student Welfare, and Career Services. In April 2023, Student Development expressed their interest in taking over the process ownership. The details of this transfer will be discussed in the short term (after April 2023) so that any necessary preparations can be made. With the transfer of process ownership, an important first step has been taken towards integrating the project into the standing organization.
In preparation for this transfer and to promote sustainable integration, several supporting materials have been developed. These supporting materials contribute to the development of standards and guidelines for the use of Thermos. Supporting materials include tutor training sessions, student work forms, implementation examples, and a RoadMap to support the implementation process. These materials can assist and inspire current or new study programs in implementing the Thermos dashboard in a way that is specific to each program.
The Thermos project has an impact on and connection with the development of LA (Learning Analytics) policies within the UU (Utrecht University). By engaging with stakeholders in various relevant areas (e.g., ITS, Student Affairs & Operations, LA-team, different privacy officers), we have communicated our development and implementation needs.
Further Suggestions
The structural continuation will be shaped in collaboration with Student Development, as they will take on the process ownership of Thermos. In May ’23, we will jointly explore how the dashboard fits into the services provided by Student Development and what the exact structure of the continuation will be. Student Development has already expressed interest in certain further developments of the dashboard.
Further reading
- See the Thermos website for more information
- Log-in to the THERMOS dashboard [Log in with SolisID]
- EU OFLA project (Onwards from Learning Analytics)
- STELA (Successful Transition in Education using Learning Analytics) from the KU Leuven
References
- Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. The Internet and Higher Education, 47, 100758.
- Lang, C., Siemens, G., Wise, A., & Gasevic, D. (Eds.). (2017). Handbook of learning analytics. SOLAR, Society for Learning Analytics and Research. DOI: 10.18608/hla17
- Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing Learning Analytics Design Knowledge in the” Middle Space”: The Student Tuning Model and Align Design Framework for Learning Analytics Use. Online Learning, 20(2), 155-182.
- Siemens, G., & Baker, R. S. D. (2012, April). Learning analytics and educational data mining: towards communication and collaboration.
In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 252-254).
You are free to share and adapt, if you give appropriate credit and use it non-commercially. More on