Can an AI obtain a master’s degree? An investigation into AI-assisted education in the MSBBSS Research Master’s programme.
In this article the upcoming project: “Can an AI obtain a master’s degree? An investigation into AI-assisted education in the MSBBSS Research Master’s programme.” will be discussed.
The aim of this project
This project proposes to run two distinct AI language models as potential `student candidates` through the Methodology and Statistics in the Behavioural, Biomedical and Social Sciences research master at Utrecht University’s School of Social and Behavioral Sciences. To highlight the timeliness and seriousness of our proposal and the potential impact our findings could have on the validity of our academic testing, most of the content in Box 1 of this proposal has not been written by the project team but has been generated by the chatGPT and Perplexity.ai language models. The recordings of the interactions with these models and conversation transcripts can be found at www.gerkovink.com/FUSO2023. All AI-generated text is highlighted in grey.
The UU board proposes to record in the new OER the requirement for students to disclose the use of AI language models. Therefore, we also disclose that we have used AI language models in generating this proposal text. More specifically, we have asked the chatGPT and Perplexity.AI language models whether an AI could be able to pass a master’s programme. The response was:
Yes, an AI can pass an academic master’s programme. A professor at the University of Pennsylvania’s Wharton School found that the artificial intelligence-driven chatbot GPT-3 was able to pass the final exam for the school’s Master of Business Administration (MBA) program. The bot scored between a B- and B on the exam, showing its remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in jobs held by MBA graduates including analysts, managers, and consultants.
The bad?
The emergence of AI-assisted learning presents new opportunities and challenges for teachers to design effective evaluations and assessments that measure students’ learning progress in a way that reflects their unique contributions and effort.
One approach could be to modify existing evaluation methods to include a requirement for students to demonstrate their understanding and critical thinking skills in addition to finding sources. For example, you could ask students to write a critical analysis of the sources they found using e.g., ChatGPT, or to apply the concepts and theories they learned to real-world scenarios.
Another approach is to design evaluations that cannot be easily replicated by an AI language model. This could involve creating problem-solving scenarios that require students to draw on their understanding of the material to identify and analyse various solutions. You could also ask students to engage in debates, simulations or role- play activities that require them to apply their knowledge and skills in real-life situations.
The good
While educators often focus on the immediate impact any AI language models can have on testing, the benefits of learning together with an AI are easily overlooked. AI-assisted learning has been shown to improve student performance. Cheng et al. developed an AI-augmented teaching program for medical students in detecting hip fractures and found significantly higher post-learning accuracy in the AI-assisted group5. A study of AI precision education at Dong Hwa University found that the AI precision education model may facilitate students’ learning experience and enhance student achievement. The study employed drawing and co-word analysis techniques to explore students’ preferences for AI-assisted learning environments, with more than half of the students agreeing that robots play important roles in AI-assisted learning.
AI-assisted foreign language education costs less, incites learners’ interests, and improves efficiency compared with traditional foreign language education. However, most experts agree that human teachers are still the most important ingredient to learning. Companies like Carnegie Learning, ALO7, and Sana Labs show how AI can help adapt to individual student needs, free up teacher time by streamlining administrative tasks, and more.
Target results
- Run GitHub co-pilot and chatGPT as a student pair through the MSBBSS research master’s programme. What grades would the student pair get and what percentage of course work can be successfully completed?
- Classify per course the AI student pair performance and investigate whether there is a relation between specific types of course work, assignment, rubrics, and course performance. In short, is there a type of work on which the AI performs well or not well? Several courses use on-location testing for some – or all – tests, which makes it mostly impossible to use AI models. We will still submit such projected on-location tests to the AI pair. This project will also study the impact of AI language models on take-home exercises.
- Submit a manuscript for publication that outlines the taken procedure, presents our analyses, summarizes our findings, and proposes guidelines for educators with respect to the challenges and opportunities in AI-assisted learning for a (statistics) master’s programme.
- Design a short workshop together with OA&T to present our findings and share our experience and tips with other educators at Utrecht University. A good opportunity would e.g., be the Onderwijsfestival 2024. Preferably, we would design a Small Private Online Course module to share our findings beyond Utrecht University.
You are free to share and adapt, if you give appropriate credit and use it non-commercially. More on