Generative AI and critical coding
GenAI can play a supportive role for novice coders in the humanities to learn programming for research purposes. However, the entry thresholds for using GenAI tools effectively to approach coding tasks are high and individual learning support limited. If critically applied, GenAI can help with understanding, conceptualizing, and trouble shooting. To this end, specific prompting skills, as well as skills in critical thinking about AI generated solutions are essential for students of computer programming and computational methods. Within the context of the 2nd year BA elective Digital Methods, students are being trained in these areas, so that they can harness GenAI as a tool for learning programming, while being aware of its limitations.
Background information
Quick and easy-to-follow solutions to coding problems are not widely available, especially for novice coders in the humanities. The course Digital Methods aims to introduce humanities (and social science) students to digital and computational methods for empirical research. Moving away from ready-made interfaces, they introduce coding for beginners with a strong focus on application for media studies research. When approached critically and within a limited extent, GenAI can play a useful role in supporting students learning process next to teacher instructions, text books, and other learning resources. However, as with all AI-generated output, this comes with challenges pertaining to the accuracy, validity, and reliability of these code solutions – as well as ethical questions especially concerning “authorship” of code. These are challenges for students and lecturers. Students may blindly trust potentially flawed or insufficient solutions generated by AI. Lecturers may find it difficult to judge and assess the work of students. This becomes especially challenging when it is not transparent for which well-defined and acceptable tasks within clearly defined assignments GenAI was used. With this intervention, the project team wants to teach students how they can make efficient use of GenAI for learning programming in a critical-reflexive manner. They aimed at supporting their wider efforts to embrace computational methods and data science for humanities research.
Project description
The project team organized three sessions around Python programming for computational methods in media research, focusing on automated text analysis. For most students, these were the first and only sessions on computer programming. The goal was to use GenAI, specifically ChatGPT, as a support tool in learning programming and executing programming tasks with a focus on data analysis. ChatGPT was chosen due to its widespread use, familiarity, and accessibility for students, also for non coding-related tasks. The exercise had two learning goals:
1. How to use (or not use) GenAI in programming
2. To be able to critically reflect on its use for research practice in the field of computational methods for media studies.
The first session focused on brainstorming possible uses for GenAI in beginner’s coding contexts. We did this first with the entire class and then in work groups of 4-5 students. Students were asked to collect ideas for what GenAI could do for them, if the task were to develop computational methods for a media studies related research question. This also included discussing for what steps in their research GenAI should play a limited or no role at all. Students where encouraged to continue the discussion in their work groups. Students were free to choose an angle for the discussion but most of the students linked their deliberations to their project ideas for the course assignment (a research report on a media studies question using computational methods).
The different use cases included generating new ideas for data analysis, clarifying variable types, offering step-by-step explanations of code, and providing targeted troubleshooting (e.g., feeding it with errors and asking for explanations). The goal was to scope what GenAI could be used for, before experimenting with it with the coding tasks at hand (analyzing a large volume of social media texts). As such, GenAI can deliver quick, foundational guidance.
In the second session, students worked on coding exercises both with and without GenAI’s assistance. For example, first they were asked to write code based on scripts shared by the teacher, form an analysis themselves and execute it. If it yielded an error, they were free to prompt ChatGPT with the error message and an explanation to see if it provided a clear and working solution. Students were asked to keep track of their prompts and make notes on the provide solutions (Are they clear and understandable? Were additional prompts needed to solve the problem and/or receive an explication?). Next, students were asked to prompt ChatGPT to provide its own code to tackle the task (e.g., preprocessing for topic modeling or a text clustering step) and compare it with “their” solution (i.e., edited code based on what the teacher provided). The goal was to show:
1. There are often different ways to reach a goal but also
2. Simply copy & pasting AI-generated code often is insufficient to arrive at a working solution.
More iterations and manual edits are often needed but this can happen through a feedback loop between the researcher and the AI. They experimented with different prompts in the groups, with each student applying their own phrasing to ChatGPT to solve the same problem. They then compared the results and discussed to what extent differences in prompting mattered for the output. Collectively, the teacher then showed them emerging prompting practices that underlined specificity for well-defined and practical tasks in the research pipeline.
The third session served for broader critical reflection, and ideating best practices for using GenAI tools. First, each group prepared a short presentation on 2-3 tasks that were shaped with the input of ChatGPT and how they experienced and evaluated coding with and without ChatGPT. They were asked to list benefits, challenges, and downsides/risks. Students reflected on the tool’s effectiveness in supporting the build-up of coding basics, and its performance in explaining code. They also reflected on why it is often necessary to use GenAi in an iterative manner (the first prompt rarely yields a solution). We reflected on this per group and eventually as a class.
Then they synthesized generalizable “critical reflection principles”: 1) understand how GenAI works in general to be able to judge its usefulness for research; 2) keep problems and tasks for GenAI support small, simple, and “controllable” (ideally you can validate the solutions and won’t blindly trust it); 3) keep track of your thinking and planning process and critically review whether you got steered a bit too much by GenAI especially at the conceptualization phase (ideation of methods); 4) try it yourself first, ask peers or lecturers before you turn to GenAI; 5) check your use of GenAI against established norms for research ethics (as discussed in the course).
The group also discussed that GenAI can only be one factor in learning computational skills and that interested students from non-technical subjects need to invest time and effort to learn programming at least to an intermediate degree to be better able to request solutions from GenAI and evalute them in terms of validity and quality. Attending further courses in computer programming and developing a personal autonomous learning plan for computational skills were collectively suggested as a viable path – that is frankly not attractive for all students (given the different time and cognitive investments that are unavoidable here).
Results
Student feedback -collected during the group and class discussions- highlighted that GenAI can be a helpful sparring and learning partner to some extent, yet it cannot fully replace fundamental code literacy. Users must still understand basic programming principles rather than relying solely on e.g., ChatGPT. Achieving solutions often requires multiple attempts, along with patience and persistence. Moreover, crafting explicit and detailed prompts is a skill—perhaps even an art—that significantly influences the quality of AI-generated responses. Still, GenAI can to some extent lower the threshold for beginners, acting as a “tutor” or “advanced explainer” if used thoughtfully. For those who want to leverage its full capabilities, the paid version offers advantages over the free model. The paid model gives access to more complex and better-trained models, with faster and more optimized capabilities, especially for data science, which in many cases directly benefits computational methods. Nevertheless, method knowledge and data literacy remain critical; while GenAI can assist to a degree, it has inherent limitations.
In the future, the exercise will be expanded to include further reflection on the differences and usefulness of different coding/programming tasks between GenAI tools. Students need to be trained more in independently conceptualizing ideas and work flow outside of GenAI and steer them towards uses that aim for more mundane, practical implementation steps.
Lessons learned
Overall, this exercise revealed both GenAI’s potential and constraints. Simply put, it can provide novice coders with accessible explanations and solution approaches that they encounter.
Relying solely on GenAI can impede learning of computer programming. While it also can become difficult to receive a working solution if one is not capable of iterating concrete prompts. So, to use GenAI productively, you still need to go the “hard way” and learn coding yourself. Then there are the concrete limitations in access and quality between paid and free versions. Generally, each individual -as a student and future professional- needs to judge if and to what extent any sort of dependency on commercial GenAI solutions is desirable. In short:
- GenAI can help with explaining code basics
- GenAI can to some extent solve errors by pointing users in the right direction, but clear and repeated prompting might be necessary (and challenging)
- GenAI can lay out to what extent computational methiods maybe used for humanities research but domain expertise is superior in determining concrete (and valid) research goals.
Take home message
- Have your own solutions ready to compare to GenAI-generated ones to show differences, mistakes, and benefits.
- Bring “effective”, “mediocre”, and “inefficient” examples for prompts and showcase their performance.
- Explain to student how generating code differs but also is similar to generating text. The main differences are the purposes of “writing”, which in programming often focuses on concrete tasks in long analytical pipelines, where some methodological knowledge about quantitative methodologies is necessary.
Central AI policy
All AI-related activities on this page must be implemented in line with Utrecht University’s central AI policy and ethical code.
Responsibility for appropriate tool choice, data protection, transparency, and assessment use remains with the instructor.
You are free to share and adapt, if you give appropriate credit and use it non-commercially. More on