About me

I am an assistant professor at the Software Technology for Learning and Teaching group of the department of Information and Computing Sciences (ICS) at Utrecht University. The overall goal of my research is to support students in learning programming with high-quality tools and feedback. To achieve this, I investigate tutoring systems and (automated) feedback generation for learning programming, and code quality in education. I am also interested in the influence of Large Language Models/generative AI on computing education.

I am the project leader of a project studying the effects of GenAI on education at the ICS department and beyond. We recently launched our website with practices and materials for programming teachers.

Between 2015 and 2020 I was a parttime PhD candidate at the Open University of the Netherlands, funded by an NWO doctoral grant for teachers.

Until August 2020 I was a (senior) lecturer in Software Engineering at Windesheim University of Applied Sciences.

Highlighted projects

  • Working group on LLMs in CEd

    In this ITiCSE 2023 working group we study several aspects of Generative AI in Computing education, such as student and teacher perceptions, ethics, and LLM performance on programming tasks.

  • Feedback on programming steps

    In an ITiCSe 2022 working group and follow-up project we study how, why and when experts would give feedback on student code while they're solving a programming task.

  • Learning about code quality

    Together with researchers from the software company JetBrains, we study how code quality can be improved in large online learning platforms.

Publications

A complete list of my publications can be found on Google Scholar.

    2025

    • "Ignore These Errors for Now" – How Experts Provide Feedback on Steps Novices Take Towards Solving Programming Problems. Dominic Lohr, Natalie Kiesler, Hieke Keuning, and Johan Jeuring. Accepted to CompEd.
    • KOALA: a Configurable Tool for Collecting IDE Data When Solving Programming Tasks. Daniil Karol, Elizaveta Artser, Ilya Vlasov, Yaroslav Golubev, Hieke Keuning and Anastasiia Birillo. Accepted to CompEd.
    • Student's Use of Generative AI as a Support Tool in an Advanced Web Development Course. Isaac Alpizar-Chacon and Hieke Keuning. Accepted to ITiCSE.
    • (poster) "Can You Refactor This for Me?": Investigating How Students Use ChatGPT in Code Refactoring Exercises. Eduardo Carneiro Oliveira, Hieke Keuning and Johan Jeuring. Accepted to ITiCSE.
    • (poster) KOALA: Customizable IDE Data Collection Tool. Daniil Karol, Elizaveta Artser, Ilya Vlasov, Yaroslav Golubev, Hieke Keuning and Anastasiia Birillo. Accepted to ITiCSE.
    • Teaching Well-Structured Code: A Literature Review of Instructional Approaches. Sara Nurollahian, Hieke Keuning, and Eliane Wiese. IEEE Conference on Software Engineering Education and Training (CSEE&T).
    • You're (Not) My Type - Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks? Dominic Lohr, Hieke Keuning, and Natalie Kiesler. Journal of Computer Assisted Learning (JCAL).
    • In-IDE Programming Courses: Learning Software Development in a Real-World Setting. Anastasiia Birillo, Ilya Vlasov, Katsiaryna Dzialets, Hieke Keuning, and Timofey Bryksin. IDE'25 Workshop at ICSE.

    2024

    • Students’ Perceptions and Use of Generative AI Tools for Programming Across Different Computing Courses. Hieke Keuning, Isaac Alpizar-Chacon, Ioanna Lykourentzou, Lauren Beehler, Sergey Sosnovsky, Christian Köppe and Imke De Jong. Accepted to Koli Calling 2024.
    • One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks. Anastasiia Birillo, Elizaveta Artser, Anna Potriasaeva, Ilya Vlasov, Katsiaryna Dzialets, Yaroslav Golubev, Igor Gerasimov, Hieke Keuning and Timofey Bryksin. Accepted to Koli Calling 2024.
    • Investigating Student Reasoning in Method-Level Code Refactoring: A Think-Aloud Study. Eduardo Oliveira, Hieke Keuning and Johan Jeuring. Accepted to Koli Calling 2024.
    • Clustering MOOC Programming Solutions to Diversify Their Presentation to Students. Elizaveta Artser, Anastasiia Birillo, Yaroslav Golubev, Maria Tigina, Hieke Keuning, Nikolay Vyahhi and Timofey Bryksin. Accepted to Koli Calling 2024.
    • Goodbye Hello World - Research Questions for a Future CS1 Curriculum (poster). Hieke Keuning, Andrew Luxton-Reilly, Claudia Ott, Andrew Petersen, and Natalie Kiesler. Accepted to Koli Calling 2024.
    • "Let Them Try to Figure It Out First" - Reasons Why Experts (Do Not) Provide Feedback to Novice Programmers. Dominic Lohr, Natalie Kiesler, Hieke Keuning, and Johan Jeuring. Proc. of ITiCSE 2024. Best student paper award.
    • Next-Step Hint Generation for Introductory Programming Using Large Language Models. Lianne Roest, Hieke Keuning and Johan Jeuring. Proceedings of the 26th Australasian Computing Education Conference.

    2023

    • The Robots are Here: Navigating the Generative AI Revolution in Computing Education. James Prather, Paul Denny, Juho Leinonen, Brett Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent Reeves and Jaromir Savelka. ITiCSE working group report
    • What Skills Do You Need When Developing Software Using ChatGPT? Johan Jeuring, Roel Groot and Hieke Keuning. Koli Calling 2023.
    • Developers Talking About Code Quality. Jürgen Börstler, Kwabena E. Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar, Rodrigo Duran, Harald Störrle, Daniel Toll and Jelle van Assema. Empirical Software Engineering 28, 128 (2023).
    • Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. Natalie Kiesler, Dominic Lohr and Hieke Keuning. Frontiers in Education (FIE) Conference 2023. Arxiv preprint .
    • A Systematic Mapping Study of Code Quality in Education. Hieke Keuning, Johan Jeuring and Bastiaan Heeren. Proc. of ITiCSE 2023. Version with complete bibliography . Mapping table.
    • Student Code Refactoring Misconceptions. Eduardo Oliveira, Hieke Keuning and Johan Jeuring. Proc. of ITiCSE 2023.
    • Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. Anastasiia Birillo, Elizaveta Artser, Maria Tigina, Yaroslav Golubev, Hieke Keuning, Nikolay Vyahhi and Timofey Bryksin. Proc. of ITiCSE 2023.
    • Analyzing the Quality of Submissions in Online Programming Courses. Maria Tigina, Anastasiia Birillo, Yaroslav Golubev, Hieke Keuning, Nikolay Vyahhi, Timofey Bryksin. Proc. of ICSE-SEET 2023.

    2022

    • Towards Giving Timely Formative Feedback and Hints to Novice Programmers. Johan Jeuring, Hieke Keuning, Samiha Marwan, Dennis Bouvier, Cruz Izu, Natalie Kiesler, Teemu Lehtinen, Dominic Lohr, Andrew Peterson, Sami Sarsa. ITiCSE Working group reports 2022.

    2021

    • A Tutoring System to Learn Code Refactoring. Hieke Keuning, Bastiaan Heeren and Johan Jeuring. Proceedings of the SIGCSE Technical Symposium on Computer Science Education 2021.

    2020

    • Student Refactoring Behaviour in a Programming Tutor. Hieke Keuning, Bastiaan Heeren and Johan Jeuring. Proceedings of the Koli Calling Conference on Computing Education Research 2020.
    • Automated Feedback for Learning Code Refactoring. Hieke Keuning. PhD Thesis, October 2020.

    2019

    • How Teachers Would Help Students to Improve Their Code. Hieke Keuning, Bastiaan Heeren and Johan Jeuring. Proceedings of Innovation and Technology in Computer Science Education (ITiCSE), 2019. + Supplemental material

    2018

    • A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. Hieke Keuning, Johan Jeuring and Bastiaan Heeren. ACM Transactions on Computing Education (TOCE), volume 19, issue 1, 2018. . Table with reviewed tools.

    2017

    • I know it when I see it: Perceptions of Code Quality. Jürgen Börstler, Harald Störrle, Daniel Toll, Jelle van Assema, Rodrigo Duran, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar. ITiCSE Working Group, 2017.
    • An intelligent tutor to learn the evaluation of microcontroller I/O programming expressions. Hugo Arends, Bastiaan Heeren, Hieke Keuning and Johan Jeuring. Proceedings of the Koli Calling Conference on Computing Education Research, 2017.
    • Code Quality Issues in Student Programs. Hieke Keuning, Bastiaan Heeren and Johan Jeuring. Proceedings of Innovation and Technology in Computer Science Education (ITiCSE), 2017.
    • Automatically Classifying Students in Need of Support by Detecting Changes in Programming Behaviour. Anthony Estey, Hieke Keuning, Yvonne Coady. Proceedings of the SIGCSE Technical Symposium on Computer Science Education, 2017.

    2016

    • Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. Hieke Keuning, Johan Jeuring and Bastiaan Heeren. Proceedings of Innovation and Technology in Computer Science Education (ITiCSE), 2016. . Extended Version, Technical report. . Table with reviewed tools.

    2014

    • Strategy-based feedback in a programming tutor. Hieke Keuning, Bastiaan Heeren and Johan Jeuring. Proceedings of the Computer Science Education Research Conference, 2014.
    • Strategy-based feedback for imperative programming exercises. Master Thesis. Open University of the Netherlands, June 2014.

    Teaching

    My interest in Computing Education (CEd) comes from, and continues to be inspired by, my extensive experience in teaching computer science and software engineering topics. At UU, I am currently involved in the following courses:

    In the past, I have taught over twenty courses, designed multiple courses, and coordinated several educational programs. Some coursed I have taught are: Artificial Intelligence for games, Discrete Mathematics, C# programming, OO programming in Java, Client-side web technologies, Software Architecture and design patterns, PHP Web Programming, ASP.NET, Functional Design and OO programming, Simulation, Compiler construction, Databases and SQL, Introduction to programming in C++, and Introduction to Logic.

    Thesis Supervision

    I currently supervise two PhD students: Eduardo Carneiro de Oliveira and Anastasiia Birillo (external).

    You can always contact me if you are interested in doing a Bachelor or Master thesis on a topic related to my reserach interests. Concrete projects can be found on my Konjoin profile.

    Other

    Recent & upcoming talks/events

    • Talk Improving tools for learning programming: What is (not) the role of GenAI?, at JetBrains. June 2025, Belgrade, Serbia.
    • Workshop GenAI in programming education at Instruct teachers day April 2025, Utrecht.
    • Talk at conference I&I vakvereniging GenAI in het programmeeronderwijs: wat weten we nu en waar gaan we naar toe? November 2024.
    • Talk at UU's SHA symposium Generative AI in (computing) education —does it lead to better and more accessible learning? November 2024.
    • Invited keynote speaker at Delfi ‘Fachtagung Bildungstechnologien’, September 2024, Fulda, Germany.
    • Attended Dagstuhl meeting ‘A Game of Shadows: Effective Mastery Learning in the Age of Ubiquitous AI’.
    • Talk at the 'GenAI for CS education workshop' at the Open University, Utrecht, May 2024.
    • Talk Using LLMs to Generate Feedback on Introductory Programming Tasks at research seminars University of Otago in Dunedin, and University of Canterbury in Christchurch, New Zealand, April 2024.
    • University of Twente FMT Colloquium talk Teaching CS in the Era of Generative AI, January 2024, Enschede.
    • Invited talk + panelist at NeurIPS 2023 workshop Generative AI for Education (GAIED), December 2023, New Orleans, USA.
    • Talk at the Vakdidactiek Informatica meeting Learning Programming in the Era of Generative AI, December 2023, Utrecht.
    • Panel discussion, VERSEN event on Generative AI and Ethics, November 2023, Utrecht.
    • Talk Copilot and ChatGPT in programming education: what do we know from a research perspective? at NIOC 2023 (Dutch Informatics Education Conference).
    • Talk A.I. tools in programming education at the U-talent ‘Docentennetwerk informatica’ meeting, January 2023.

    Media

    • Generative AI in computing education - Perceptions and practices of teachers at Utrecht University. Article on Surfnet, April 2024.
    • Improving computer science education. Feature in I/O magazine, April 2023.
    • Betere code met automatische feedback en hints. Hieke Keuning. AG Connect, December 2020.

    Service

    I serve in the program committee of several CEd research conferences (ITiCSE, SIGCSE, ICER, AIED, Koli, QUATIC), and regularly review for journals (e.g. TOCE, TLT, C&E, JCAL).