A Human-in-the-Loop Orchestration Protocol for AI Co-Teaching and Assessment in Engineering Capstone Courses

Authors

1 Department of Physical Education and Training, Sichuan Sports College, Chengdu, China 610043
2 Innovation and Entrepreneurship College, Jiangxi Institute of Applied Science and Technology, Nanchang, China 330100
Corresponding author: 112206470@qq.com

DOI:

https://doi.org/10.63646/JSIE4302

Keywords:

Engineering Capstone Education, Human-AI Collaboration, AI Co-Teaching, Learning Orchestration, Formative Assessment, Engineering Design, Generative AI

Abstract

Engineering capstone courses ask teams to integrate technical knowledge, stakeholder requirements, design iteration, testing, documentation, and professional judgment over an extended project. Generative artificial intelligence can assist with explanation, critique, and synthesis, but a conversational assistant alone does not determine when it should intervene, which evidence it may use, how individual learning remains visible within team work, or who is accountable for assessment decisions. This article specifies a human-in-the-loop orchestration protocol for AI co-teaching in engineering capstones. The protocol binds each AI action to a versioned learning objective, project milestone, evidence scope, consent state, and instructor-defined authority limit. A course orchestrator coordinates learner and team activity, instructor review, approved AI services, and an auditable evidence ledger. AI may offer bounded explanations, alternative analyses, rubric-linked formative feedback, and instructor alerts; students retain authorship of engineering decisions, and instructors retain responsibility for instructional and summative judgments. The design defines role permissions, state transitions, evidence records, escalation rules, assessment separation, safety controls, and a synthetic campus microgrid capstone scenario. It contributes an implementable architecture, protocol schemas, a rubric-to-evidence model, and a prospective verification plan. It reports no deployment, participant data, or measured learning effects; those claims require future validation.

How to Cite

Wang, M., & Liang, Q. (2026). A Human-in-the-Loop Orchestration Protocol for AI Co-Teaching and Assessment in Engineering Capstone Courses. Trends and Reviews of Technological Engineering in Education, 2(3), 117-149. https://doi.org/10.63646/JSIE4302

References

R. Ajjawi and D. Boud, “Researching feedback dialogue: An interactional analysis approach”, Assessment & Evaluation in Higher Education, vol. 42, no. 2, pp. 252–265, 2017, https://doi.org/10.1080/02602938.2015.1102863.

S. Amershi, D. Weld, M. Vorvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. Iqbal, P. N. Bennett, K. Inkpen, et al., “Guidelines for human-AI interaction”, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Paper 3, pp. 1–13, 2019, https://doi.org/10.1145/3290605.3300233.

C. J. Atman, R. S. Adams, M. E. Cardella, J. Turns, S. Mosborg, and J. Saleem, “Engineering design processes: A comparison of students and expert practitioners”, Journal of Engineering Education, vol. 96, no. 4, pp. 359–379, 2007, https://doi.org/10.1002/j.2168-9830.2007.tb00945.x.

G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. S. Weld, “Does the whole exceed its parts?”, The effect of AI explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16, 2021, https://doi.org/10.1145/3411764.3445717.

B. J. S. Barron, D. L. Schwartz, N. J. Vye, A. Moore, A. Petrosino, L. Zech, J. D. Bransford, and The Cognition and Technology Group at Vanderbilt, “Doing with understanding: Lessons from research on problem- and project-based learning”, The Journal of the Learning Sciences, 7(3–4), pp. 271–311, 1998, https://doi.org/10.1080/10508406.1998.9672056.

S. Bell, “Project-based learning for the 21st century: Skills for the future”, The Clearing House, vol. 83, no. 2, pp. 39–43, 2010, https://doi.org/10.1080/00098650903505415.

P. Black and D. Wiliam, “Assessment and classroom learning”, Assessment in Education: Principles, Policy & Practice, vol. 5, no. 1, pp. 7–74, 1998, https://doi.org/10.1080/0969595980050102.

P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial, and A. Palincsar, “Motivating project-based learning: Sustaining the doing, supporting the learning”, Educational Psychologist, 26(3–4), pp. 369–398, 1991, https://doi.org/10.1080/00461520.1991.9653139.

D. Boud and E. Molloy, “Rethinking models of feedback for learning: The challenge of design”, Assessment & Evaluation in Higher Education, vol. 38, no. 6, pp. 698–712, 2013, https://doi.org/10.1080/02602938.2012.691462.

Z. Buçinca, M. B. Malaya, and K. Z. Gajos, “To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making”, Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Art. no. 188, 2021, https://doi.org/10.1145/3449287.

D. Carless and D. Boud, “The development of student feedback literacy: Enabling uptake of feedback”, Assessment & Evaluation in Higher Education, vol. 43, no. 8, pp. 1315–1325, 2018, https://doi.org/10.1080/02602938.2018.1463354.

L. Chen, P. Chen, and Z. Lin, “Artificial intelligence in education: A review”, IEEE Access, vol. 8, pp. 75264–75278, 2020, https://doi.org/10.1109/ACCESS.2020.2988510.

M. T. H. Chi, “Active-constructive-interactive: A conceptual framework for differentiating learning activities”, Topics in Cognitive Science, vol. 1, no. 1, pp. 73–105, 2009, https://doi.org/10.1111/j.1756-8765.2008.01005.x.

D. R. E. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of ChatGPT”, Innovations in Education and Teaching International, vol. 61, no. 2, pp. 228–239, 2023, https://doi.org/10.1080/14703297.2023.2190148.

Y. Dai, “Why students use or not use generative AI: Student conceptions, concerns, and implications for engineering education”, Digital Engineering, vol. 4, Art. no. 100019, 2025, https://doi.org/10.1016/j.dte.2024.100019.

D. Dellermann, P. Ebel, M. Söllner, and J. M. Leimeister, “Hybrid intelligence”, Business & Information Systems Engineering, vol. 61, pp. 637–643, 2019, https://doi.org/10.1007/s12599-019-00595-2.

P. Denny, V. Kumar, and N. Giacaman, “Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language”, Proceedings of the 54th ACM Technical Symposium on Computer Science Education, pp. 1136–1142, 2023, https://doi.org/10.1145/3545945.3569823.

A. R. Doshi and O. P. Hauser, “Generative AI enhances individual creativity but reduces the collective diversity of novel content”, Science Advances, eadn5290, vol. 10, no. 28, 2024, https://doi.org/10.1126/sciadv.adn5290.

A. J. Dutson, R. H. Todd, S. P. Magleby, and C. D. Sorensen, “A review of literature on teaching engineering design through project-oriented capstone courses”, Journal of Engineering Education, vol. 86, no. 1, pp. 17–28, 1997, https://doi.org/10.1002/j.2168-9830.1997.tb00260.x.

C. L. Dym, A. M. Agogino, O. Eris, D. D. Frey, and L. J. Leifer, “Engineering design thinking, teaching, and learning”, Journal of Engineering Education, vol. 94, no. 1, pp. 103–120, 2005, https://doi.org/10.1002/j.2168-9830.2005.tb00832.x.

C. Evans, “Making sense of assessment feedback in higher education”, Review of Educational Research, vol. 83, no. 1, pp. 70–120, 2013, https://doi.org/10.3102/0034654312474350.

S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. P. Wenderoth, “Active learning increases student performance in science, engineering, and mathematics”, Proceedings of the National Academy of Sciences, vol. 111, no. 23, pp. 8410–8415, 2014, https://doi.org/10.1073/pnas.1319030111.

T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford, “Datasheets for datasets”, Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021, https://doi.org/10.1145/3458723.

J. Hattie and H. Timperley, “The power of feedback”, Review of Educational Research, vol. 77, no. 1, pp. 81–112, 2007, https://doi.org/10.3102/003465430298487.

C. E. Hmelo-Silver, “Problem-based learning: What and how do students learn?”, Educational Psychology Review, vol. 16, pp. 235–266, 2004, https://doi.org/10.1023/B:EDPR.0000034022.16470.f3.

K. A. Hoff and M. Bashir, “Trust in automation: Integrating empirical evidence on factors that influence trust”, Human Factors, vol. 57, no. 3, pp. 407–434, 2015, https://doi.org/10.1177/0018720814547570.

E. Horvitz, “Principles of mixed-initiative user interfaces”, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 159–166, 1999, https://doi.org/10.1145/302979.303030.

G.-J. Hwang, H. Xie, B. W. Wah, and D. Gašević, “Vision, challenges, roles and research issues of artificial intelligence in education”, Computers and Education: Artificial Intelligence, vol. 1, Art. no. 100001, 2020, https://doi.org/10.1016/j.caeai.2020.100001.

S. Järvelä and A. F. Hadwin, “New frontiers: Regulating learning in CSCL”, Educational Psychologist, vol. 48, no. 1, pp. 25–39, 2013, https://doi.org/10.1080/00461520.2012.748006.

S. Järvelä, A. Nguyen, and A. F. Hadwin, “Human and artificial intelligence collaboration for socially shared regulation in learning”, British Journal of Educational Technology, vol. 54, no. 5, pp. 1057–1076, 2023, https://doi.org/10.1111/bjet.13325.

E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, et al., “ChatGPT for good?”, On opportunities and challenges of large language models for education. Learning and Individual Differences, vol. 103, Art. no. 102274, 2023, https://doi.org/10.1016/j.lindif.2023.102274.

M. Kazemitabaar, J. Chow, C. Ma, B. J. Ericson, D. Weintrop, and T. Grossman, “Studying the effect of AI code generators on supporting novice learners in introductory programming”, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Art. no. 455, 2023, https://doi.org/10.1145/3544548.3580919.

L. Lawrence, V. Echeverria, K. Yang, V. Aleven, and N. Rummel, “How teachers conceptualise shared control with an AI co-orchestration tool: A multiyear teacher-centred design process”, British Journal of Educational Technology, vol. 55, no. 3, pp. 823–844, 2024a, https://doi.org/10.1111/bjet.13372.

L. Lawrence, E. Mercier, T. Tucker Parks, N. Bosch, and L. Paquette, “Accuracy and effectiveness of an orchestration tool on instructors’ interventions and groups’ collaboration”, Computers and Education Open, vol. 7, Art. no. 100203, 2024b, https://doi.org/10.1016/j.caeo.2024.100203.

J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance”, Human Factors, vol. 46, no. 1, pp. 50–80, 2004, https://doi.org/10.1518/hfes.46.1.50_30392.

Z. C. Lipton, “The mythos of model interpretability”, Queue, vol. 16, no. 3, pp. 31–57, 2018, https://doi.org/10.1145/3236386.3241340.

C. K. Lo, “What is the impact of ChatGPT on education?”, A rapid review of the literature. Education Sciences, vol. 13, no. 4, Art. no. 410, 2023, https://doi.org/10.3390/educsci13040410.

M. A. Madaio, L. Stark, J. Wortman Vaughan, and H. Wallach, “Co-designing checklists to understand organizational challenges and opportunities around fairness in AI”, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2020, https://doi.org/10.1145/3313831.3376445.

M. A. Marks, J. E. Mathieu, and S. J. Zaccaro, “A temporally based framework and taxonomy of team processes”, Journal of Applied Psychology, vol. 86, no. 3, pp. 356–376, 2001, https://doi.org/10.2307/259182.

J. Mathieu, M. T. Maynard, T. Rapp, and L. Gilson, “Team effectiveness 1997–2007: A review of recent advancements and a glimpse into the future”, Journal of Management, vol. 34, no. 3, pp. 410–476, 2008, https://doi.org/10.1177/0149206308316061.

C. McGregor, “Generative artificial intelligence and human collaboration in educational design”, Computer Applications in Engineering Education, vol. 34, no. 4, Art. no. e70238, 2026, https://doi.org/10.1002/cae.70238.

T. Miller, “Explanation in artificial intelligence: Insights from the social sciences”, Artificial Intelligence, vol. 267, pp. 1–38, 2019, https://doi.org/10.1016/j.artint.2018.07.007.

M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, “Model cards for model reporting”, Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 220–229, 2019, https://doi.org/10.1145/3287560.3287596.

T. Nazaretsky, H. Gabbay, and T. Käser, “Can students judge like experts?”, A large-scale study on the pedagogical quality of AI and human personalized formative feedback. Computers and Education: Artificial Intelligence, vol. 10, Art. no. 100533, 2026, https://doi.org/10.1016/j.caeai.2025.100533.

D. J. Nicol and D. Macfarlane-Dick, “Formative assessment and self-regulated learning: A model and seven principles of good feedback practice”, Studies in Higher Education, vol. 31, no. 2, pp. 199–218, 2006, https://doi.org/10.1080/03075070600572090.

S. Noy and W. Zhang, “Experimental evidence on the productivity effects of generative artificial intelligence”, Science, vol. 381, no. 6654, pp. 187–192, 2023, https://doi.org/10.1126/science.adh2586.

F. Ouyang and P. Jiao, “Artificial intelligence in education: The three paradigms”, Computers and Education: Artificial Intelligence, vol. 2, Art. no. 100020, 2021, https://doi.org/10.1016/j.caeai.2021.100020.

K. Pahi, S. Hawlader, E. Hicks, A. Zaman, and V. Phan, “Enhancing active learning through collaboration between human teachers and generative AI”, Computers and Education Open, vol. 6, Art. no. 100183, 2024, https://doi.org/10.1016/j.caeo.2024.100183.

E. Panadero, “A review of self-regulated learning: Six models and four directions for research”, Frontiers in Psychology, vol. 8, Art. no. 422, 2017, https://doi.org/10.3389/fpsyg.2017.00422.

E. Panadero and A. Jonsson, “The use of scoring rubrics for formative assessment purposes revisited: A review”, Educational Research Review, vol. 9, pp. 129–144, 2013, https://doi.org/10.1016/j.edurev.2013.01.002.

R. Parasuraman and V. Riley, “Humans and automation: Use, misuse, disuse, abuse”, Human Factors, vol. 39, no. 2, pp. 230–253, 1997, https://doi.org/10.1518/001872097778543886.

R. Parasuraman, T. B. Sheridan, and C. D. Wickens, “A model for types and levels of human interaction with automation”, IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, vol. 30, no. 3, pp. 286–297, 2000, https://doi.org/10.1109/3468.844354.

M. Prince, “Does active learning work?”, A review of the research. Journal of Engineering Education, vol. 93, no. 3, pp. 223–231, 2004, https://doi.org/10.1002/j.2168-9830.2004.tb00809.x.

M. T. Ribeiro, S. Singh, and C. Guestrin, ““Why should I trust you?” Explaining the predictions of any classifier”, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144, 2016, https://doi.org/10.1145/2939672.2939778.

D. R. Sadler, “Formative assessment and the design of instructional systems”, Instructional Science, vol. 18, pp. 119–144, 1989, https://doi.org/10.1007/BF00117714.

E. Salas, D. E. Sims, and C. S. Burke, “Is there a “Big Five” in teamwork?”, Small Group Research, vol. 36, no. 5, pp. 555–599, 2005, https://doi.org/10.1177/1046496405277134.

S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic generation of programming exercises and code explanations using large language models”, Proceedings of the 2022 ACM Conference on International Computing Education Research, pp. 27–43, 2022, https://doi.org/10.1145/3501385.3543957.

I. Seeber, E. Bittner, R. O. Briggs, T. de Vreede, G.-J. de Vreede, A. Elkins, R. Maier, A. B. Merz, S. Oeste-Reiß, N. Randrup, G. Schwabe, and M. Söllner, “Machines as teammates: A research agenda on AI in team collaboration”, Information & Management, vol. 57, no. 2, Art. no. 103174, 2020, https://doi.org/10.1016/j.im.2019.103174.

B. Shneiderman, “Human-centered artificial intelligence: Reliable, safe & trustworthy”, International Journal of Human-Computer Interaction, vol. 36, no. 6, pp. 495–504, 2020, https://doi.org/10.1080/10447318.2020.1741118.

V. J. Shute, “Focus on formative feedback”, Review of Educational Research, vol. 78, no. 1, pp. 153–189, 2008, https://doi.org/10.3102/0034654307313795.

G. Stasser and W. Titus, “Pooling of unshared information in group decision making: Biased information sampling during discussion”, Journal of Personality and Social Psychology, vol. 48, no. 6, pp. 1467–1478, 1985, https://doi.org/10.1037/0022-3514.48.6.1467.

J. Strobel and A. van Barneveld, “When is PBL more effective?”, A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning, vol. 3, no. 1, pp. 44–58, 2009, https://doi.org/10.7771/1541-5015.1046.

H. Suresh and J. V. Guttag, “A framework for understanding sources of harm throughout the machine learning life cycle”, Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Art. no. 17, 2021, https://doi.org/10.1145/3465416.3483305.

J. Tai, R. Ajjawi, D. Boud, P. Dawson, and E. Panadero, “Developing evaluative judgement: Enabling students to make decisions about the quality of work”, Higher Education, vol. 76, pp. 467–481, 2018, https://doi.org/10.1007/s10734-017-0220-3.

A. Tlili, B. Shehata, M. A. Adarkwah, A. Bozkurt, D. T. Hickey, R. Huang, and B. Agyemang, “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education”, Smart Learning Environments, vol. 10, Art. no. 15, 2023, https://doi.org/10.1186/s40561-023-00237-x.

K. J. Topping, “Peer assessment between students in colleges and universities”, Review of Educational Research, vol. 68, no. 3, pp. 249–276, 1998, https://doi.org/10.3102/00346543068003249.

B. W. Tuckman, “Developmental sequence in small groups”, Psychological Bulletin, vol. 63, no. 6, pp. 384–399, 1965, https://doi.org/10.1037/h0022100.

M. van Zundert, D. Sluijsmans, and J. van Merriënboer, “Effective peer assessment processes: Research findings and future directions”, Learning and Instruction, vol. 20, no. 4, pp. 270–279, 2010, https://doi.org/10.1016/j.learninstruc.2009.08.004.

B. Wilder, E. Horvitz, and E. Kamar, “Learning to complement humans”, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pp. 1526–1533, 2020, https://doi.org/10.24963/ijcai.2020/212.

N. E. Winstone, R. A. Nash, M. Parker, and J. Rowntree, “Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes”, Educational Psychologist, vol. 52, no. 1, pp. 17–37, 2017, https://doi.org/10.1080/00461520.2016.1207538.

D. Wood, J. S. Bruner, and G. Ross, “The role of tutoring in problem solving”, Journal of Child Psychology and Psychiatry, vol. 17, no. 2, pp. 89–100, 1976, https://doi.org/10.1111/j.1469-7610.1976.tb00381.x.

O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, “Systematic review of research on artificial intelligence applications in higher education—Where are the educators?”, International Journal of Educational Technology in Higher Education, vol. 16, Art. no. 39, 2019, https://doi.org/10.1186/s41239-019-0171-0.