Assessment scenario: Students in ENGR 2301 present highlights of their weeks-long research projects in 3- to 5-minute Lightning Talks, but it's become increasingly difficult to assess whether they used AI, and, if so, how well they engaged in critical thinking about the findings. Learn how the addition of Q&A sessions gave students opportunities to demonstrate their knowledge and on-the-spot thinking while enabling the instructor to better assess their analysis of findings.
Instructor: Jenny Patton
Department: Engineering Education
Course: ENGR 2301 Citizenship in Engineering Contexts
Contact: patton.220@osu.edu
Course Details
Format: In-person
Typical Enrollment: 24
Instructional Staffing: One primary instructor, one undergraduate TA
Learning Context: Students have taken a GE foundation writing and info literacy course
GE Category: Research and Creative Inquiry
Expected Learning Outcomes
- Use credible and relevant sources of information, evaluate assumptions, and consider alternative viewpoints or hypothesis to express ideas and develop arguments
- Develop scholarly, creative, or professional products that are meaningful to them and their audience
Problem
Student analysis of research findings is difficult to assess
In 3- to 5-minute Lightning Talks over two days, each of my 24 students communicates highlights of their weeks-long research in an engaging presentation (6-8 slides required, bulleted notes optional) designed to help the audience understand the problem or need area explored and argue for the significance of addressing the problem. Student success has been assessed according to content, audience awareness, visuals, delivery and presentation skills, citations, mechanics, and time. In this AI era, it’s become difficult to assess whether they used AI (not allowed for the research portion) and, if so, how well they engaged in critical thinking about the findings.
Change
Q&A sessions showcase on-your-feet thinking
A few days prior to the Lightning Talks, I introduced students to the format of academic conferences in which presenters are expected to know and understand their topics beyond what they share in prepared talks to respond on the spot to questions. I let them know that we will follow this format, so they needed to
- demonstrate their knowledge of related research, its implications, and their own analysis beyond what they share in their prepared presentation
- take notes and write down questions to ask of classmate presenters
During two Q&A sessions, I took notes assessing how deeply each presenter answered questions, and I tracked who asked what questions. In written feedback to the presenters, I commented on their responses to the questions posed to them immediately after their presentation and assessed their questions as part of their overall class participation grade.
The Q&A
- showcased how well they responded to on-the-spot questions
- helped me assess how deeply students understood their research beyond what they had prepared to present
- resulted in a higher level of audience engagement among classmates than in the past
However, this adjustment took up more class time than I had allocated, so other class content was cut short.
Next steps
Use AI as a feedback tool to prep for 3-day Colloquium with Q&As
In the future, I will integrate AI as a feedback tool and as a way to prep for the Q&A. I will reframe the Lightning Talks as part of a three-day Colloquium that groups the talks by topic (i.e., climate change, digital citizenship, civil/social inequities) and allow more time for a post-presentation Q&A session. I revised the lesson plan, assignment, and rubric to better teach students how to develop good questions and to formally incorporate the Q&A session, requiring all students to ask a certain number of questions that would also be assessed as part of a new Colloquium format instead of as part of the class participation grade.
For now, I plan to ask students how well AI helped them prepare for the questions they received; in the future, I may formalize this through a simple set of Likert scale questions.
Resources
Further explore this assessment scenario
Note: These changes represent how Jenny Patton has slightly modified the long-standing ENGR 2301 Lightning Talk assignment created by Dr. Jennifer Herman and Professor Leah Wahlin.