Context & Scale
Assessment serves multiple purposes in higher education. It frames the curriculum and drives student learning. It enables teachers to gauge student understanding and make judgments on the effectiveness of their teaching. It is also enables universities to assess whether a student has achieved the program learning outcomes that the award of a degree designates. Assessing students orally is a valid way to assure learning and has been around since universities began in the Middle Ages. The viva voce is probably the best-known type of oral assessment but is more like an exam than a conversation, and as such can induce stress and anxiety in students (Alcorn & Cheesman, 2022). The idea of an oral assessment based on a free-flowing conversation set in a real-world scenario though, is relatively new. The Interactive Oral (IO) came to prominence in COVID-19 lockdowns as an alternative to on-campus exams (Logan et al., 2020) and is now considered a viable, scalable and sustainable means of assuring learning in the AI age (Lodge et al., 2023).
Problem
Designing assessment that assures learning in the AI Age is a challenge. Concerns over contract cheating (Harper et al., 2021) and the rapid rise of ChatGPT (Lo, 2023) challenge traditional methods of assessment and the assurance of learning, as it becomes more difficult to detect and assure that written work is original. Returning to supervised on-campus exams is one option to assure assessment, but the pedagogical value of the exam is questionable as it focuses on short-term rather than long-term learning (Boud, 2000). How then can we assess student learning, with a view to their future needs and ensure that a student has achieved the learning outcomes that we say they have?
Solution
The Interactive Oral (IO), in which students are required to orally explain their decisions and processes in an imagined real-world scenario, is a pedagogically sound and assured means of assessing learning in the AI age. It has also been found to be less stress-inducing for students than other types of oral exams such as the viva voce (Logan & Sotiriadou, 2020). The IO is not an examination and nor is it a test of language proficiency. All students are assessed equally on the quality of their ideas and thinking, not their language proficiency, and is designed for supportive, inclusive assessment. Most assessment types only satisfy two dimensions of the Assessment Iron Triangle: Integrity | Authenticity | Scalability (Hillier & Fluck, 2017), but the IO excels in all three. The IO is ideally situated as the final assessment item in a unit because it gives the student – and the assessor – the opportunity to look back on what was learned and allows the student to look forward to a future version of themselves. It is therefore highly motivating and engaging.
Interactive Orals can be facilitated either online or on-campus and can be used in any discipline in which real-world scenarios can be imagined (see Karltun & Karltun, 2014; Logan & Sotiriadou, 2020; Ward et al., 2023 for examples). The IO is sustainable as it requires no special equipment (Hillier, 2023) and scalable, and has been used in classes of up to 800 students (Logan et al., 2023). Unlike voce vivas, which can induce stress, students view IOs positively, as an opportunity for real-world engagement (Logan et al., 2017).
Implementation
Investigate where an IO might fit into your assessment design, aligning it to subject and programme LOs (5 weeks before teaching period)
Plan strategy to prepare students for conversational assessment
Orient the teaching team to the approach for the IO (preparation tasks and assessment)
Finalise the scenario for the IO and the practice IO
Finalise the marking rubric for the IO, ensuring it also guides the flow of the conversation
Record the Practice IO
Finalise IO student briefing documents/resources and practice IO video
Consult and train markers to facilitate IO assessment. In a large course this can be done in a workshop that provides markers with the opportunity to practice with each other using the conversation prompts and rubric
Set up booking schedule for IO slots. For large courses, this requires a booking system such as Microsoft Bookings or Calendly to effectively manage scheduling
Brief students on the IO, and start preparing them for the conversational oral assessment (Week 1)
Students mark a practice IO in class with marking rubric. IO booking schedule made available. Students book their IO slot (>3 weeks before conducting IOs)
Facilitate IOs, mark, moderate and provide feedback to students
Note. The above steps have been adapted from Logan-Fleming et al., 2024.
Examples of pattern in use
Interactive Oral in WORK5003 Management and Organisations
Practice Interactive Oral
An Interactive Oral was designed and implemented as the final assessment item (35%) in WORK5003 Management and Organisations coordinated by Dr Elly Meredith. To familiarise students with this novel type of assessment and help them understand how they will be marked, students used the IO marking rubric to mark a practice IO a few weeks in advance of the actual assessment. This practice IO was shot in a studio with the author playing the CEO and the Unit Coordinator, Dr Elly Meredith, playing the management consultant. The student performance was deliberately constructed to be at a Credit level, with one key aspect of the performance becoming a teaching point for the in-class activity – the common error of not keeping to the stated time limit of a presentation. The scenario was designed to run along similar lines to the real IO but different enough so students would not be tempted to copy the student performance when it came time for their assessmeent. Students watched the video and used the marking rubric to arrive at a mark. Most students gave the practice IO a credit grade. View the Practice IO below.
Video of Practice IO with the student as Management Consultant on left in conversation with the assessor as CEO on right.
Unit Coordinator’s thoughts on IOs
- Practice IO: In marking the practice IO, students were given the actual rubric that would be used in their assessment. This was helpful as it showed how the marks would be allocated against their presentation. Interestingly, in doing the practice IO, the students were asked to mark me, as the consultant, in the video. All of the students, bar one, gave the mark as a credit. The student who gave a higher mark, gave the presentation a very low distinction. So, in essence, the mark given by the students was the same as the mark intended. The student found the practice IO very useful. They felt it gave them confidence in preparing and presenting their cases, especially as they understood how marks could be achieved. The video was made available to the students to watch later, and they found this very useful as they identified areas where they could improve their presentation.
- Scheduling: The scheduling of the IO was easy as I used Microsoft Bookings which is an available app in Outlook. Students selected a 30-minute slots over a three-day period in Exam Week on a ‘first in best dressed’ served basis. However, if a student had a particular time required, then it could be negotiated with the student who had selected the time. This did not happen, but it was a possibility.
- Conducting the IO: Conducting the IO is mentally taxing for the me as the examiner, and I designed the IO to build in breaks. The breaks were built in after four (4) sessions where are break of 1.5 hours. However, this did mean that the afternoon session could be long and there were five (5) sessions that the students could nominate. Since the students made the afternoon sessions their preferred time, it was difficult to manage, and I would need to build in breaks after three (3) sessions. Possibly a 1hour break after three (3) sessions, but as this was the first time the IO had been developed and used, it would need another session to work out what would be best.
- Marking the IO: The rubric was very helpful for the students, it showed how and where their marks were being allocated.
- Feedback from students and teacher: As this was the first time the IO was introduced for the students, I did ask for feedback from them. The result was overwhelming positive, one student said how much she had enjoyed the learning that was associated with IO. She said that in previous exams she learned what she needed to know, but three weeks later she could remember anything. This was not the case with the IO because what she was studying linked back to the case and she connected it with a real scenario. Another student also felt that the IO was something that he might be doing in a job, and having the opportunity to answer questions from different areas prepared him for the future. One student told me that I have to ‘keep the IO, it was so much fun!’I enjoyed that process and found it helpful and for me it showed how well the students prepared themselves for the IO. As the marking was contemporaneous it made it easier to mark, though I might build in more time to complete it.
Leave a Reply