Interview with Dr. Nimesh Patel
Written By Sreeja Roy
Interview with Dr. Nimesh Patel
The dreaded exam season is quickly approaching and all that time we thought we had is disappearing just as fast, or even faster. Suddenly doing back to back Sports Ball and Jashn doesn’t seem like such a good idea anymore… There’s too many lectures that haven’t been reviewed, more lectures that haven’t even been viewed, overall too much content to cram. Anxiety-fueled study sessions fill the day and restless sleep overcomes us while we think about how these sadistic examiners sleep at night knowing the misery they put us through.
Turns out, these examiners aren’t sleeping that well either. But not for the reason we might think. Apparently writing exams that 300 20-year olds aren’t complaining about is quite a difficult process, and no matter what you do students will have a fundamental problem with it. There’s process after process, lists of people to go through, examiners to train, content to revise and questions to answer from students and professors alike. These ungrateful students I mention, are me, and to satisfy my curiosity again I terrorize UCL Medical School (UCLMS). But this time I leave the students alone and instead turn my attention to a member of staff, Dr. Nimesh Patel. In my three years in UCLMS the exams have changed significantly and I was curious as to why.
This interview has been edited for length and clarity.
I: I remember some of the changes that you made when you first came in. We had a lecture about how questions are assessed; this transparency was valued as we could understand how to get marks. What are some other changes that you’ve made and why?
NP: The biggest change that I made when I came to UCL was giving students back their formative exam papers.I was thinking What? You don’t get your formative questions back? That’s the whole point of formative assessment - it’s an opportunity to learn. Another big change is how we run things in the background. There’s an exam platform that we used to have called Speedwell. From my perspective, it is a very basic system… and very cumbersome. [Uploading questions and getting them sent to external examiners] was very convoluted and mismanaged. It was all very complicated, the team would spend 3-4 days just setting up an exam in the system. And then, someone would have to double check it.
Now we use Exam-Write and everything stays within the system - nothing needs to be uploaded, the delivery happens there. With our OSCEs we use a platform called Risr. It helps us put the stations together and review them.
The OSCE marksheets have changed as well so they are clearer. I know previously students complain that going to a certain site will mean it's harder. So I thought- let’s standardize this a bit more. Now we have the same template that runs across all years, rather than allowing examiners to have that subjective judgement around what you’re doing. We don’t allow examiners who are specialists in their field to examine their station, so we’ve been very prescriptive in our markschemes. Now the mark scheme says that- in order for the student to get the full marks, they have to have done certain things.
We’ve also allocated domains, like clinical reasoning, and average scores in each for every questionThe Year 2 cohort average for clinical reasoning is 55% whereas for Year 6 it’s about 80%.
I: There are also complaints about examiners in the OSCEs, how do you plan on tackling that?
NP: We have improved our Examiner training. So previously it was a half an hour self directed session. Now, we’ve got this new training that hopefully helps examine the biases in assessing. We use the example of Susan Boyle… to emphasise to the examiners to not judge our students… by their appearances, by how they speak. Be conscious of your biases. Everyone has that. And we make sure that our examiners are just conscious of their own.
I: In terms of OSCEs and the content we are examined on it feels like there are certain topics that are very heavily examined and certain topics that are never touched on like pharmacology. What is the reasoning for this because it feels disproportionate to how many lectures we’ve had on the content.
NP: When we design assessments, we talk to the year leads about, what proportion of questions are going to come from each module. We know that we have a limit of 40% very short answer (VSA) questions, so module leads need to decide how many SBAs they are going to ask. The year leads have a responsibility to then tell the module leads that, make sure that questions contain the breadth of the module’s content. So there’s almost three steps before we even get to the person who’s writing questions.
Every question has to have a lecture or tutorial or skill session attached to it. I always ask lecturers to think about the questions’ relevance. Why is it important for a Year 1 or 2 to know this?
I: Where did all the written questions go? The longest questions we answer are ‘Very Short Answers (VSAs). It sometimes feels like a word association game rather than “do you actually understand this process”.
NP: So there’s two key issues. The lesser issue is that the Exam-Write system doesn’t allow us to do that type of questions. Part of the reason for going to Exam-Write was not having that question type because they can be very challenging when it comes to marking with many inconsistencies. The module leads would often comment that students had written too much and the answer was buried in there. It didn’t show what the student had actually learned, because they’re just regurgitating information and it’s not really showing that they understand the content. Sometimes students wouldn’t know exactly what’s been asked. Essentially, it comes down to question quality. With all of those factors put together, it takes a lot of time to mark.
VSA is a good discriminating question because in a clinic, doctors aren’t presented with A, B, C, etc. At some point the VSA style of question may be introduced to the Medical Licencing Exam (MLA), so by the time students get to Year 6 at least they know the question style.
I hope this interview provided you with some insight into how exam questions are created and the thinking behind the decisions made. Dr. Patel provided some examples of how UCLMS has improved its exam system, in my opinion for the better. However, there are still some things I question. For example, the use of a very strict, prescriptive mark scheme for the examiners in the OSCEs. While I strongly agree that examiners should have longer training sessions to help identify biases they may internalise, I question whether having a prescriptive markscheme is helpful, as I believe that a biased examiner will always be a biased examiner. Instead it may be more helpful to have a stricter criteria for choosing examiners. It would also be interesting to know how examiners are chosen, and whether the selection process could reduce the bias inherent in the examination system. Furthermore, I think not allowing examiners any freedom in their marking will fail to reflect the kind of bias doctors will face from patients.
In my (admittedly naïve) opinion, it may be more beneficial for OSCE examiners to have to justify their scores with either student quotations or timestamps from the recorded session of an examination session. For example if a student is marked down for being unprofessional because of the way they talk it would be a lot more insightful for the student to receive that feedback with examples. Did the examiner find this student unprofessional because of the words they used and mannerisms or is it because of their accent? This sort of feedback is important for improvement, as the former allows the student to know what to change but the latter is important for UCLMS to flag an examiner’s bias. While I understand this adds time and work, I believe this provides transparency. I don’t see the point in being examined by a human examiner if the markscheme is so strict that there’s no space for human judgement.
In a similar vein, I think removing longer written questions negatively impacts medical students’ ability to effectively communicate ideas in writing. For example explaining how a biochemical pathway occurs. Written communication is an integral part of a doctor’s job, from writing referrals to research papers. I believe that not being assessed on how well we communicate, in my view, is detrimental to our future practice. The reason longer questions were eliminated from examinations is because students waffleand it takes too long to mark. While I understand this from an efficiency standpoint, I think extra time should be given to understanding students’ drivel and marks should be awarded based on clarity and ease of comprehension. I suck at writing, my grammer is atrocious and I fall into the stereotype of medical students not being able to write well. Just because the MLA doesn’t assess us on this doesn’t mean we should be deprived of the opportunity to practice this skill.
Finally, I have to thank Dr. Patel for giving me this interview and the changes he has implemented in his short time here. I urge all of you to critically evaluate the examination system we have and ask the same questions he does. Why is it important we know this? If you believe there’s something wrong in the way you’re assessed, something you’d like removed or added, bring it up, make a big fuss, but be ready to defend yourself. It is an extreme privilege we all have to learn medicine, so make the most of it. Make sure you get all you can from UCLMS.
Read the rest of the edition here.