Friday 8 November - Short Presentations
10:50-12:30, Room: 1.11, 2nd floor

Feedback Theme

Chair: Assistant Professor Carlos F. Collares, Maastricht University

The use of Natural Language Processing to aid understanding of free text feedback

* Brian Lunn
* Corresponding author: Brian Lunn, Newcastle University, United Kingdom
brian.lunn@newcastle.ac.uk 

Feedback on teaching and assessment often comprises a mix of quantitative (number based) and qualitative (free text) elements. By dint of being numerical the former is often seen as more ‘objective’ despite the latter containing rich data in the form of narrative. Another issue is that in reporting free text there can be bias and selectivity introduced. With the development of algorithms to allow machine processing of free text using Free/Open Source Text Analysis Software it is possible to understand the semantics of what is written and quantify the weight of meaning and/or intent of the writer(s). An analysis of feedback in an undergraduate medical programme was undertaken using commonly available textual analyses for the free software environment for statistical computing and graphics, R. An introduction to the methodology will be presented, along with data that demonstrates how such tools can be used in understanding feedback, whether about or from students. This approach can also be used to allow collection of data about the quality of feedback provided to students, which can then be used to better train and inform their practice. Amongst issues covered will be the development of a corpus that allows sentiment analysis in a medical context as in standard corpora negative sentiment is attached to words that have an affectless technical meaning in medicine.

Developing dental student’s feedback literacy and development using a self-regulated learning approach to support reflective responses to assessment

* Mark Jellicoe, Alex M. Forsythe, Kathryn Fox, Luke Dawson
* Corresponding author: Mark Jellicoe, University of Liverpool, United Kingdom
mark.jellicoe@liverpool.ac.uk 

Background 
Evidence suggests that students’ grade focus may be a destabilising force. This renders some students paralysed and may undermine longitudinal programmatic assessment strategies. The self-management skills necessary for continuing professional development may pay the price for such a maladaptive focus. It has been suggested that student self-management skills need to be scaffolded and modelled to secure patient safety (Molloy, 2019). 

Summary of work 
Following previous work, a pilot project exposed second-year dental students to four workshops focused on core aspects of Zimmerman’s self-regulated learning model (Panadero, 2017; Zimmerman, 1989). These included lectures and workshops focused on the utility of feedback and reflection processes, values and goal setting, and approaches towards motivated performance including beliefs and mindsets. Sessions were distributed across two academic semesters, with a final summary session. Following the sessions, students volunteered to participate in a focus group session. 

Summary of results 
Approximately half of the students in the cohort attended these optional timetabled sessions and participated in the active learning approaches. Evaluation of data indicated that students attending sessions were those who held moderately high levels of self-efficacy and growth mindset beliefs. Within active learning sessions, students endorsed the utility of feedback comments that provided coaching and mentoring support, versus those focusing on assessment and evaluation. Thematic analysis was used to analyse focus group data (Braun & Clarke, 2006). Themes indicated that sessions promoted ‘perspective taking’ and ‘growth focused transitions’ when considering feedback. These were reported to support ‘stress and anxiety reduction’ and ‘realistic development ‘ that was ‘future focused’. It was not possible to measure the behaviours of those students that did not attend, it may be that these students are most in need of intervention. 

Discussion and conclusion 
The approach piloted here aims to scaffold students development focus around the integration of feedback appears to be scalable. This approach appears to enable students to answer Hattie and Timperley’s (2007) three fundamental questions relating to effective feedback integration; ‘where am I going?’; ‘how am I going’; and ‘what do I have to do next?’. The approach does this by scaffolding students' learning with practical tools that can be used to aid development around reflection, goal setting and motivated performance that should lead to marginal gains in development (Winstone & Carless, 2019a, 2019b). The findings from the pilot will be used to develop personal development and wellbeing curricula, with research tracking longitudinal changes in development. 

Take home message 
For best patient outcomes, students need to be trained in the skills necessary to secure developmental change. A pilot approach focused on developing the self-regulatory skills to secure feedback integration shows promise in securing these developments.

Does the publication of student created assessment resources have any effect on student performance in summative assessments?

* Alison Sturrock, Daniel Ntuiabiane, Osman Janjua
* Corresponding author: Alison Sturrock, UCL Medical School, United Kingdom
a.sturrock@ucl.ac.uk 

Background 
At UCL medical school, over the last couple of years there has been increasing complaints from students about the banks of previously used questions that are held by student societies/clubs. Students believe that by having access to these banks, students who are members of these clubs have an unfair advantage in their summative assessments. 

Summary of work  
To try to address this issue, UCL medical school introduced an Amnesty in which students could submit (anonymously if they wished) any material related to assessments. These submissions were subsequently uploaded to a SharePoint site which could be accessed by all students. Although we were aware that some material uploaded onto the Amnesty site was already included in the summative assessment, we did not alter the assessment. Instead, we will analyse the results to see if there has been any effect on the marks that students have achieved.  

Summary of results  
The Amnesty site was open from December 08-January 09 and again from May 09-date. We received over 125 submissions which include past SBA questions, OSCE stations and tutorial notes. All of this material was uploaded onto the SharePoint site at least one month before their summative assessments. We will analyse the results of this year’s summative assessments (SBA and OSCE) compared to the previous three years. We will also seek the views of this process from students. 

Discussion and conclusion  
As the students have not yet sat their summative assessments, we do not know if the release of these resources has had any effect on student performance. However, the feedback from students so far has been positive with a significant increase in satisfaction about the fairness of the assessment process.  

Take home message  
We will analyse whether the publication of a large amount of student created resources related to assessment has any effect on student performance.

Ιnterprofessional Education in European Medical Schools

* Evangelos Papageorgiou, Beatriz Atienza Carbonell, Irem Aktar
* Corresponding author: Evangelos Papageorgiou, European Medical Students’ Association (EMSA), Greece
vagpapageorge@gmail.com 

Background 
The European Medical Students´ Association is an organisation representing medical students from all across Europe. One of EMSA’s pillars of action is Medical Education and one of this pillar’s priorities is the promotion of Interprofessional Education in Europe. EMSA is also a partner of the European Healthcare Students’ Associations’ Summit (EHSAS), a group of medical, pharmaceutical, dental and psychology students that have gathered to foster communication between healthcare students of different backgrounds and exchange good practices about interprofessional education in European universities. For this reason, EMSA and EHSAS have planned many projects over the years so they can highlight the importance of interprofessional education. Following, an EHSAS survey that was launched in 2014, EMSA decided to open a new survey for its members so we can understand what medical students specifically think about interprofessionalism and how this is integrated in the medical education. 

Summary of Work 
A group of students with experience in EMSA and the Medical Education field was gathered in order to structure a cross-sectional survey and plan ways to disseminate it. When the survey was created, it was disseminated in a small group of medical students and experts in order to validate it. After the validation period was completed, the survey was promoted online in EMSA channels, trying to reach as many of our members as possible. 

Summary of Results 
In total, there were 326 responses in the online survey. The students are coming from 28 different European countries representing all levels of medical school (from 1st to 6th year students). More than 50% of the students (171 answers) declared that they haven’t collaborated with other healthcare students during their medical training while the vast majority of them (274 answers) think that it is really important to study and collaborate with other health professions. More results will be presented during the conference. 

Discussion & Conclusion 
Medical students are aware of the importance of Interprofessional Education but they still feel the need of integration of this concept into their curriculum. Big proportion of them present confusion about what precise roles other professions have, nevertheless are highly interested in the topic and they would like to get more in touch with students of different healthcare backgrounds. The answers and best practices strongly vary depending on each country. There is a huge gap in the standardization of IPE in medical schools across the continent as well as preclinical and clinical year applications of IPE. In conclusion, future doctors realize the need to take part in multidisciplinary teams so they can provide quality healthcare and therefore, universities must pay more attention to the concept of IPE during the undergraduate training. 

Take-home message 
Interprofessional Education can be very beneficial for medical students and it should be integrated more in the medical students’ curriculum in Europe.

Evaluation of undergraduate medical students’ self-assessment of reflective writing and in a structured clinical assessment (ISCE)

* Elizabeth P Metcalf, Sarju Patel, Timothy Johnson, Jeff Allen, Sue Emerson
* Corresponding author: Elizabeth P Metcalf, Cardiff University School of Medicine, United Kingdom
metcalfep@cf.ac.uk 

Background 
Doctors require reflective competencies1. Self-assessment is beneficial for students self-validation skills and tutors identifying development areas .2,3 Accurate self-reflection enables clinicians understanding of strengths/ weaknesses, to formulate personal learning objectives, gain awareness of learning styles and sense of achievement4. Having self-belief in one’s competencies can be highly motivating. Equally, believing one is incompetent can undermine motivation and lead to poorer engagement5,6. Students’ ability to accurately self-assess varies and may impact confidence, mental health or patient safety 7.It is important to develop the ability of students to accurately self-assess their competencies to maximise engagement and optimise learning. 

Summary of Work 
Various strategies develop students’ ability to self-assess, however before the impact of potential intervention can be evaluated, an understanding of students’ current abilities was required. We undertook a pilot study in 2017/8 where year 2 students self-evaluated an assignment before submitting assessment by academic staff. Secondly, students self-assessed performance in their formative Integrated Structured Clinical Examination (ISCE) assessment, in addition to receiving examiner and peer marking and feedback. 

Summary of Results 
Analysis showed no statistical difference between overall marks awarded by assessor and student. Analysis of individual domains indicated students were more likely to award a lower mark than assessor when the assessor awarded the highest mark for the domain. Students were more likely to award a higher mark than the assessor when assessor awarded the lowest pass mark for the domain. Students significantly underestimated their performance in 1% and in 3% (Reflective writing) and 1.5% (ISCE), the students significantly overestimated performance. 

Discussion & Conclusion 
The ability to self-assess allows progress from an assessment driven internal process to a process that promotes personal responsibility for achieving required learning outcomes 8. The results suggest most students could either self-assess their work appropriately, or were reluctant to overestimate their abilities and tend to assess themselves as good or competent. We demonstrated that it is possible to determine which of these categories any student falls. This study highlights significant issues with regards some students’ metacognitive abilities. Some higher performing students were more critical of their work than others. Perfectionism tendencies can lead to academic burnout9 and in the longer-term can have a detrimental effect on mental health 8,10. Self-deprecating perfectionism can have a negative effect on academic self-efficacy and ultimately result in academic burnout, therefore it is important to have educational and counselling interventions to improve academic performance and help prevent academic burnout 6. Less well performing students can overestimated their competencies and/or quality of their work. This may be due in part because individuals who are unskilled lack the metacognitive ability to realise their incompetence 8,11. 

Take-home Message 
Self-assessment can enhance learning and self-regulation of work. Most students can self-assess correctly, however, borderline students over value their work raising concerns regarding patient safety, whereas higher performing students under value their work raising concerns regarding welfare. Introducing self-assessment and feedback opportunities can help address metacognitive deficits and address patient safety and student welfare concerns. 

References 
1. GMC. 2018. Outcomes for graduates. Available at: www.gmc-uk.org/education/standards-guidance-and-curricula/standards-and-outcomes/outcomes-for-graduates   
2. Eva, K. W. and Regehr, G. 2005. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 80(10 Suppl), pp. S46-54. 
3. Blanch-Hartigan, D. 2011. Medical students' self-assessment of performance: results from three meta-analyses. Patient Education & Counseling 84(1), pp. 3-9. doi: 10.1016/j.pec.2010.06.037Grant et al., 2006 
4. Cole, D. A. et al. 1999a. Children's Over- and Underestimation of Academic Competence: A Longitudinal Study of Gender Differences, Depression, and Anxiety. 70(2), pp. 459-473. doi: doi:10.1111/1467-8624.00033 
5. Yu, J. H. et al. 2018. Do basic psychological needs affect student engagement in medical school? Korean Journal of Medical Education 30(3), pp. 237-241. 
6. Cole, D. A. et al. 1999b. Children's Over- and Underestimation of Academic Competence: A Longitudinal Study of Gender Differences, Depression, and Anxiety. Child Development 70(2), pp. 459-473. doi: doi:10.1111/1467-8624.00033 
7. Kruger, J. and Dunning, D. 1999. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol 77(6), pp. 1121-1134. 
8. Zhang, Y. et al. 2007. Perfectionism, academic burnout and engagement among Chinese college students: A structural equation modeling analysis. Personality and Individual Differences 43(6), pp. 1529-1540. doi: https://doi.org/10.1016/j.paid.2007.04.010 
9. Kawamura, K. Y. et al. 2001. Perfectionism, Anxiety, and Depression: Are the Relationships Independent? Cognitive Therapy and Research 25(3), pp. 291-301. doi: 10.1023/a:1010736529013 
10. Emke, A. R. et al. 2017. A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Multisource Feedback Through Paired Self- and Peer Evaluations. Teach Learn Med 29(4), pp. 402-410. doi: 10.1080/10401334.2017.1306446 
11. Tan, C. H. et al. 2016. Developing metacognition through effective feedback. Medical Teacher 38(9), pp. 959-959. doi: 10.1080/0142159X.2016.1209472 
12. Keister, D. M. et al. 2017. Teaching Resident Self-Assessment Through Triangulation of Faculty and Patient Feedback. Teaching and Learning in Medicine 29(1), pp. 25-30. doi: 10.1080/10401334.2016.1246249

An insight into UCL medical school student opinion on summative assessments, resources and feedback on OSCE and SBA examinations

Anjali Gondhalekar, Dan Ntuiabane, George Choa, Ozzie Eborime, Alison Sturrock
* Corresponding author: Anjali Gondhalekar, UCL Medical School, United Kingdom
a.gondhalekar@ucl.ac.uk 

Background 
Feedback has been shown to be one of the most important influences on achievement1. Despite learners feeling that they often don’t receive adequate feedback and educators believing that feedback that they provide is often ignored, both groups agree that that feedback is an important tool to drive learning 2. UCL medical school provides summative feedback to medical students after their OSCEs (Objective Structured Clinical Examinations) and SBAs (Single Best Answer) through a variety of mediums. However a recent survey noted that in every year group cohort, over 70% of students believed that they did not have equal access to resources as their peers. This has triggered an amnesty at UCL medical school encouraging a pooling and sharing of resources between students to re-establish equity. This project aimed to evaluate student perceptions on the amnesty, and the actions taken by the medical school to support their learning and provide greater equity amongst their peers. The project also focused on considering ways of improving the feedback provided in future examinations by UCL medical school.  

Summary of the work 
An assessment and feedback task and finish group was established which consisted of members of both the medical school faculty and student body. Face to face questionnaires were disseminated to medical students in years 4, 5 and 6 and students were invited to attend focus groups at the Royal Free Hospital, UCLH and Whittington Hospitals, London. This enabled us to garner student opinion on topics including the introduction of an amnesty, the student’s views about their access to learning resources and of the assessment process in general and also the feedback that they receive after their summative assessments.

Summary of the results 
410 students responded to our questions regarding feedback for examinations in years 4, 5 and 6. 88% of respondents found the written examiner feedback useful. Moreover, 81% of students appreciated the statistical sub-speciality breakdown of SBAs. 78% of students found histogram and ranking data for each station to be useful, even though some felt it didn’t always accurately correlate to written feedback. Students found the OSCE debrief lecture least helpful of the feedback methods with 55% of students finding it helpful. Never the less, overall 67% of respondents found there to be considerable improvements in feedback compared to previous years.

Discussion & Conclusions 
UCL medical school’s collaborative approach and introduction of examination feedback methods have shown considerable improvement, whilst some students remain unsatisfied. Having garnered the opinions of the students and their suggestions for improvements in feedback, UCL medical school continues to strive to meet student demands and bridge the feedback gap4.

Take Home messages 
This research indicates that a UCL medical school is taking a number of active steps to work in close collaboration with UCL medical students to both provide equity in learning resources available to students and also to tailor summative feedback to the needs of their students to better support their learning needs through forming an educational alliance 5,6.

References 
1. Boud D, Molloy E. Feedback in higher and professional education: understanding it and doing it well / edited by David Boud and Elizabeth Molloy. Abingdon, Oxon; London: Abingdon, Oxon : Routledge; 2013. 
2. Norcini J. The power of feedback. Medical Education. 2010;44(1):16-7 
3. Miller L. Giving effective feedback. GP. 2009:44 
4. Murdoch-Eaton D, Bowen L. Feedback mapping – The curricular cornerstone of an “educational alliance”. Medical Teacher. 2017;39(5):540-7. 
5. Bowen L, Marshall M, Murdoch-Eaton D. Medical Student Perceptions of Feedback and Feedback Behaviors Within the Context of the “Educational Alliance”. Academic Medicine. 2017;92(9):1303-12. 
6. Telio S, Ajjawi R, Regehr G. The “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Academic Medicine. 2015;90(5):609-14

© Copyright 2019 The European Board of Medical Assessors (EBMA) | Disclaimer