Introduction
Small-incision cataract surgery (SICS) is a widely accepted, appropriate, and affordable procedure that can deliver high-quality visual outcomes.1, 2, 3, 4
Despite this need, concerns remain in several regions over the safety, quality, and efficiency of surgical training for cataract surgery.5, 6
Ophthalmic surgery involves very precise surgical skills which are difficult to teach. The surgical technique of a resident is also difficult to assess. Currently, teaching ophthalmic surgical skill is not structured. Every surgeon have their own way of imparting surgical training, which creates confusion amongst the residents, thus hampering their learning skills.
Hence there is need for education system to come up with some teaching and assessment tool for evaluation of surgical performance of the post graduate students.
It is also desirable for any assessment method to be objective, thus removing bias and favoritism.
For assessing performances in clinical settings Miller developed a theoretical framework known as Millers Pyramid clinical assessment where bottom levels begins as ‘Knows’, then ‘Knows How’, ‘Shows how’ and at top ‘Does’. The top level ‘Does’ has been considered as gold standard of assessment in surgical training measured by direct faculty observation. There may be some barriers for assessment of surgical trainings of trainees e.g. fear of being labelled for strict and harsh teacher, it may be challenging for supervisor to assess progress of underperforming trainees because of poor documentation, lack of opportunities to observe performance, or fear of legal action, hence the Master of surgery degree is conferred without assessing actual operative skill of surgeon which may put health of society at risk. Hence there is need for education system to come up with some assessment tool for evaluation of surgical performance of post graduate student.
In practice, ICO has developed and implemented various assessment tool to evaluate the different level of millers pyramid in form of task specific checklist called Ophthalmic surgery clinical assessment Rubric(OSCAR).1, 7
The ICO OSCAR for SICS was developed by experts at the ICO using a modified Dreyfus scale (novice, beginner, advanced beginner, and competent).8
Significance of the study:
Materials and Methods
This study was prospective observational study, carried out in Department of Ophthalmology of Medical College and Research Centre between January 2019 to February 2020 after institutional ethical clearance.
Seven final year Post Graduate (PG) residents were chosen for the study with convenience sample technique Small incision cataract surgery training assessment was done by Rubric designed by tool called ICO-Ophthalmic Surgery Competency Assessment Rubric. The same tool was used to assess video recorded cataract surgery of residents by 2 different faculties and the response was noted. Assessor independently assessed the surgical technique without any prior influence and knowledge about previous surgical assessment, PG resident’s surgical video were numbered as PG 1 to PG 7 to maintain anonymity and prevent information bias. The assessor simply circled the observed performance description at each step of the procedure. The ICO-OSCAR score was completed. Observations of each resident were recorded.
At the end of each case assessor immediately discussed case with student to provide timely, structured, specific performance feedback. Surgical performances were recorded and their assessment were done by different faculties with same tool. Post-operative patient’s record were reviewed for outcome of surgery and complications if any. Rubric Score were analysed with inter rater agreement. Post assessment feedback was collected from faculties as well as resident about the assessment technique. Patients were assessed in post-operative period for visual recovery and any other complications. The recorded Data was entered in Master Chart on Word Excel document.
Result
Research analysis requirement: statistical analysis (descriptive and inferential), graphical representation Specific statistical test applied: kappa statistics Statistical packages used: SPPS and graph pad.
Table 1
Oscar score of PG by different assessor as well as self-minimum and maximum
Table 2
Mean score of scorer 1 & 2 is almost same. There is not much standard of deviation of mean is noted
Figure 1
OSCAR Score of two assessor is almost same that shows there is no subjective variation in the assessment of students by assessor
Final Master Table: Table 2 Interclass correlation coefficient on chronbachs alpha analysis shows is 0.961 in 95% confidence interval it shows there was a very good agreement in all 3 raters.
Table 3
ntraclass correlation coefficient
Two-way random effects model where both people effects and measures effects are random.
Inference
There was a very good inter rater agreement (0.96). Cohen’s Kappa: 0.65 again shows good interrater agreement. (Table 2, Table 3)
From these statistics, we found ICO-OSCAR offers a reliable way to objectively assess the surgical skills acquired during residency training. OSCAR TOOL has very good inter rater agreement i.e. (0.96).
Descriptive feedback analysis of student & observer regarding this assessment tool & entire procedure of this assessment revealed:
Constructive feedback from assessor & recorded surgeries definitely helped PG students to improve surgically. Which ultimately resulted in best outcome for patient in terms of good visual acuity.
Both assessor as well as examine found this tool very easy to use and non-subjective, structured and above all can be used as learning tool as well as it has described the expected performance. With possibilities of error which novice can make and if person has an idea beforehand it is least likely he will repeat the possible errors. This is very important in learning phase.
Our study results showed Oscar score gradually increased in all PG student with every successive cases. There was no much difference in the score between the assessors.
Discussion
Continuous professional development relies on the link between performance and an educational process aimed at improving knowledge and skill, one of the most broadly used frameworks for assessing skills is Miller’s Pyramid. This Pyramid has a series of levels of achievement beginning with knowledge (at the base) and ending with routine application in the clinical setting.9 Intraoperative evaluation is the gold standard. Intraoperative assessment achieves highest level evaluate ‘DOES’ in Miller’s pyramid. In a study of the Objective Structured Assessment of Technical Skills, Goff et al. (2002) had two faculty members complete checklists to rate the performance of OB/GYN trainees; ratings increased with amount of training.10
While standardized patient examinations have been successfully implemented in medical education, certification, and licensure, they cannot be used to assess all aspects of competence.7, 11
In our study we have seen it has very good inter rater agreement similar findings were noted by Golnic KC et al.8, 12
In our studies we have taken descriptive feedback on this assessment method by students as well as assessor and we found out it is reliable tool, adding educational value, it is feasible and acceptable by all.
As it is time tested, skills sharpen with multiple attempts or practice. Similarly if multiple assessment with constructive & time specific feedback is given it will lead to better surgical performance. Hence multiple assessment improves abilities of trainee in the surgical performance. This type of assessment tool serves three distinct purposes.13, 14 It minimizes subjectivity by precise instruction, the rubric states what is expected of the trainee to attain competence at each level. Self-Assessment by reflective practice is possible when the trainee’s surgical procedures are recorded. This ICO-Oscar tool was easy to use for assessor as well as learner. ICO-Oscar Tool in our study has been precise and valid for PG student’s surgical assessment. This tool can be used by Training institutes as bench mark for trainers to decide competency and necessary certification for Small Incision Cataract Surgery proficiency.
