Face and content validity of a virtual-reality simulator for myringotomy with tube placement
- Caiwen Huang1,
- Horace Cheng2,
- Yves Bureau4, 5,
- Sumit K. Agrawal†1, 2, 6Email author and
- Hanif M. Ladak†1, 2, 3, 5Email author
© Huang et al. 2015
Received: 18 May 2015
Accepted: 12 October 2015
Published: 20 October 2015
Myringotomy with tube insertion can be challenging for junior Otolaryngology residents as it is one of the first microscopic procedures they encounter. The Western myringotomy simulator was developed to allow trainees to practice microscope positioning, myringotomy, and tube placement. This virtual-reality simulator is viewed in stereoscopic 3D, and a haptic device is used to manipulate the digital ear model and surgical tools.
To assess the face and content validity of the Western myringotomy simulator.
The myringotomy simulator was integrated with new modules to allow speculum placement, manipulation of an operative microscope, and insertion of the ventilation tube through a deformable tympanic membrane. A questionnaire was developed in consultation with instructing surgeons. Fourteen face validity questions focused on the anatomy of the ear, simulation of the operative microscope, appearance and movement of the surgical instruments, deformation and cutting of the eardrum, and myringotomy tube insertion. Six content validity questions focused on training potential on surgical tasks such as speculum placement, microscope positioning, tool navigation, ear anatomy, myringotomy creation and tube insertion. A total of 12 participants from the Department of Otolaryngology—Head and Neck Surgery were recruited for the study. Prior to completing the questionnaire, participants were oriented to the simulator and given unlimited time to practice until they were comfortable with all of its aspects.
Responses to 12 of the 14 questions on face validity were predominantly positive. One issue of concern was with contact modeling related to tube insertion into the eardrum, and the second was with the movement of the blade and forceps. The former could be resolved by using a higher resolution digital model for the eardrum to improve contact localization. The latter could be resolved by using a higher fidelity haptic device. With regard to content validity, 64 % of the responses were positive, 21 % were neutral, and 15 % were negative.
The Western myringotomy simulator appears to have sufficient face and content validity. Further development with automated metrics and skills transference testing is planned.
Myringotomy with tube insertion is one of the most common procedures in Otolaryngology—Head & Neck Surgery, and is encountered by residents throughout their training. Despite the fact that it is a ubiquitous procedure, the instruction of junior trainees, who often have little experience in microscopic procedures, is often challenging. Montague et al.  have analyzed surgical errors through video analysis of actual procedures and note that the 4 most frequently occurring errors in order from most to least occurring include (1) failure to perform a unidirectional myringotomy, (2) making multiple attempts to place the tube, (3) making multiple attempts to complete the myringotomy, and (4) setting the microscope magnification too high. More serious intraoperative complications can also occur including external auditory canal lacerations, medial displacement of tubes into the middle ear, and vascular injuries [2–4]. Although surgical residents can eventually perform standard cases well, they often struggle with narrow canals, retracted tympanic membranes, T-tubes, and procedures performed under local anaesthestic. The goal of simulation is to decrease the learning curve prior to entering the operating, minimize complications in patients, and provide the ability to practice difficult cases.
Several physical models have been described in the literature to provide practice without potential harm to patients [5–9]. Generally, these consist of a tube to mimic the ear canal with a synthetic membrane attached to one end to represent the eardrum. These models do not appear to have gained general acceptance in residency programs, presumably because they are not able to represent anatomical variability easily and the mechanical properties of the materials used do not mimic that of the actual tissues.
Compared with physical models, simulators based on virtual-reality (VR) technologies have the ability to simulate difficult anatomy, model various pathologies, provide automated feedback, and even allow trainees to practice on patient-specific models generated from CT/MRI scans. VR-based simulators have been applied in Otolaryngology, especially for endoscopic sinus surgery [10–14] and for temporal bone drilling [15–18].
In VR simulators, the trainee interacts with realistic 3D digital models of anatomical structures and views them using 3D displays. Simulated tissues can be operated upon using digital representations of actual surgical tools that can be moved in the workspace using devices such as a haptic arm. The sensation of contact force between a digital surgical tool and simulated tissue can be computed and applied to the trainee’s hand via the haptic arm.
The Auditory Biophysics Laboratory at Western University has developed and reported on several aspects of VR-based myringotomy simulation. A blade navigation software system [19, 20] and a system for real-time deformation and cutting of the tympanic membrane  were implemented on different software platforms as separate training modules. These versions of the simulator were not integrated and they did not include speculum placement, operating microscope controls for positioning/zooming, or tube insertion through the myringotomy.
As recently reported , the Western myringotomy simulator has integrated the previous modules into a common software platform. Moreover, new software modules have been added to allow the user to adjust their surgical view through positioning and tilting of the virtual speculum and operative microscope, and to allow insertion of a ventilation tube into the myringotomy created in a deformable tympanic membrane. The goal is to further expand this simulator in the future to allow trainees to raise tympanomeatal flaps andto eventually perform tympanoplasty/ossiculoplasty on patient-specific anatomy.
In order for training simulators to be accepted into a residency curriculum, a variety of validation studies need to be conducted starting with face validity and culminating in the demonstration that skills acquired in the VR environment transfer to the OR (operating room) environment. Face validity refers to the degree to which a simulation appears like the real situation  and content validity measures whether the simulator would be appropriate or useful in training [24, 25]. Although face validity has previously been established for individual software modules [19–21], validation testing has not been performed on the current integrated system, which simulates the entire procedure from microscope positioning to ventilation tube insertion .
The objective of this paper is to determine the face and content validity of the new integrated Western myringotomy simulator.
The haptic arm can be used to position and rotate the virtual speculum, position and tilt the microscope, and adjust magnification to obtain different views of the operative site as shown in Fig. 2b. The user can then create a myringotomy as shown in Fig. 2c using a virtual myringotomy blade; the position and orientation of the blade are controlled by moving the handle of the haptic arm. A tube may be inserted using virtual forceps, which is also controlled by the user using the haptic device [Fig. 2d]. The opening and closing of the forceps can be toggled using a button on the haptic arm. During tube insertion, the eardrum deforms and the incision splays as the tube enters the myringotomy. The tube may also be repositioned with various instruments until it is in its final position [Fig. 2e].
Research ethics board approval was obtained from Western University (#105239) and participants were contacted via telephone or electronic mail. All participants were recruited from the Department of Otolaryngology - Head & Neck Surgery, Western University. A total of 12 subjects agreed to participate, which included seven junior Otolaryngology residents (postgraduate years 1 to 3) and five senior Otolaryngologists who routinely performed ventilation tube insertions in their practice. These groups were chosen to reflect the target group of the simulator (junior residents) as well as experts in the field (Otolaryngologists). The participants did not have any previous exposure to myringotomy simulation.
Tasks involved in the face validity study
Rotate and tilt the speculum to obtain view of tympanic membrane
Translate and rotate the microscope to obtain a proper view
Navigate surgical blade through the external auditory canal
Make an incision in the tympanic membrane
Ventilation tube insertion
Insert ventilation tube into the myringotomy using forceps
Questions in Section A for face validity
Question: Rate whether the following aspects of the simulator are realistic
Visual appearance of the auricle and ear canal
Visual appearance of the speculum
Movement of the speculum
Movement of the microscope/camera
Zoom of the microscope/camera
Visual appearance of the eardrum
Movement of the eardrum when physically contacted
Visual appearance of the myringotomy blade
Visual appearance and splay of the myringotomy
Visual appearance of the forceps
Movement and stability of the myringotomy blade and forceps
Visual representation of the tube
Movement of the tube within the myringotomy
Three-dimensional microscopic view of the scene based on light rendering, shadows, and 3D goggles
Questions in Section B for training potential
Question: Do you feel that the simulator would be useful in teaching Otolaryngology trainees the following skills
Ear canal and eardrum anatomy
In Sections A and B, study participants were asked to answer each question using a 7-point Likert scale, an equal appearing interval measurement. The scale had values of “1”—Strongly Disagree, 2—“Mostly Disagree”, 3—“Disagree”, 4—“Neither Agree/Disagree”, 5—“Agree”, 6—“Mostly Agree” and 7—“Strongly Agree”.
In Section C, a free-form comment area was provided for each participant to provide feedback to elaborate on previous questions and to address issues not covered in Sections A and B.
The responses were initially divided by group (junior resident or practising Otolaryngologist), and the median, quartiles, minimum, and maximum response values were computed for each question. The sample size was maximized to include all eligible participants at a single academic institution. For each question, the Mann–Whitney U-test was used to test the significance of the differences in responses between the two groups. A frequency distribution histogram was plotted to investigate the number of favourable responses (score ≥ 5), neutral responses (score = 4), and negative responses (score ≤ 3) to each question. All data were computed and analysed using the SPSS statistical software (SPSS Inc, Chicago, IL). The significance was set at p ˂ .05 and the Holm-Bonferroni method was used to correct for multiple comparisons.
The first group was comprised of seven junior Otolaryngology residents in postgraduate years 1 to 3. They were all familiar with the operating microscope and the procedure, however they were in the active phase of learning with each resident having performed fewer than 20 myringotomy and tube insertions in training. The second group had five fellowship trained Otolaryngologists who routinely performed myringotomy and tube insertions in their practice. Each member of this group had performed at least 200 procedures since completing their fellowship.
Comparison of groups
Face and content validity
Given that mean responses were not different at the p = .05 level, the results for the two groups were pooled when analyzing face and content validity. The responses to the questionnaires were categorized as positive (score ≥ 5), neutral (score = 4) or negative (score ≤ 3).
The training potential of the simulator was tested through 6 questions in Section B of the questionnaire. As shown in Fig. 4, the number of positive responses was greater than the number of negative responses for each question in this section. Among the total 72 responses (6 questions x 12 participants), 46 (63.9 %) were positive, 15 (20.8 %) were neutral, and 11 (15.3 %) were negative.
The MS3 scale used in this study had to be developed at our institution as no other validated measure was available to assess a virtual-reality myringotomy simulator. This questionnaire has not been externally validated by other centres, however content validity was assessed by a group of experts during the development of the questionnaire. In addition, previous publications [20, 21] did demonstrate reliability of the MS3 with a strong correlation across raters. The MS3 was also correlated against a visual analogue scale measuring the same construct, thus providing a measure of concurrent validity .
The lack of statistically significant differences in mean responses between residents and senior Otolaryngologists to Questions 1 to 20 suggests that even with limited exposure to the actual procedure of myringotomy with tube insertion, junior residents had similar assessments of the realism and utility of the simulator as those experienced in the OR.
The only differences between the groups approaching significance were in Questions 13 and 20, which pertained the movement of the tube within the myringotomy. Senior Otolaryngologists perceived the simulated tube movement to be less realistic than did residents. Similarly, Question 9 in the pooled responses dealt with the splay of the myringotomy, and this had a higher number of negative responses overall. From the written comments in Section C of the questionnaire, it appears that splaying (i.e., spreading) of the virtual eardrum when it is contacted by the virtual blade is realistic, and this was also the case in our previous report ; however, splaying is less realistic during tube insertion when the virtual tube contacts the eardrum and causes it to spread.
Second, the physics of tympanic membrane ‘tearing’ with large forces and displacements during tube insertion are difficult to model in real-time. To overcome this, pre-programmed animations were used based on the length of incision, the trajectory of the tube, and the contact between the flange of the tube and the myringotomy. Although this significantly reduced computation time, Question 13 revealed that this lack of realism was noted by the experts and not the residents. This could be explained by the fact that senior surgeons would have had much more experience knowing how the ventilation tube should slide into the incision, therefore they were able to notice the subtle differences more than the junior trainees still learning the procedure. On average, Otolaryngologists’ rankings fell between “Disagree” to “Neither Agree/Disagree”, suggesting that slight improvements to the tube insertion simulation could make this aspect more acceptable.
Question 11 was the only other question with a higher proportion of negative responses, and this pertained to the movement and stability of the blade and forceps. Section C clarified this finding as concerns were raised about the limited range of motion of the haptic device and that the friction of the device affected the movements of the virtual blade and forceps. The haptic arm used in this study is a low-cost device that is suitable for design of a prototypical simulator. The device can easily be swapped for a higher fidelity device with greater range of motion and substantially reduced friction (e.g., Geomagic Phantom Premium device from Geomagic, Inc., Morrisville, NC), albeit at greater financial cost. Utilizing the higher fidelity device may result in acceptable range of motion and unnoticeable friction. A second concern with the device was the feel of the handle of the haptic arm when it was used to control the blade and forceps (Fig. 1). As the handle is thick, it feels unnatural compared to holding an actual surgical tool. We have implemented approaches described in the literature to replace the haptic arm handle with actual surgical tools to improve the feel and realism of the simulation . The goal in this hybrid simulator would be have one haptic arm attached to a myringotomy blade or forceps, and have the second haptic arm attached to a real speculum to maximize realism.
Face and content validity are only initial steps in validation, and they do not ensure that a simulator will be useful in training residents [24, 25]. Future development on the Western myringotomy simulator will address concerns raised in this study. Refinement and optimization of the tube insertion and tympanic membrane splay may help to increase the realism of the simulator, but it is unclear if increased fidelity will actually result in additional skills transference . In order to determine the construct validity of the simulator, automated metrics including time, length and direction of incision, collisions, magnification, etc. have been incorporated into the simulator. A separate study will examine if these metrics are capable of distinguishing experts from residents, and a skills transference study will be needed to determine if the simulator can result in better operating room performance. A multi-centred study will be considered at that time to maximize sample size and feedback from different centres.
The authors hope that by using standardized libraries while programming the simulator, and the ability of the simulator to run on low-cost hardware, will allow easy adoption by Otolaryngology training programs and allow other groups to make modifications as needed.
The Western myringotomy simulator has a number of new features including microscope handling, speculum positioning and ventilation tube insertion. The simulator has good face and content validity, except with respect to splaying of the myringotomy during tube insertion and with respect to the haptic arm. These issues are currently being addressed with further refinements and adaptations. Automated metrics have been developed and they will be used to assess for construct validity of the simulator. Although the entire myringotomy and ventilation tube insertion can now be simulated, a skills transference study is needed to establish training efficacy and clinical impact.
The authors would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC), Medtronic of Canada Ltd., and the Ontario Research Fund (ORF) for financial support of this project.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Montague M, Lee MSW, Hussain SSM. Human error identification: an analysis of myringotomy and ventilation tube insertion. Arch Otolaryngol Head Neck Surg. 2004;130:1153–7.View ArticlePubMedGoogle Scholar
- Brodish BN, Woolley AL. Major vascular injuries in children undergoing myringotomy for tube placement. Am J Otolaryngol. 1999;20:46–50.View ArticlePubMedGoogle Scholar
- Kumar M, Khan AM, Davis S. Medial displacement of grommets: an unwanted sequel of grommet insertion. J Laryngol Otol. 2000;114:448–9.PubMedGoogle Scholar
- Groblewski JC, Harley EH. Medial migration of tympanostomy tubes: an overlooked complication. Int J Pediatr Otorhinolaryngol. 2006;70:1707–14.View ArticlePubMedGoogle Scholar
- Walker T, Duvvi S, Kumar BN. The wigan grommet trainer. Clin Otolaryngol. 2006;31:349–50.View ArticlePubMedGoogle Scholar
- Duijvestein M, Borgstein J. The bradford grommet trainer. Clin Otolaryngol. 2006;31:163.View ArticlePubMedGoogle Scholar
- Leong A, Kundu S, Martinez-Devesa P, Aldren C. Artificial ear: a training tool for grommet insertion and manual dexterity. ORL. 2006;68:115–7.View ArticlePubMedGoogle Scholar
- Hong P, Webb AN, Corsten G, Balderston J, Haworth R, Ritchie K, et al. An anatomically sound surgical simulation model for myringotomy and tympanostomy tube insertion. Int J Pediatr Otorhinolaryngol. 2008;78:522–9.View ArticleGoogle Scholar
- Volsky PG, Hughley BB, Peirce SM, Kesser BW. Construct validity of a simulator for myringotomy with ventilation tube insertion. Otolaryngology - Head and Neck Surg. 2009;141:603–8.View ArticleGoogle Scholar
- Weghorst S, Airola C, Oppenheimer P, Edmond CV, Patience T, Heskamp D, et al. Validation of the madigan ESS simulator. Stud Health Technol Informat. 1998;50:399–405.Google Scholar
- Anil SM, Kato Y, Hayakawa M, Yoshida K, Nagahisha S, Kanno T. Virtual 3-dimensional preoperative planning with the Dextroscope for excision of a 4th ventricular ependymoma. Minim Invasive Neurosurg. 2007;50:65–70.View ArticlePubMedGoogle Scholar
- Audette M, Delingette H, Fuchs A, Astley O, Chinzei K. A topologically faithful, tissue-guided, spatially varying meshing strategy for computing patient-specific head models for endoscopic pituitary surgery simulation. Stud Health Technol Inform. 2006;119:22–7.PubMedGoogle Scholar
- Tolsdorff B, Pommert A, Höhne KH, Petersik A, Pflesser B, Tiede U, et al. Virtual reality: a new paranasal sinus surgery simulator. Laryngoscope. 2010;120:420–6.PubMedGoogle Scholar
- Varshney R, Frenkiel S, Nguyen LHP, Young M, Del Maestro R, Zeitouni A, et al. The McGill simulator for endoscopic sinus surgery (MSESS): a validation study. J Otolaryngol Head Neck Surg. 2014;43:40.PubMed CentralView ArticlePubMedGoogle Scholar
- Wiet GJ, Stredney D, Kerwin T, Hittle B, Fernandez SA, Abdel-Rasoul M, et al. Virtual temporal bone dissection system: development and testing. Laryngoscope. 2012;122 Suppl 1:S1–S12.PubMed CentralView ArticlePubMedGoogle Scholar
- Morris D, Sewell C, Barbagli F, Salisbury K, Blevins NH, Girod S. Visuohaptic simulation of bone surgery for training and evaluation. IEEE Comput Graph Appl. 2006;26:48–57.View ArticlePubMedGoogle Scholar
- Sewell C, Morris D, Blevins NH, Dutta S, Agrawal S, Barbagli F, et al. Providing metrics and performance feedback in a surgical simulator. Comput Aided Surg. 2008;13:63–81.View ArticlePubMedGoogle Scholar
- Arora A, Khemani S, Tolley N, Singh A, Budge J, Varela DA, et al. Face and content validation of a virtual reality temporal bone simulator. Otolaryngol Head Neck Surg. 2012;146:497–503.View ArticlePubMedGoogle Scholar
- Wheeler B, Doyle PC, Chandarana S, Agrawal S, Husein M, Ladak HM. Interactive computer-based simulator for training in blade navigation and targeting in myringotomy. Comput Meth Programs Biomed. 2010;98:130–9.View ArticleGoogle Scholar
- Sowerby LJ, Rehal G, Husein M, Doyle PC, Agrawal S, Ladak HM. Development and face validity testing of a three-dimensional myringotomy simulator with haptic feedback. J Otolaryngol Head Neck Surg. 2010;39:122–9.PubMedGoogle Scholar
- Ho AK, Alsaffar H, Doyle PC, Ladak HM, Agrawal SK. Virtual reality myringotomy simulation with real-time deformation: development and validity testing. Laryngoscope. 2012;122:1844–51.View ArticlePubMedGoogle Scholar
- Huang C, Agrawal SK, Ladak HM. Virtual-reality simulator for training in myringotomy with tube placement. BC: Vancouver; 2014. Proceedings of the 37th Canadian medical and biological engineering conference: 20–23 May 2014.Google Scholar
- Carter FJ, Schijven MP, Aggarwal R, Grantcharov T, Francis NK, Hanna GB, et al. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc. 2005;19:1523–32.View ArticlePubMedGoogle Scholar
- Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability rigorous science for the assessment of surgical education and training. Surg Endosc. 2003;17:1525–9.View ArticlePubMedGoogle Scholar
- Schout BM, Hendrikx AJ, Scheele F, Bemelmans BL, Scherpbier AJ. Validation and implementation of surgical simulators: a critical review of present, past, and future. Surg Endosc. 2010;24:536–46.PubMed CentralView ArticlePubMedGoogle Scholar
- Coles TR, John NW, Sofia G, Gould DA, Caldwell DG. Modification of commercial force feedback hardware for needle insertion simulation. Stud Health Technol Inform. 2011;163:135–7. MMVR18 – Medicine Meets Virtual Reality 2011 Poster: 8–12 February 2011; Newport Beach, CA.PubMedGoogle Scholar
- Hamstra SJ, Brydges R, Hatala R, Zendejas B, Cook DA. Reconsidering fidelity in simulation-based training. Acad Med. 2014;89:387–92.View ArticlePubMedGoogle Scholar