The Feasibility of the Use of Video Capture, Feedback Process in the Obstetrics and Gynecology Residents

The Feasibility of the use of Video Capture, Feedback Process in the Obstetrics and Gynecology Residents Sean Adams Martin A. Martino, MD; Joseph E. Patruno, MD; Timothy M. Pellini, MD Abstract Educating a resident and proving that he is capable of consistently performing a procedure is a difficult task. This is vastly important for the patients safety. One of the key parts of becoming a quality surgeon is technical skill. To assess the technical skills of the obstetrics and gynecology residents, myself and four other students performed a video capture, feedback process using a product called SimCapture. This process involved us video recording four different types of surgeries performed by residents. After each case, the resident and attending participated in a feedback process to review the strengths and weaknesses of the resident for that particular case. The feasibility of the process is very significant in helping determine if this method is effective and if it should be used in the future. Often times, we planned on recording a case for our data in which something went wrong such as a resident who didn’t operate at all or a change in the operating room (OR) schedule we were not made aware of. This caused us to miss quite a few cases that we planned on recording. Nurses, attendings, and residents were slightly skeptical about our study initially, but over time they became more comfortable with us recording the surgeries. The consistent issues and obstacles forced us to be very proactive and flexible. Ultimately, we were successful in collecting the research we wanted. Background Becoming an expert surgeon is a long and difficult process including many years of medical school and residency. This calls for thousands of hours of learning information, practicing on simulators, and performing surgeries with an attending surgeon. It is important that the attending surgeon mentors and assists the resident(1). The education of these residents must be maximized for the purpose of patient safety. While there are many factors in determining a surgeons abilities, technical skill is the most related to the outcome of the patient (2). Evaluating the technical skill of residents is an especially challenging subject. An educational program called FLS (The Fundamentals of Laproscopic Surgery) was developed to improve the knowledge and technical skills of residents (3). A study done by three Washington D.C hospitals compared the FLS examination scores of residents to their objective OR skill evaluations (3). The results showed a clear relationship as residents with higher FLS scores also received better evaluations in the OR (3). The FLS test includes a written section to express knowledge of laproscopic surgery as well as skills tests on box trainers (3). Another way to evaluate the technical skill of surgeons is through video analysis. A study done by the Michigan Bariatric Surgery Collaborative (MBSC) had 20 surgeons submit a video of themselves performing a laproscopic gastric bypass (2). The skills of these surgeons were then evaluated by blinded surgeons and rated on a 1 to 5 scale. These scores were then compared to the outcomes of surgeries done by the same 20 surgeons on around 10,000 patients (2). The results showed that the top quartile of scores were related to lower complication rates (2). The bottom quartile were related to higher complication rates (2). The use of videotape assessment is a very effective way for residency programs to improve the technical skill of residents through feedback. It is also a way to determine whether or not their residents are competent performing certain surgical procedures. If hospitals are considering a form of video taping and feedback process, it is important that they know the feasibility of the process. How accepting were the residents and attending surgeons to the process? Were participating in the debriefing after? Was the process overcrowding the OR? Was it difficult to record the cases? A type of video recording technology called SimCapture was purchased by the Lehigh Valley Hospital for the purpose of recording residents. The technology includes a software programmed into a laptop computer that then allows you to record surgeries from a webcam. The data recorded on the webcam and through connection to the endoscopic tower is then stored on the software for further analysis. Purpose The purpose of my project is to determine the feasibility of a video capture, feedback process that will be used to improve and calculate the technical skill of obstetrics and gynecology residents at the Lehigh Valley Hospital. The video capture, feedback process involves recording the surgeries performed by a number of different residents. The attending surgeons will provide feedback based on their real time analysis of the residents performance. The videotapes of each case will be sent out to blinded experts for analysis using the same evaluation forms that were completed by the attending surgeon and resident that were present at the case. In the future, the recorded surgeries could also be used to build a portfolio for each resident to show their competency performing certain procedures. My complete focus will be on determining the degree of how convenient a video capture, feedback process can be done. Methods For our data collection, we recorded the obstetrics and gynecology residents performing four different types of surgeries. These different surgeries included C section, Hysteroscopy, Robotic Hysterectomy, and Laparoscopic tubal ligation (BTL). Our exact methods for recording the surgeries varied slightly depending on the case. The residents varied in experience from first through fourth year. The fourth year residents would often do large portions of the cases while the less experienced residents did less. The night before the cases, we would look at the OR schedule and plan out the cases we were going to record the next day. A group of two scholars would show up about a half hour before the first surgery. At this time they would find the resident and attending for the case to introduce themselves, tell them about the project, and ask them to participate in a debrief session following the case. The scholars would then enter into the OR to set up the technology. The computer with the Sim Capture program is connected to the endoscopic tower using the appropriate adaptors. This allows us to get an internal view of the patient. A camera attached to an extension chord is mounted on top of an IV pole using a clamp and then angled toward the incision made on the patient. For C sections, there is no endoscopic tower used. For robotic hysterectomy cases, we record the resident’s hands when using the robot instead of the incisions. One scholar then logs into the SimCapture program and runs a new session. To run the session, they have to input information such as the resident number, name of scholar operating the system, and the type of surgery. Once everything is set up, both scholars leave the OR until the patient is all ready to be operated on. Upon re-entering the OR, the scholar working the laptop then starts recording once time out is called. Every time the resident stops or starts operating, an annotation is made in the video tape. The scholar who is not working the SimCapture program fills out the form which keeps track of each time the attending teaches the resident. That scholar also completes the feasibility form which notes if the resident, attending, and other staff were understanding of our study as well as any problems that occurred. Once the surgery is completed, recording is stopped and both scholars disassemble the equipment. After leaving the OR; both scholars, the resident, and the attending participate in a short debriefing session. In this debriefing session, the attending and resident each complete the appropriate OPRS and milestone forms. The resident then discusses what he thought he did well and also what could be improved. The attending follows up with his feedback. Once the debriefing is completed, the scholars then plug all the forms into the SimCapture program and discuss the case. Results Figure 1: Number of cases recorded for each procedure Procedure Number of cases recorded Hysteroscopy 11 Robotic Hysterectomy 8 C section 17 BTL 3 Total 39 Figure 2: Percentage of cases recorded Percentage of Cases Recorded Recorded Missed Total Percent Recorded 39 25 64 61% Figure 4: Summary of feasibility form Average time to set up equipment 5.7 min Average time to disassemble equipment 3.2 min Average time to debrief 3.6 min Percent of cases where debrief occurred 61% Percent of cases where resident was receptive to process 97% Percent of cases where attending was receptive to process 95% Percent of cases where OR staff was receptive to process 97% Conclusion / discussion After about six weeks of collecting data, we were able to record 39 cases as shown in figure 1. You can see in figure 2 that we did miss quite a few of the cases that we had planned on recording. Our 61% success rate had to do with the number of issues and obstacles that occurred throughout the process. Some of the obstacles included technology issues, resident not operating, changes in the OR schedule, surgeon preference, and patient preference. Figure 3 allows you to see the breakdown of how often certain issues occurred in the 25 cases we missed. As we became more comfortable with our video capture feedback process, the issues became less and less frequent. Some of the initial technology issues included delays in the camera feed, one camera not working at all, SimCaptu re not picking up the camera and tower feeds, and not having the correct adaptor to connect the program into the tower. We sorted out almost all of these issues in the first few weeks. Experience also allowed us to perform the process more efficiently, including setting up the equipment. Over time, we had developed a consistent system to record cases and stay out of the way of the staff. This allowed our process to gain acceptance by attendings, residents, and OR staff. After performing a number of cases, they became more comfortable with us recording the cases in the OR. Figure 4 shows a summary of the results for the feasibility forms we completed for each case. One of the other key difficulties of our process was getting the resident and attendings to participate in the debriefing after the case. This is shown by the fact that only 61% of the cases had debriefings. The attendings, residents, or both of them were often needed elsewhere and had little time after the cases.Throughout the 6 weeks, we were forced to make quite and few innovations and be flexible with our process. For example, we needed to purchase an adaptor that would allow us to connect the SimCapture program into the endoscopic tower. Often times, the nurses and OR staff got very nervous about all the wires that we were bringing into the OR. We often had to be innovative by attaching the camera to the IV pole to get a good angle and laying a mat over all the wires. Even with all these issues, we were able to record a very good amount of cases over the 6 week period. This shows that a video capture, feedback process using a system such as SimCapture can be very effective in evaluating the technical skills of residents. References Levy, B. (2012). Experience Counts. American College of Obstetricians and Gynecologists, 119(4), 693-694. Birkmeyer, J. (2013, October 10). Surgical Skill and Complication Rates after Bariatric Surgery. www.nejm.org. 3. Antosh, D. (2012, December 8). Blinded Assessment of Operative Performance After Fundamentals of Laparoscopic Surgery in Gynecology Training. www.jmig.org

The Eucational Value of the Use of a Simcapture-Feedback Process in the Training of Obstetrics and Gynecology Residents

The educational value of the use of a Simcapture-feedback process in the training of obstetrics and gynecology Residents Joseph Patruno, MD, Timothy Pellini, MD, Martin Martino, MD, Grace Bova Lehigh Valley Health Network, Allentown, Pennsylvania Abstract: Background: The basis of renowned training hospitals like LVHN is a well-defined and effectively applied method for optimal teaching, especially in the operating room. Unfortunately, many physicians don’t provide the immediate and detailed feedback most valuable to residents. Thus, video feedback and detailed surveys shown to surgeons upon procedure completion are tools used to potentially optimize the educational experience for Obstetrics and Gynecology residents in the operating room. Method: For each chosen OB/Gyn procedure the surgical field was recorded. After the surgery, detailed surveys were given to each physician to assess the resident’s performance on each particular part of the operation. A debrief was also stimulated to discuss the specific weaknesses and strengths that the resident displayed during the procedure. Later that day, an Educational Value Survey was sent to both physicians to obtain feedback on how educationally effective the video and debriefing process were. Simultaneously, the same survey was sent to a control resident and attending who had done a similar procedure without the video and survey aspect present. Results: Attending surgeons’ responses in the subject cases were significantly higher with respect to the timeliness and effectiveness of the debrief than the responses from the control cases. Generally, both attendings and residents found the surgeries with the SimCapture video and evaluation tools more educationally valuable than those without the stimulated feedback resources. Conclusions: SimCapture video feedback as well as detailed evaluation forms can enhance the teaching and learning for residents doing OB/Gyn procedures. Additionally, with this foundation in place attending surgeons are able to engage the resident in a structured debrief after a surgery. Introduction: In a teaching hospital, attending surgeons are expected to provide immediate and comprehensive feedback to residents in order to improve surgery techniques and patient care. Unfortunately this expected debrief does not always occur or often the feedback given isn’t direct & clear enough to be effective. However the SimCapture technology used for video feedback as well as a series of surveys filled out by both surgeons are possible resources that can be utilized in order to optimize the educational value of a procedure. According to the BID model for teaching in the operating room created by Dr. Nicole K. Roberts (2009) the learner must be “actively involved in creating learning objective & proposing how instruction might be extended to future practice” (p.300). This model is easy to use because it solely requires brief conversation between attending and resident while scrubbing, operating, & closing. By utilizing these events that occur at every operation, very little additional time is required from either surgeon. Furthermore, the educational value of written assessments regarding resident’s surgical performance has also been rendered valid and useful. Such assessments “encourage the faculty and the resident to focus on areas of demonstrated competence and those areas needing improvement” (Dougherty et all, 2013, p.333). Detailed evaluation forms allow for the surgeons to reflect on each step of the procedure. These forms are most useful when completed directly after the surgery, allowing for more comprehensive recollection of the operation. Likewise, the resident benefits more from the stimulated debrief soon after he finishes the procedure. The stimulated debrief in this project utilizes the “Operative Performance Rating System “ (OPRS). Using this form, both surgeons can assess the resident’s performance for each part of the procedure. Although these evaluations have not get been validated for Ob/Gyn procedures, they have proven to be especially effective in general surgery. Unlike more broad evaluations of technical skill, the OPRS offers a “controlled, systematic, observation/evaluation and appropriate sampling of performance across the entire performance domain” (Larson et al, 2005, p. 646). Thus, the OPRS allows evaluators to rate the residents on a likert scale for each step of the procedure as well as more general surgical flow and knowledge. Detailed feedback like this provides the residents with specific surgical skills and methods that need improvement. The objective of our study is to determine the educational value of SimCapture video feedback as well as post-procedure evaluation forms with respect to Obstetrical and Gynecological procedures. We anticipate that these tools will enhance both the learning and teaching process for the residents and attendings respectively. Materials and Methods: In order to evaluate whether the SimCapture video and survey forms combined made a positive effect on the educational value of an operation we filmed about 40 OR cases. Among these cases included Cesarean sections, Robotically-assisted Laparoscopic Hysterectomies, Hysteroscopies, Bilateral Tubal Ligations, and Laparoscopically-Assisted Vaginal Hysterectomies. After choosing several cases to record from the OR list, the team found both the attending and resident on the case and informed them on the SimCapture project and asked for permission to film their upcoming procedure in the OR. Upon consent, the team set up the video system and oriented the OR staff to the project. During the procedure, only the surgeon’s hands were recorded. The residents were all assigned a number; thus, a resident number was associated with each video instead of individual’s names. Annotations were made within the video to indicate when the resident is operating. After the procedure was complete, both attending and resident were asked to fill out both a milestone form and an OPRS evaluation. The OPRS having questions specific to each step of the particular type of case. Having completed the surveys, the resident was asked to reflect on what he thought his weaknesses and strengths were in that particular case. This ignited the debriefing conversation, often resulting in meaningful discussion between both surgeons regarding the resident’s technique and skill level. Later that day, a survey was sent to both the attending and resident to assess the educational significance of their procedure that day. Among these questions were ones about the questions asked, the effectiveness of the debrief, and the overall educational value of the case. Each questions required a response between 1-5, usually 1 relating to least educational and 5 being most educationally valuable. Concurrently, another set of the same educational value surveys was sent to a control attending and resident that had participated in a similar case that day. Thus, the control data would allow us to evaluate the overall effect of the SimCapture and feedback process on the educational process. Using these cases for controls allows us to determine how well residents are regularly being effectively taught, especially via a debrief after the procedure. The questions in this survey asked responders to answer the following questions on a likert scale 1 being “strongly disagree” and 5 being “strongly agree”. 1. I/the resident came adequately prepared 2. Optimal training provided? 3. The questions asked by the attending were fair & appropriately challenging 4. There was an appropriate debrief before, during & after the case? 5. My opinion of my performance correlates with the attending’s perception of my performance. 6. Specific, timely, & valuable feedback was given. 7. I would rate the overall educational value of this case as… (1-Poor and 5-Excellent) The responses were then compiled into four separate groups: control residents, subject residents, control attendings, and subject attendings. The mean response for each question for each group was calculated. Then the controls’ and subjects’ mean responses were compared to assess the overall educational effect of the video and feedback process. Statistical analysis was also done. A t-test was performed to obtain p-values for the data set. Using the results of the t-test, we were able to determine the statistical significance and validity of each question for the residents (controls and subjects) and attendings (controls and subjects). Results: By the end of our six-week collection period, forty total cases were recorded: seventeen Cesarean Sections, eight Robotic Hysterectomies, twelve Hysteroscopies, and three Bilateral Tubal Ligations. We also collected fifteen control cases: four Cesarean Sections, one Robotic Hysterectomy, eight Hysteroscopies, and two Bilateral Tubal Ligations. Twelve out of twenty-three OB/Gyn residents participated in the study: two interns, five in their second year of residency, three in their third, and two in their fourth. Twenty-two different attendings participated as well. A brief comparison of the controls and subjects was done with the mean responses to each question. These results are as follows. The more statistically significant differences will be further discussed. Refer to Table 1 in the Appendix to see the average values for each question. 1. I/the resident came adequately prepared Residents: Similar responses regardless of process Attendings: Similar responses regardless of process 2. Optimal training provided? Residents: Interestingly, the controls felt more optimally trained in the OR than the subject did Attendings: Similar responses regardless of process 3. The questions asked by the attending were fair & appropriately challenging Residents: Controls’ responses were higher; felt they had been asked more appropriate questions Attendings: Controls’ responses were slightly higher 4. There was an appropriate debrief before, during & after the case? Residents: Subjects’ responses were higher. Not surprisingly, as there was better communication during the debrief developed by the process Attendings: Subjects’ responses were generally much higher. The attendings were alert to the stimulated debriefing process 5. My opinion of my performance correlates with the attending’s perception of my performance. Residents: Similar responses regardless of process Attendings: Similar responses regardless of process 6. Specific, timely, & valuable feedback was given Residents: Similar responses; feel they are getting feedback in either process Attendings: Subjects’ responses were higher; attendings felt much more confident regarding their feedback with this process guiding them 7. I would rate the overall educational value of this case as… (1-Poor and 5-Excellent) Residents: Subjects’ responses were higher. The educational value ties together importance of preparation, supervision, education, debriefing, and feedback, several of which the SimCapture/feedback process aimed to enhance. Attendings: Subjects’ responses were higher as well. The following is a chart (Figure 1) of the average response for residents (both control and subject) and the p values from the t-test. The highlighted columns are the values that are most statistically significant. (Figure 1) The following is a chart (Figure 2) of the average response for attendings (both control and subject) and the p values from the t-test. The highlighted columns are the values that are most statistically significant. (Figure 2) Furthermore, in the subject’s surveys there was a question about the value of using SimCapture video as a form of feedback. 87% of subject cases had residents who saw value in using video as a method of feedback. 75% of the subject cases had attendings who felt the video feedback could be useful for optimal teaching and learning Discussion: The residents in the subject cases found the surgeries with the SimCapture and feedback process generally more educationally valuable. Many of the residents enjoyed the opportunity to receive direct and detailed feedback from their mentors. The residents also benefitted from reflecting on their own performance, considering both their strengths and weaknesses. However, the residents’ responses that were particularly statistically significant were about the questions asked during surgery; interestingly, the controls felt that the questions asked were more fair and appropriate than the subjects. This difference may be due to the presence of the SimCapture equipment and team. This addition to the OR may have made the attendings feel as though the resident was in the spotlight and, therefore, did not need as many questions. The attendings who participated in the SimCapture and feedback process were able to recognize the educational importance of the debrief after a procedure. The attendings in subject cases felt significantly more confident about the timeliness and value of the feedback they gave to the resident after the procedure. By stimulating a debrief session between both surgeons directly after the procedure, attendings were able to recall specific challenges and strengths the resident displayed in the surgery. Thus, they felt they were able to deliver complete and procedure-specific criticism to the residents. Additionally the attendings also became more aware of the debriefing process. Many attendings further discovered the value and effectiveness of giving immediate, detailed feedback. Some of the anonymous comments left on the educational value survey were, “I need to debrief more. We have to go after cases to meet the next patient and sometime don\u27t take the time that we should be”… “I may ask the resident more questions during the case to assess their understanding”. Thus, this process increased personal teaching awareness, hopefully motivating the attendings to provide more comprehensive and timely feedback to the residents after a procedure. Regarding the use of SimCapture video as feedback, most of the faculty and residents who participated in the cases felt the video technology could be an effective and useful form of feedback. Several surgeons requested to view their videos upon completion and were able to easily visualize their weaknesses and strengths with the objective point of view on the camera. One of the limits of this research was the lack of a large cohort of control cases. Due to our time constraint, we were unable to get the widespread responses we had hoped for both control and subject cases. Additionally many residents and attednings did not fill out the educational value surveys at all. Others completed them days after the surgery, which could possibly limit their ability to recall the specifics of the residents’ performances in the particular cases. Lastly, in some of the subject cases we were not able to stimulate a debrief due the surgeon’s schedules. In these cases both surgeons filled out the OPRS evaluations but were unable to discuss the resident’s performance in the procedure. The lack of this debrief may have affected our final response data. Conclusion: Overall the SimCapture video and the OPRS evaluations feedback method was educationally valuable. Attendings especially found the guided feedback process specifically helpful and effective. We expect that attendings will try to incorporate more timely and detailed feedback to their residents in their everyday practice. The detailed OPRS evaluations provided residents with specific areas of technique that need improvements. By enhancing the teaching and learning process in the OR, residents are able to learn more advanced skills faster and more efficiently. Attendings are able to assess the resident’s direct areas of weakness or strength with a simple feedback mechanism. Essentially, the SimCapture with feedback process can potentially advance the way residents are trained, thus improving patient care. Acknowledgements: Thank you to my project mentor Dr. Joseph Patruno, as well as Dr. Timothy Pellini, Dr. Martin Martino, Dr. Hubert Huang, Diane Leuthard, the library services, and all the residents, physicians, nurses, and faculty in the OB/GYN department. This project would not have been possible without your support. Finally, thank you to Dr. Joseph Napolitano and the Dorothy Rider Pool Health Care Trust for giving me this opportunity and incredible experience. References: Dougherty, P, Kasten, S.J., Reynolds, R.K., Prince, M.E., Lypson, M.L. (2013, June). Intraoperative Assessment of Residents. Jornal of Graduate Medicine, 333-334. http://dx.doi.org/10.4300/JGME-D-13-00074.1 Larson, J.L., Williams, R. G., Ketchum, J, Boehler, M.L., Dunnington, G.L. (2005, October). Feasibility, reliability, and validity of operative performance rating system for evaluating surgery residents.Surgery, 138 (4), 640-649. http://dx.doi.org/10.1016/j.surg.2005.07.017 Levy, B. (2012, April). Experience Counts. Obstetrics & Gynecology, 119(4), 693-694. Roberts, N.K., Williams, R.G., Kim, M.J., Dunnington, G.L. (2009, February). The Briefing, Intraoperative Teaching, Debriefing Model for Teaching in the Operating Room. Journal of the American College of Surgeons, 208(2), 299-303. http://dx.doi.org/10.1016/j/jamcollsurg.2008.10.024 Swing, S.R. (2007).The ACGME outcome project: retrospective and prospective. Medical Teacher, 29, 648-654. http://dx.doi.org/10.1080/0142159070139290