TY - JOUR
T1 - Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity
AU - DeStephano, Christopher C.
AU - Chen, Anita H.
AU - Heckman, Michael G.
AU - Chimato, Nicolette T.
AU - Guha, Paulami
AU - Espinal, Mariana
AU - Dinh, Tri A.
N1 - Publisher Copyright:
© 2018 American Association of Gynecologic Laparoscopists
PY - 2018/9/1
Y1 - 2018/9/1
N2 - Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ,.68; 95% confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95% confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.
AB - Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ,.68; 95% confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95% confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.
KW - Hysterectomy
KW - Performance assessment
KW - Surgical simulation
UR - http://www.scopus.com/inward/record.url?scp=85043477922&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85043477922&partnerID=8YFLogxK
U2 - 10.1016/j.jmig.2018.02.003
DO - 10.1016/j.jmig.2018.02.003
M3 - Article
C2 - 29432903
AN - SCOPUS:85043477922
SN - 1553-4650
VL - 25
SP - 1051
EP - 1059
JO - Journal of Minimally Invasive Gynecology
JF - Journal of Minimally Invasive Gynecology
IS - 6
ER -