Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity

Christopher Destephano, Anita H. Chen, Michael G. Heckman, Nicolette T. Chimato, Paulami Guha, Mariana Espinal, Tri A. Dinh

Research output: Contribution to journalArticle

Abstract

Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ,.68; 95% confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95% confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.

Original languageEnglish (US)
Pages (from-to)1051-1059
Number of pages9
JournalJournal of Minimally Invasive Gynecology
Volume25
Issue number6
DOIs
StatePublished - Sep 1 2018

Fingerprint

Operating Rooms
Hysterectomy
Extremities
Gynecology
Confidence Intervals
Robotics
Advisory Committees
ROC Curve
Obstetrics
Prospective Studies

Keywords

  • Hysterectomy
  • Performance assessment
  • Surgical simulation

ASJC Scopus subject areas

  • Obstetrics and Gynecology

Cite this

Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity. / Destephano, Christopher; Chen, Anita H.; Heckman, Michael G.; Chimato, Nicolette T.; Guha, Paulami; Espinal, Mariana; Dinh, Tri A.

In: Journal of Minimally Invasive Gynecology, Vol. 25, No. 6, 01.09.2018, p. 1051-1059.

Research output: Contribution to journalArticle

Destephano, Christopher ; Chen, Anita H. ; Heckman, Michael G. ; Chimato, Nicolette T. ; Guha, Paulami ; Espinal, Mariana ; Dinh, Tri A. / Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity. In: Journal of Minimally Invasive Gynecology. 2018 ; Vol. 25, No. 6. pp. 1051-1059.
@article{f92f7d6310e5471a94993d530fce426e,
title = "Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity",
abstract = "Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4{\%} (weighted κ,.68; 95{\%} confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95{\%} confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1{\%}) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.",
keywords = "Hysterectomy, Performance assessment, Surgical simulation",
author = "Christopher Destephano and Chen, {Anita H.} and Heckman, {Michael G.} and Chimato, {Nicolette T.} and Paulami Guha and Mariana Espinal and Dinh, {Tri A.}",
year = "2018",
month = "9",
day = "1",
doi = "10.1016/j.jmig.2018.02.003",
language = "English (US)",
volume = "25",
pages = "1051--1059",
journal = "Journal of Minimally Invasive Gynecology",
issn = "1553-4650",
publisher = "Elsevier",
number = "6",

}

TY - JOUR

T1 - Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity

AU - Destephano, Christopher

AU - Chen, Anita H.

AU - Heckman, Michael G.

AU - Chimato, Nicolette T.

AU - Guha, Paulami

AU - Espinal, Mariana

AU - Dinh, Tri A.

PY - 2018/9/1

Y1 - 2018/9/1

N2 - Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ,.68; 95% confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95% confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.

AB - Study Objective: To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design: Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting: Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants: Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions: Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results: OR scores for TLH steps were significantly higher than simulation assessments (p <.001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r =.30, p =.14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r =.39–.58, p =.001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ,.68; 95% confidence interval,.45–.90), and intraclass correlation for the GOALS overall assessment was.71 (95% confidence interval,.46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r =.32–.42, p =.11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion: The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.

KW - Hysterectomy

KW - Performance assessment

KW - Surgical simulation

UR - http://www.scopus.com/inward/record.url?scp=85043477922&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85043477922&partnerID=8YFLogxK

U2 - 10.1016/j.jmig.2018.02.003

DO - 10.1016/j.jmig.2018.02.003

M3 - Article

VL - 25

SP - 1051

EP - 1059

JO - Journal of Minimally Invasive Gynecology

JF - Journal of Minimally Invasive Gynecology

SN - 1553-4650

IS - 6

ER -