Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review

Benjamin Zendejas, Raaj K. Ruparel, David Allan Cook

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

Background: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. Methods: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). Results: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert–novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. Conclusion: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

Original languageEnglish (US)
Pages (from-to)512-520
Number of pages9
JournalSurgical Endoscopy and Other Interventional Techniques
Volume30
Issue number2
DOIs
StatePublished - Feb 1 2016

Fingerprint

Laparoscopy
Ligation
Medical Students
Reproducibility of Results
Physicians

Keywords

  • Assessment
  • Fundamentals of Laparoscopic Surgery
  • Simulation
  • Validation

ASJC Scopus subject areas

  • Surgery

Cite this

Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool : a systematic review. / Zendejas, Benjamin; Ruparel, Raaj K.; Cook, David Allan.

In: Surgical Endoscopy and Other Interventional Techniques, Vol. 30, No. 2, 01.02.2016, p. 512-520.

Research output: Contribution to journalArticle

@article{e2b34d9cd2d94f9cb61399e00946699a,
title = "Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review",
abstract = "Background: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. Methods: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). Results: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert–novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. Conclusion: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.",
keywords = "Assessment, Fundamentals of Laparoscopic Surgery, Simulation, Validation",
author = "Benjamin Zendejas and Ruparel, {Raaj K.} and Cook, {David Allan}",
year = "2016",
month = "2",
day = "1",
doi = "10.1007/s00464-015-4233-7",
language = "English (US)",
volume = "30",
pages = "512--520",
journal = "Surgical Endoscopy",
issn = "0930-2794",
publisher = "Springer New York",
number = "2",

}

TY - JOUR

T1 - Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool

T2 - a systematic review

AU - Zendejas, Benjamin

AU - Ruparel, Raaj K.

AU - Cook, David Allan

PY - 2016/2/1

Y1 - 2016/2/1

N2 - Background: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. Methods: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). Results: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert–novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. Conclusion: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

AB - Background: The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment. Methods: We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences). Results: From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert–novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability. Conclusion: Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

KW - Assessment

KW - Fundamentals of Laparoscopic Surgery

KW - Simulation

KW - Validation

UR - http://www.scopus.com/inward/record.url?scp=84957839290&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84957839290&partnerID=8YFLogxK

U2 - 10.1007/s00464-015-4233-7

DO - 10.1007/s00464-015-4233-7

M3 - Article

C2 - 26091982

AN - SCOPUS:84957839290

VL - 30

SP - 512

EP - 520

JO - Surgical Endoscopy

JF - Surgical Endoscopy

SN - 0930-2794

IS - 2

ER -