Agreement Between Upper Digestive Disease App and Provider Evaluation of Esophagectomy Symptoms in a Mobile App Tool

Background: The objective of this study was to assess the criterion validity of score thresholds for the Upper Digestive Disease (UDD) App. Methods: From December 15, 2017, to December 15, 2020, patients presenting after esophagectomy were offered the UDD App concurrent with a provider visit. This tool consists of 67 questions including 5 novel domains. Score thresholds were used to assign patients to a good, moderate, or poor category on the basis of domain scores. Providers were given performance descriptions for each domain and asked to assign patients to a category on the basis of their clinical evaluation. The weighted κ statistic was used to determine the magnitude of agreement between classifications based on the patients’ UDD App scores and the providers’ clinical evaluation. Results: Fifty-nine patients in the study (76% male; median age, 63 years [interquartile range, 57-72 years]) reported outcomes using the UDD App. Providers reviewed between 1 and 10 patients at a median time of 296.5 days (interquartile range, 50-975 days) after esophagectomy. The magnitude of agreement between patients and providers was moderate for dysphagia (κ = 0.52; P < .001) and reflux (κ = 0.42; P < .001). Dumping-related hypoglycemia (κ = 0.03; P = .148), gastrointestinal complaints (κ = 0.02; P = .256), and pain (κ = 0.05; P < .184) showed minimal agreement, with providers underestimating the symptoms and problems reported by patients in these domains. Conclusions: Although there was agreement between UDD App assessment and provider evaluation of dysphagia and reflux after esophagectomy, there was discordance of scoring for dumping-related symptoms and pain. Future research is needed to determine whether thresholds for pain and dumping domains need to be revised or whether additional provider education on performance descriptions is needed.