Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos

Barbara André, Tom Vercauteren, Anna M. Buchner, Michael B. Wallace, Nicholas Ayache

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Evaluating content-based retrieval (CBR) is challenging because it requires an adequate ground-truth. When the available ground-truth is limited to textual metadata such as pathological classes, retrieval results can only be evaluated indirectly, for example in terms of classification performance. In this study we first present a tool to generate perceived similarity ground-truth that enables direct evaluation of endomicroscopic video retrieval. This tool uses a four-points Likert scale and collects subjective pairwise similarities perceived by multiple expert observers. We then evaluate against the generated ground-truth a previously developed dense bag-of-visual-words method for endomicroscopic video retrieval. Confirming the results of previous indirect evaluation based on classification, our direct evaluation shows that this method significantly outperforms several other state-of-the-art CBR methods. In a second step, we propose to improve the CBR method by learning an adjusted similarity metric from the perceived similarity ground-truth. By minimizing a margin-based cost function that differentiates similar and dissimilar video pairs, we learn a weight vector applied to the visual word signatures of videos. Using cross-validation, we demonstrate that the learned similarity distance is significantly better correlated with the perceived similarity than the original visual-word-based distance.

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages297-304
Number of pages8
Volume6893 LNCS
EditionPART 3
DOIs
StatePublished - 2011
Event14th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2011 - Toronto, ON, Canada
Duration: Sep 18 2011Sep 22 2011

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 3
Volume6893 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other14th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2011
CountryCanada
CityToronto, ON
Period9/18/119/22/11

Fingerprint

Content based retrieval
Distance Learning
Distance education
Retrieval
Content-based Retrieval
Evaluation
Video Retrieval
Metadata
Cost functions
Differentiate
Cross-validation
Margin
Cost Function
Similarity
Observer
Pairwise
Signature
Truth
Metric
Evaluate

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

André, B., Vercauteren, T., Buchner, A. M., Wallace, M. B., & Ayache, N. (2011). Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (PART 3 ed., Vol. 6893 LNCS, pp. 297-304). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6893 LNCS, No. PART 3). https://doi.org/10.1007/978-3-642-23626-6_37

Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos. / André, Barbara; Vercauteren, Tom; Buchner, Anna M.; Wallace, Michael B.; Ayache, Nicholas.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6893 LNCS PART 3. ed. 2011. p. 297-304 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6893 LNCS, No. PART 3).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

André, B, Vercauteren, T, Buchner, AM, Wallace, MB & Ayache, N 2011, Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 3 edn, vol. 6893 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 3, vol. 6893 LNCS, pp. 297-304, 14th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2011, Toronto, ON, Canada, 9/18/11. https://doi.org/10.1007/978-3-642-23626-6_37
André B, Vercauteren T, Buchner AM, Wallace MB, Ayache N. Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 3 ed. Vol. 6893 LNCS. 2011. p. 297-304. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3). https://doi.org/10.1007/978-3-642-23626-6_37
André, Barbara ; Vercauteren, Tom ; Buchner, Anna M. ; Wallace, Michael B. ; Ayache, Nicholas. / Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6893 LNCS PART 3. ed. 2011. pp. 297-304 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3).
@inproceedings{31ac60d3910347778908c6de7726b2c9,
title = "Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos",
abstract = "Evaluating content-based retrieval (CBR) is challenging because it requires an adequate ground-truth. When the available ground-truth is limited to textual metadata such as pathological classes, retrieval results can only be evaluated indirectly, for example in terms of classification performance. In this study we first present a tool to generate perceived similarity ground-truth that enables direct evaluation of endomicroscopic video retrieval. This tool uses a four-points Likert scale and collects subjective pairwise similarities perceived by multiple expert observers. We then evaluate against the generated ground-truth a previously developed dense bag-of-visual-words method for endomicroscopic video retrieval. Confirming the results of previous indirect evaluation based on classification, our direct evaluation shows that this method significantly outperforms several other state-of-the-art CBR methods. In a second step, we propose to improve the CBR method by learning an adjusted similarity metric from the perceived similarity ground-truth. By minimizing a margin-based cost function that differentiates similar and dissimilar video pairs, we learn a weight vector applied to the visual word signatures of videos. Using cross-validation, we demonstrate that the learned similarity distance is significantly better correlated with the perceived similarity than the original visual-word-based distance.",
author = "Barbara Andr{\'e} and Tom Vercauteren and Buchner, {Anna M.} and Wallace, {Michael B.} and Nicholas Ayache",
year = "2011",
doi = "10.1007/978-3-642-23626-6_37",
language = "English (US)",
isbn = "9783642236259",
volume = "6893 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
number = "PART 3",
pages = "297--304",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
edition = "PART 3",

}

TY - GEN

T1 - Retrieval evaluation and distance learning from perceived similarity between endomicroscopy videos

AU - André, Barbara

AU - Vercauteren, Tom

AU - Buchner, Anna M.

AU - Wallace, Michael B.

AU - Ayache, Nicholas

PY - 2011

Y1 - 2011

N2 - Evaluating content-based retrieval (CBR) is challenging because it requires an adequate ground-truth. When the available ground-truth is limited to textual metadata such as pathological classes, retrieval results can only be evaluated indirectly, for example in terms of classification performance. In this study we first present a tool to generate perceived similarity ground-truth that enables direct evaluation of endomicroscopic video retrieval. This tool uses a four-points Likert scale and collects subjective pairwise similarities perceived by multiple expert observers. We then evaluate against the generated ground-truth a previously developed dense bag-of-visual-words method for endomicroscopic video retrieval. Confirming the results of previous indirect evaluation based on classification, our direct evaluation shows that this method significantly outperforms several other state-of-the-art CBR methods. In a second step, we propose to improve the CBR method by learning an adjusted similarity metric from the perceived similarity ground-truth. By minimizing a margin-based cost function that differentiates similar and dissimilar video pairs, we learn a weight vector applied to the visual word signatures of videos. Using cross-validation, we demonstrate that the learned similarity distance is significantly better correlated with the perceived similarity than the original visual-word-based distance.

AB - Evaluating content-based retrieval (CBR) is challenging because it requires an adequate ground-truth. When the available ground-truth is limited to textual metadata such as pathological classes, retrieval results can only be evaluated indirectly, for example in terms of classification performance. In this study we first present a tool to generate perceived similarity ground-truth that enables direct evaluation of endomicroscopic video retrieval. This tool uses a four-points Likert scale and collects subjective pairwise similarities perceived by multiple expert observers. We then evaluate against the generated ground-truth a previously developed dense bag-of-visual-words method for endomicroscopic video retrieval. Confirming the results of previous indirect evaluation based on classification, our direct evaluation shows that this method significantly outperforms several other state-of-the-art CBR methods. In a second step, we propose to improve the CBR method by learning an adjusted similarity metric from the perceived similarity ground-truth. By minimizing a margin-based cost function that differentiates similar and dissimilar video pairs, we learn a weight vector applied to the visual word signatures of videos. Using cross-validation, we demonstrate that the learned similarity distance is significantly better correlated with the perceived similarity than the original visual-word-based distance.

UR - http://www.scopus.com/inward/record.url?scp=80053499673&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80053499673&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-23626-6_37

DO - 10.1007/978-3-642-23626-6_37

M3 - Conference contribution

SN - 9783642236259

VL - 6893 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 297

EP - 304

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -