ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

Bálint Cserni; Darren Kilmartin; Mark O’Loughlin; Xavier Andreu; Zsuzsanna Bagó-Horváth; Simonetta Bianchi; Ewa Chmielik; Paulo Figueiredo; Giuseppe Floris; Maria Pia Foschini; Anikó Kovács; Päivi Heikkilä; Janina Kulka; Anne Vibeke Laenkholm; Inta Liepniece-Karele; Caterina Marchiò; Elena Provenzano; Peter Regitnig; Angelika Reiner; Aleš Ryška; Anna Sapino; Elisabeth Specht Stovgaard; Cecily Quinn; Vasiliki Zolota; Mark Webber; Sharon A. Glynn; Rita Bori; Erika Csörgő; Orsolya Oláh-Németh; Tamás Pancsa; Anita Sejben; István Sejben; András Vörös; Tamás Zombori; Tibor Nyári; Grace Callagy; Gábor Cserni

doi:10.3390/cancers15041199

ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

Bálint Cserni, Darren Kilmartin, Mark O’Loughlin, Xavier Andreu, Zsuzsanna Bagó-Horváth, Simonetta Bianchi, Ewa Chmielik, Paulo Figueiredo, Giuseppe Floris, Maria Pia Foschini, Anikó Kovács, Päivi Heikkilä, Janina Kulka, Anne Vibeke Laenkholm, Inta Liepniece-Karele, Caterina Marchiò, Elena Provenzano, Peter Regitnig, Angelika Reiner, Aleš RyškaAnna Sapino, Elisabeth Specht Stovgaard, Cecily Quinn, Vasiliki Zolota, Mark Webber, Sharon A. Glynn, Rita Bori, Erika Csörgő, Orsolya Oláh-Németh, Tamás Pancsa, Anita Sejben, István Sejben, András Vörös, Tamás Zombori, Tibor Nyári, Grace Callagy, Gábor Cserni^*

^*Corresponding author for this work

Pathology

5 Citations (Scopus)

Abstract

Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2–11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.

Original language	English
Article number	1199
Journal	Cancers
Volume	15
Issue number	4
ISSN	2072-6694
DOIs	https://doi.org/10.3390/cancers15041199
Publication status	Published - Feb 2023

Keywords

breast cancer
European Working Group for Breast Screening Pathology
international immuno-oncology biomarker working group
observers needed to evaluate subjective tests
ONEST
reproducibility
sTILs
TILs
triple-negative
tumour-infiltrating lymphocytes

Access to Document

10.3390/cancers15041199Licence: CC BY

Cite this

Cserni, B., Kilmartin, D., O’Loughlin, M., Andreu, X., Bagó-Horváth, Z., Bianchi, S., Chmielik, E., Figueiredo, P., Floris, G., Foschini, M. P., Kovács, A., Heikkilä, P., Kulka, J., Laenkholm, A. V., Liepniece-Karele, I., Marchiò, C., Provenzano, E., Regitnig, P., Reiner, A., ... Cserni, G. (2023). ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations. Cancers, 15(4), Article 1199. https://doi.org/10.3390/cancers15041199

Cserni, B, Kilmartin, D, O’Loughlin, M, Andreu, X, Bagó-Horváth, Z, Bianchi, S, Chmielik, E, Figueiredo, P, Floris, G, Foschini, MP, Kovács, A, Heikkilä, P, Kulka, J, Laenkholm, AV, Liepniece-Karele, I, Marchiò, C, Provenzano, E, Regitnig, P, Reiner, A, Ryška, A, Sapino, A, Stovgaard, ES, Quinn, C, Zolota, V, Webber, M, Glynn, SA, Bori, R, Csörgő, E, Oláh-Németh, O, Pancsa, T, Sejben, A, Sejben, I, Vörös, A, Zombori, T, Nyári, T, Callagy, G & Cserni, G 2023, 'ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations', Cancers, vol. 15, no. 4, 1199. https://doi.org/10.3390/cancers15041199

@article{760c985c6f8d493a89059eb44b5b85d5,

title = "ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations",

abstract = "Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2–11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.",

keywords = "breast cancer, European Working Group for Breast Screening Pathology, international immuno-oncology biomarker working group, observers needed to evaluate subjective tests, ONEST, reproducibility, sTILs, TILs, triple-negative, tumour-infiltrating lymphocytes",

author = "B{\'a}lint Cserni and Darren Kilmartin and Mark O{\textquoteright}Loughlin and Xavier Andreu and Zsuzsanna Bag{\'o}-Horv{\'a}th and Simonetta Bianchi and Ewa Chmielik and Paulo Figueiredo and Giuseppe Floris and Foschini, {Maria Pia} and Anik{\'o} Kov{\'a}cs and P{\"a}ivi Heikkil{\"a} and Janina Kulka and Laenkholm, {Anne Vibeke} and Inta Liepniece-Karele and Caterina Marchi{\`o} and Elena Provenzano and Peter Regitnig and Angelika Reiner and Ale{\v s} Ry{\v s}ka and Anna Sapino and Stovgaard, {Elisabeth Specht} and Cecily Quinn and Vasiliki Zolota and Mark Webber and Glynn, {Sharon A.} and Rita Bori and Erika Cs{\"o}rg{\H o} and Orsolya Ol{\'a}h-N{\'e}meth and Tam{\'a}s Pancsa and Anita Sejben and Istv{\'a}n Sejben and Andr{\'a}s V{\"o}r{\"o}s and Tam{\'a}s Zombori and Tibor Ny{\'a}ri and Grace Callagy and G{\'a}bor Cserni",

note = "Publisher Copyright: {\textcopyright} 2023 by the authors.",

year = "2023",

month = feb,

doi = "10.3390/cancers15041199",

language = "English",

volume = "15",

journal = "Cancers",

issn = "2072-6694",

publisher = "M D P I AG",

number = "4",

}

TY - JOUR

T1 - ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

AU - Cserni, Bálint

AU - Kilmartin, Darren

AU - O’Loughlin, Mark

AU - Andreu, Xavier

AU - Bagó-Horváth, Zsuzsanna

AU - Bianchi, Simonetta

AU - Chmielik, Ewa

AU - Figueiredo, Paulo

AU - Floris, Giuseppe

AU - Foschini, Maria Pia

AU - Kovács, Anikó

AU - Heikkilä, Päivi

AU - Kulka, Janina

AU - Laenkholm, Anne Vibeke

AU - Liepniece-Karele, Inta

AU - Marchiò, Caterina

AU - Provenzano, Elena

AU - Regitnig, Peter

AU - Reiner, Angelika

AU - Ryška, Aleš

AU - Sapino, Anna

AU - Stovgaard, Elisabeth Specht

AU - Quinn, Cecily

AU - Zolota, Vasiliki

AU - Webber, Mark

AU - Glynn, Sharon A.

AU - Bori, Rita

AU - Csörgő, Erika

AU - Oláh-Németh, Orsolya

AU - Pancsa, Tamás

AU - Sejben, Anita

AU - Sejben, István

AU - Vörös, András

AU - Zombori, Tamás

AU - Nyári, Tibor

AU - Callagy, Grace

AU - Cserni, Gábor

PY - 2023/2

Y1 - 2023/2

N2 - Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2–11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.

AB - Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2–11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.

KW - breast cancer

KW - European Working Group for Breast Screening Pathology

KW - international immuno-oncology biomarker working group

KW - observers needed to evaluate subjective tests

KW - ONEST

KW - reproducibility

KW - sTILs

KW - TILs

KW - triple-negative

KW - tumour-infiltrating lymphocytes

UR - http://www.scopus.com/inward/record.url?scp=85149152852&partnerID=8YFLogxK

U2 - 10.3390/cancers15041199

DO - 10.3390/cancers15041199

M3 - Journal article

C2 - 36831541

AN - SCOPUS:85149152852

SN - 2072-6694

VL - 15

JO - Cancers

JF - Cancers

IS - 4

M1 - 1199

ER -

ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this