HPO inter-rater agreement

How reproducible is HPO phenotyping?

Deep phenotyping based on the Human Phenotype Ontology (HPO) has become standard in human genetics. Recording clinical signs and symptoms is undoubtedly more accurate than just storing a diagnosis but no one has ever assessed how reproducible HPO-based phenotyping really is.
In our (albeit anecdotal) experience, different physicians rarely assign the same HPO terms to the same patient.
Variant- and gene-hunting software and software for differential diagnosis is currently trained with simulated imprecision (more unspecific parent terms instead of the term annotated to the gene) and "noise" (completely unrelated clinical signs), however without any evidence for the level of uncertainty.

Project description

With HPO-ira, we aim to close this gap. We want to study inter-rater agreement in HPO phenotyping on a scientific basis.
It is planned as a two-staged project:

1) Testing inter-rater agreement on synthetic data

We provide several fictional discharge letters / epicrises. Physicians and genetic counsellors are asked to assign the most suitable HPO terms to these 'patients'. This will reveal general problems with assigning HPO terms, uncertainties in medical reports, and show how much influence medical background and experience have on phenotyping.

2) Testing inter-rater agreement on real patients

In the second part of the project, we would like physicians and genetic counsellors to see the same patient on the same day and carefully phenotype them. In this part of the project, a potential bias from the epicrises is removed.

Expected results

Data from HPO-ira will provide insights into reproducibility and hence accuracy of HPO-based phenotyping. We expect to find out whether differences are only in the level of detail (e.g. Chronic Kidney Disease stage 2 vs. Chronic Kidney Disease) or much larger (e.g. conditions not even recorded by another persons).
This will allow novel variant-hunting software to be trained with more realistic noise and imprecision. It will also help improve the HPO - whenever there is a large discrepancy for certain terms, it may be the consequence of unprecise names and definitions of the terms. Results from HPO-ira can be used to change the location of a term within the tree-like structure of the HPO as well as its name, synonyms, and definition.

How can you help?

Please register here. We do not store any data that would reveal your identity, only:

job (e.g. physician or genetic counsellor)
medical profession
experience with the HPO
experience with rare diseases

How will you benefit?

Refining the HPO and gaining insights about the reproducibility of HPO phenotyping will obviously help you (and anyone else) to better assess the reliability of HPO phenotypes. Furthermore, it will provide the possibility to better train HPO-based software and hence lead to better tools for differential diagnosis or variant prioritisation.
Besides, every collaborator will be co-author on the resulting publication.

Contact

Jean-Tori Pantel	UK Aachen	jepantel	@ukaachen.de
Janina Schönberger	BIH@Charité	janina.schoenberger	@bih-charite.de
Dominik Seelow	BIH@Charité	dominik.seelow	@bih-charite.de