Skip to contents

fairmetrics: Fairness evaluation metrics with confidence intervals for binary protected attributes

CRAN_Status_Badge License Downloads total arXiv

A collection of functions for computing fairness metrics for machine learning and statistical models, including confidence intervals for each metric. The package supports the evaluation of group-level fairness criterion commonly used in fairness research, particularly in healthcare for binary protected attributes. It is based on the overview of fairness in machine learning written by Gao et al (2024) (https://arxiv.org/abs/2406.09307).

  • Link to online tutorial.
  • Link to preprint.

Installation

To install the latest CRAN release run:

install.packages("fairmetrics")

To install the package from the Github repository run:

devtools::install_github("jianhuig/fairmetrics")

Citation

To cite package ‘fairmetrics’ in publications use:

Gao J, Smith B, Chou B, Gronsbell J (2025). fairmetrics: Fairness Evaluation Metrics with Confidence Intervals. https://github.com/jianhuig/fairmetrics.

Smith, Gao, and Gronsbell (2025). fairmetrics: An R package for group fairness evaluation. arXiv:2506.06243.

Gao et al. (2024). What is Fair? Defining Fairness in Machine Learning for Health. arXiv:2406.09307.

A BibTeX entry for LaTeX users is

bibentry(
  bibtype = "Manual",
  title = "fairmetrics: Fairness Evaluation Metrics with Confidence Intervals for Binary Protected Attributes",
  author = c(
    person("Jianhui", "Gao"),
    person("Benjamin", "Smith"),
    person("Benson", "Chou"),
    person("Jessica", "Gronsbell")
  ),
  year = "2025",
  url = "https://github.com/jianhuig/fairmetrics"
)

bibentry(
    bibtype = "Misc",
    key = "Smith_Gao_Gronsbell_2025",
    title = "fairmetrics: An R package for group fairness evaluation",
    author = c(
      person("Benjamin", "Smith"),
      person("Jianhui", "Gao"),
      person("Jessica", "Gronsbell")
    ),
    year = "2025",
    month = "jun",
    note = "arXiv:2506.06243",
    url = "https://arxiv.org/abs/2506.06243",
    textVersion = "Smith, Gao, and Gronsbell (2025). fairmetrics: An R package for group fairness evaluation. arXiv:2506.06243."

  )

bibentry(
  bibtype = "Misc",
  key = "Gao_Chou_McCaw_Thurston_Varghese_Hong_Gronsbell_2024",
  title = "What is Fair? Defining Fairness in Machine Learning for Health",
  author = c(
    person("Jianhui", "Gao"),
    person("Benson", "Chou"),
    person("Zachary R.", "McCaw"),
    person("Hilary", "Thurston"),
    person("Paul", "Varghese"),
    person("Chuan", "Hong"),
    person("Jessica", "Gronsbell")
  ),
  year = "2024",
  month = "jun",
  note = "arXiv:2406.09307",
  url = "https://arxiv.org/abs/2406.09307",
  textVersion = "Gao et al. (2024). What is Fair? Defining Fairness in Machine Learning for Health. arXiv:2406.09307."
)

Similar Works

References

  1. Gao, J. et al. What is Fair? Defining Fairness in Machine Learning for Health. arXiv.org https://arxiv.org/abs/2406.09307 (2024).