Discrimination and calibration performances of non-laboratory-based and laboratory-based cardiovascular risk predictions: a systematic review
•,,,,.
...
Abstract
Background and objective This review compares non-laboratory-based and laboratory-based cardiovascular disease (CVD) risk prediction equations in populations targeted for primary prevention.
Design Systematic review.
Methods We searched five databases until 12 March 2024 and used prediction study risk of bias assessment tool to assess bias. Data on hazard ratios (HRs), discrimination (paired c-statistics) and calibration were extracted. Differences in c-statistics and HRs were analysed. Protocol: PROSPERO (CRD42021291936).
Results Nine studies (1 238 562 participants, 46 cohorts) identified six unique CVD risk equations. Laboratory predictors (eg, cholesterol and diabetes) had strong HRs, while body mass index in non-laboratory models showed limited effect. Median c-statistics were 0.74 for both models (IQR: lab 0.77–0.72; non-lab 0.76–0.70), with a median absolute difference of 0.01. Calibration measures between laboratory-based and non-laboratory-based equations were similar, although non-calibrated equations often overestimated risk.
Conclusion The discrimination and calibration measures between laboratory-based and non-laboratory-based models show minimal differences, demonstrating the insensitivity of c-statistics and calibration metrics to the inclusion of additional predictors. However, in most reviewed studies, the HRs for these additional predictors were substantial, significantly altering predicted risk, particularly for individuals with higher or lower levels of these predictors compared with the average.
Introduction
Cardiovascular disease (CVD) is the major cause of mortality and morbidity globally.1 More than half a billion people globally were affected by CVD in 2021, resulting in 20.5 million deaths, representing nearly a third of all deaths worldwide.2 3 The majority of CVD deaths occur in low- and middle-income countries (LMICs).4 The mortality trends in high-income countries demonstrate that deaths attributed to CVD are mostly preventable.5
CVD risk prediction equations, which account for various risk factors of CVD, are frequently used in primary care settings to identify individuals who have a higher risk of developing CVD and who would likely benefit from preventive measures.6 7
While numerous CVD risk prediction equations have been developed and used for estimating CVD risk and guiding treatment strategies,8–10 their application in LMICs is limited due to the high cost of blood lipid-level measurements, which many equations rely on as inputs.11 There are also CVD risk equations that use alternative non-laboratory measures, such as body mass index (BMI).12 13 Although developed for use in LMIC settings, most non-laboratory CVD risk equations have been developed using data from non-LMIC populations.14 15
While previous studies have compared non-laboratory and laboratory equations in various settings,16–18 a comprehensive review comparing measures of discrimination and calibration between laboratory-based and non-laboratory-based risk equations, as well as evaluating the effect of hazard ratios (HRs) for additional predictors in predicting CVD risk, was lacking. There was a need for a systematic comparison between laboratory-based and non-laboratory-based equations, focusing on discrimination, calibration measures and the HRs of additional predictors, which is important for assessing the relative predictive performances of the competing CVD risk equations across diverse populations to ensure their generalisability.19 20
This review aims to assess and compare the performance of laboratory and non-laboratory CVD risk equations in populations different from those in which they were originally developed. The review has relevance to LMICs, aiming to identify effective risk assessment and intervention strategies with limited facilities, thus contributing to global efforts to reduce the burden of CVD.
Methods
Scope of review
This review addressed three key questions: (1) to compare the HRs of additional predictors in CVD risk prediction; (2) to identify externally validated non-laboratory-based and laboratory-based CVD equations from the same cohort and outline the reported model performance measures (discrimination and calibration) and (3) to analyse overall differences in model performance measures between laboratory-based and non-laboratory equations.
Search strategy and selection criteria
This systematic review protocol was registered under PROSPERO (CRD42021291936). Our reporting adheres to the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines21 and the Meta-Analysis of Observational Studies in Epidemiology22 (online supplemental appendix B). We systematically searched for studies published in five databases: PubMed, Scopus, Web of Science, ProQuest Dissertations and Theses Global and Google Scholar. The search began with the start of the review on 4 April 2023 and continued until 12 March 2024. We used combinations of search terms related to “laboratory-based”, “non-laboratory-based” and “cardiovascular risk scores” (detailed search terms and strategies provided in online supplemental appendix A). Searches were deliberately broad to encompass all relevant studies. Additionally, reference lists of included studies were screened to find further relevant studies. The included studies were published between 2002 and 2021.
Inclusion and exclusion criteria
Articles were included in this study if they met the following criteria: (a) compared laboratory-based and non-laboratory-based CVD risk prediction equations within the same study population; (b) were undertaken in a study population that differed to the population that the risk equation was derived in (ie, external validation study); (c) equations that were not recalibrated for the target population and (d) papers published in the English language. We focused on externally validated equations since external validation is essential for assessing the reproducibility and generalisability of a prediction model in diverse sets of new populations.23 We only examined equations externally validated in populations where recalibration was not undertaken prior to validation. Recalibration aims to align the risks predicted by the equation with the risks observed in the target population. Recalibration approaches would, therefore, mask any differences in calibration between laboratory and non-laboratory equations if included in the analysis.24 25 Articles were excluded if: (a) the study included people with existing CVD or a history of CVD at baseline; (b) non-English literature; (c) conference abstracts and (d) case reports.
Screening, quality assessment and data extraction
Titles and abstracts were independently reviewed by two authors (YMA and SMA). Full-text reviews were independently undertaken by the same authors. Research articles that fulfilled the inclusion criteria were evaluated for quality assessment by two authors (YMA and SMA). Disagreements on article selection and quality assessment were resolved through discussion. Two authors (YMA and SMA) extracted data using a predefined data extraction form. We extracted study-level data on age, sex, year of data collection, number of study participants, types of CVD risk equations and outcome measures (HRs, paired c-statistics, CIs, calibration χ2, calibration plots, calibration slopes (CSs), observed and expected ratio). A preliminary version of the Cochrane Prediction Study Risk of Bias Assessment Tool for cohort studies was used to assess the level of bias by evaluating selected parameters, including participant selection, predictors, outcome, sample size, participant flow and analysis26 27(online supplemental appendix D).
Data synthesis and analysis
This systematic review compared CVD risk equations using laboratory-based models with cholesterol and non-laboratory-based models using variables, such as BMI, synthesising adjusted HRs and regression coefficients from multivariable models. For example, the WHO-2019 and Ueda Globorisk equations extracted HRs for risk factors, such as age, systolic blood pressure, diabetes, smoking, total cholesterol and BMI. Similarly, the D’Agostino Framingham and Persian Atherosclerotic CVD equations incorporated cholesterol and diabetes in laboratory-based models, and BMI, waist–hip ratio (WHR) or diabetes history in non-laboratory models. The D’Agostino Framingham equations used log-transformed continuous variables to improve discrimination, calibration and minimise the effects of extreme observations. The estimated regression coefficients for the D’Agostino Framingham and Persian Atherosclerotic CVD equations were presented alongside the HRs. We included HRs because they directly quantify the impact of each predictor on individual CVD risk, providing clinically relevant insights beyond c-statistics. HRs are particularly useful for comparing laboratory-based and non-laboratory-based models, as large differences in HRs can significantly influence risk predictions, thereby enhancing model sensitivity and clinical utility.28 29 Calibration measures are also less sensitive to changes in predictor inclusion and may be less effective in capturing the influence of additional predictors.24 Discrimination, a measure of how well a risk prediction equation distinguishes between those with and without the disease, was assessed by extracting data on c-statistics from the included studies. C-statistics typically range from 0.5 (random concordance) to 1 (perfect concordance).30–32 As a standard, c-statistics<0.70 indicate inadequate discrimination, between 0.70 and 0.80 are considered acceptable and between 0.80 and 0.90 are considered excellent.6 33
We computed c-statistics differences between laboratory-based and non-laboratory-based equations by subtracting the non-laboratory-based c-statistics from the laboratory-based, which were compared within the same population. Forest plots were used to present the c-statistics and c-statistics differences. We calculated the absolute differences for each pair of c-statistics values; then, we computed the median absolute difference in c-statistics across all studies. Differences in c-statistics (changes in c-statistics) are classified into four categories. Large is used when the difference is 0.1 or greater; moderate for 0.05 to 0.1; small for 0.025 to 0.05 and very small for less than 0.025. Because c-statistics range from 0.5 to 1.0, the 0.1 cut-off point for large was set because it represents 20% of the possible range.34
We compared calibration between laboratory and non-laboratory CVD risk equations using four calibration measures (where available in the publications). First, we examined two χ2 metrics: the Hosmer–Lemeshow χ2 and Greenwood Nam-D’Agostino statistics, considering a significance level of p value<0.05 or a χ2 statistic exceeding 20 as indicative of a significant lack (poor calibration).35 36 Second, we examined how the population’s CVD risk was divided into risk deciles and plotted the predicted event rates against the observed.16 Third, we evaluated the ratio of expected to observed outcomes, or their probabilities, with a ratio close to 1 indicating effective model calibration.37 Finally, we considered a model’s CS, where a slope below 1 suggests overfitting, while slopes above 1 suggest underfitting. A slope near 1 indicates good calibration in the validation dataset.38 All analyses were performed using R (version 4.3.0).
Results
Overall, nine studies met the inclusion criteria,11 16 17 39–43 with 1 238 562 study participants, from 46 cohorts included (online supplemental appendix C).11 16 18 39 41 44–65 The cohorts consisted of 5 LMICs, along with 19 upper-middle-income and high-income countries, making a total of 24 countries analysed (online supplemental table 1). The median years of enrollment of cohorts used for external validation of the included studies ranged from 1961 to 2008 (a study may use more than one cohort for external validation). Studies excluded from the full-text stage are detailed in online supplemental appendix E. In general, the studies included were found to have a minimal risk of bias.
Table 1
•
HRs and performance of the WHO-2019 and ueda globorisk CVD risk equations
For included studies, six non-laboratory-based risk equations along with their corresponding laboratory-based risk equations were reported: WHO-2019; D’Agostino Framingham; Ueda Globorisk extension; European Prospective Investigation into Cancer and Nutrition (EPIC); INTERHEART and Persian Atherosclerotic CVD Risk Stratification (PARS). The laboratory measurements included in the risk equations varied. WHO-2019 and the Ueda Globorisk extension laboratory equations included both total cholesterol and diabetes as laboratory predictors,11 18 while D'Agostino Framingham, PARS and EPIC used total cholesterol, HDL cholesterol and diabetes,15 16 66 and the INTERHEART equation used apolipoproteins.67 (online supplemental table 2) provides a summary of the variables included in each equation, the study populations and predicted outcome events. Non-laboratory-based versions of WHO-2019, the Ueda Globorisk extension and D’ Agostino Framingham used BMI as a substitute for laboratory measures,11 15 18 while EPIC, SPARS and INTERHEART used WHR and/or non-clinical factors, such as diet.16 65 67
Table 2
•
HR and performance of D’Agostino Framingham and Persian atherosclerotic CVD risk equations
From the included studies, we extracted data on 64 HRs, 30 paired c-statistics (30 laboratory-based and 30 non-laboratory-based),11 16 17 39–43 5 paired calibration χ2,41 43 65 22 paired calibration plots,11 16 17 40 41 43 65 7 paired CSs17 and 1 paired observed–expected ratio43 (online supplemental table 3).
Comparison of HRs and performance between non-laboratory-based and laboratory-based CVD risk equations
For most risk equations examined, HRs for laboratory-based measures (eg, diabetes and cholesterol) were higher than those for non-laboratory-based measures, such as BMI. It is important to note that these HRs are unstandardised, meaning the variables (eg, binary for diabetes, continuous for cholesterol in mg/dL and BMI in kg/m²) are measured in their original units, they are not expressed on a common scale. For example, in the WHO-2019 CVD risk study using data from the emerging risk factors collaboration in women, the HRs for diabetes (HR 2.91, 95% CI 2.59 to 3.27) and total cholesterol (HR 1.23, 95% CI 1.20 to 1.26) in the laboratory-based model were higher than the HR for BMI (HR 1.14, 95% CI 1.10 to 1.18), which was included in the non-laboratory-based models. Both models included smoking, age and systolic blood pressure. The laboratory-based model had a c-statistics of 0.757 (95% CI 0.749 to 0.765), compared with 0.738 (95% CI 0.730 to 0.746) for the non-laboratory-based model.
In the Ueda Globorisk extension risk equation, the laboratory-based model included diabetes (HR 1.88, 95% CI 1.71 to 2.06) and total cholesterol (HR 1.19, 95% CI 1.16 to 1.22), while the non-laboratory-based model included BMI (HR 1.14, 95% CI 1.11 to 1.17). The c-statistics for the laboratory-based model was 0.71 (95% CI 0.70 to 0.72) compared with 0.69 (95% CI 0.68 to 0.70) for the non-laboratory-based model (table 1). In the Framingham cohort dataset in women, the laboratory-based model included total cholesterol (HR 3.35, 95% CI 2.00 to 5.62) and HDL cholesterol (HR 0.49, 95% CI 0.35 to 0.69), while the non-laboratory-based model substituted BMI (HR 1.67, 95% CI 0.98 to 2.85). The c-statistics for the laboratory-based model was 0.793 (95% CI 0.772 to 0.814), and for the non-laboratory-based model, 0.785 (95% CI 0.764 to 0.806). In the ICS cohort, the PARS laboratory-based model included the additional predictor of total cholesterol>300 mg/dL (HR 1.73, 95% CI 1.17 to 2.56), with c-statistics of approximately 0.73 for both the non-laboratory-based model (95% CI 0.71 to 0.74) and the laboratory-based model (95% CI 0.71 to 0.75) (table 2).
Comparison of discrimination measures
Overall, most CVD risk equations showed good discrimination, with 26 of the 30 c-statistics pairs being>0.7. The median external validation c-statistics in the laboratory-based equations was 0.74 (IQR, 0.77–0.72), and the median external validation c-statistics in the non-laboratory-based was also 0.74 (IQR, 0.76–0.70) online supplemental figure 1. There was little difference in c-statistics between laboratory-based and non-laboratory-based. The median absolute difference in the c-statistics between laboratory-based and non-laboratory-based was 0.01 (IQR, 0.01–0.00) (online supplemental figure 2). Within individual studies, 26 out of the 30 c-statistics differences were very small (differences in c-statistics<0.025) and 4 c-statistics differences were considered small (differences in c-statistics were 0.025–0.05); 3 of which were observed in the INTERHEART equation and 1 in the Globorisk equation.
Comparison of calibration measures between laboratory-based and non-laboratory-based equations
Overall, calibration measures of the externally validated risk equations suggested both the laboratory-based and non-laboratory-based D’Agostino Framingham risk prediction equations,68 overestimating the observed risk in Australia,39 Germany16 and the UAE population40 and the Atherosclerosis Risk in Communities from four distinct US provinces: Forsyth County, North Carolina; Jackson, Mississippi; suburbs of Minneapolis, Minnesota and Washington County, Maryland. Both the laboratory-based and non-laboratory-based EPIC-Potsdam equations marginally overestimated risk in the highest decile of predicted risk. The observed and expected ratios for the EPIC-Potsdam equation were O: E=1.05 (95% CI 0.97 to 1.13) for the non-laboratory-based and O: E=1.11 (95% CI 1.03 to 1.20) for the laboratory-based in the Heidelberg, German populations, respectively.16
According to the INTERHEART equation CSs, in South America, the equation showed a degree of overfitting in the non-laboratory-based version, with a CS of 0.87 (95% CI 0.77 to 0.98). Similarly, in China, both the non-laboratory-based and laboratory-based equations showed overfitting, with CS of 0.81 (0.71–0.91) and 0.88 (0.78–0.98), respectively. The non-laboratory-based equation showed overfitting in Africa with a CS of 0.75 (0.36–1.15). Overfitting patterns persist in South Asia and North America/Europe, with CS values of 0.75 (0.65–0.86) and 0.77 (0.68–0.87), respectively, for the non-laboratory-based equation. Conversely, the INTERHEART laboratory-based equation underfits in Middle Eastern populations, with a CS of 1.41 (1.18–1.63).17
External validations of both non-laboratory-based and laboratory-based PARS/SPARS equations showed an overestimation of observed event rates in the Iranian population (χ2 p value of less than 0.001).42 Similarly, external validations of both non-laboratory-based and laboratory-based WHO-2019 equations identified an overestimation of CVD risk in Chinese populations (χ2 p value of less than 0.001).45 None of the included studies assessed the calibration of the Globorisk extension equation11 in an external validation study.
Discussion
This study is the first systematic review that summarises and compares the HRs, discrimination and calibration performance of non-laboratory-based and laboratory-based CVD risk prediction equations; the evaluated equations were externally validated in primary prevention populations. In most CVD equations, the inclusion of predictors, such as cholesterol and diabetes, demonstrated stronger HRs than BMI. Discrimination performance was similar between the laboratory and non-laboratory among the six CVD risk equations reviewed, with absolute differences in c-statistics of less than 0.05. While c-statistics differences may overlook correlations between exposures and outcomes in competing models, net reclassification improvement (NRI) addresses this; however, most studies in the review did not use NRI, emphasising the need for its inclusion in future model evaluations.69 The majority of the CVD risk equations, both laboratory-based and non-laboratory-based, showed discrimination>0.7. However, few studies were well calibrated to external validation datasets or populations.
In the systematic review, laboratory-based predictors, such as total cholesterol and diabetes, demonstrated stronger HRs, indicating better risk stratification, whereas non-laboratory measures, such as BMI, had weaker HRs and a more limited influence on risk stratification. In most risk equations, laboratory-based models show similar c-statistics or slight improvements over non-laboratory-based models. Adding new variables may not significantly affect c-statistics, which often remain similar despite added predictors. However, incorporating predictors, such as diabetes and cholesterol, enhances risk stratification by yielding significantly higher HRs, thereby improving the identification of individuals at higher CVD risk and supporting clinical decision-making.70–73
Similar to our findings, previous studies have identified that most CVD equations in use had a c-statistics greater than 0.7 discrimination performance in external validation studies.68 74 75 In our review, 26 out of 30 pairs of discrimination measures had a c-statistics of 0.70 or higher.
In this review, many of the CVD risk equations showed poor calibration during external validation, emphasising the need for recalibration. Models with poor calibration may either underestimate or overestimate the outcome of interest.76 Factors, such as limited derivation dataset diversity, changes in population characteristics and shifts in underlying CVD risk, contribute to lower calibration of CVD risk equations.77 However, in our review, we found no evidence that calibration differed between laboratory-based and non-laboratory-based CVD risk equations when assessed in the same population.
Recalibration is a statistical adjustment aligning predicted and observed risk in the target population.78 Recalibration is necessary when applying CVD risk equations to populations with different underlying risk levels.9 35 79 Recalibration methods, such as intercept and slope adjustment, piecewise recalibration and transformation of predictors, can be employed to ensure that predicted risks closely match observed outcomes in predictive models, thereby enhancing the accuracy and reliability of CVD risk assessment.80 81
C-statistics are relatively insensitive to the inclusion of new predictors in a model, even when those predictors are statistically and clinically significant.71 Evaluating calibration in risk equations is challenging, as a slope of 1 suggests good calibration, but poor calibration can still occur if the intercept is overlooked, emphasising the need for a comprehensive assessment that includes both slope and intercept, along with other calibration metrics.82 The Hosmer–Lemeshow test assesses calibration by comparing observed and expected event rates in subgroups, but it has limitations, such as arbitrary groupings and reduced power in small datasets.83 Calibration plots compare predicted and observed risks, but smoothing techniques and arbitrary groupings (eg, by deciles) can affect accuracy, especially in small datasets. These plots may also overlook important or subtle differences across risk groups, so using them with other metrics is recommended for a more comprehensive model evaluation.24 83 84 Like other calibration measures, the observed-to-expected outcome ratio has a limitation, as it only assesses the average agreement between predicted and observed risks, without considering variations across different risk levels.85 86
This review identified five LMICs, including Bangladesh, India, Iran, Pakistan and Zimbabwe, out of a total of 24 countries analysed.11 17 18 42 Given that the majority of studies were conducted in high-income countries, there is a need for further research in LMICs comparing laboratory and non-laboratory equations. Most CVD risk equations globally have been derived or validated using cohorts from high-income settings. However, prospective cohort studies in LMICs are scarce, particularly where non-laboratory-based CVD risk equations are needed due to limited access to laboratory facilities, such as lipid testing. This review focuses on externally validated CVD risk equations to assess model reproducibility and generalisability. Future research that compares the predictive performance of non-laboratory risk equations with laboratory-based models that were not externally validated is warranted.16 39–41
Strengths and limitations
This is the first review of the discrimination and calibration performance of paired non-laboratory-based and laboratory-based CVD risk prediction equations, reflecting both comprehensiveness and broad scope, while also incorporating the HRs of additional predictors evaluated in competing models to further enhance comparison. This review evaluated six externally validated CVD risk equations in primary prevention populations across 24 countries, providing a comparative analysis that offers valuable insights into the performance of various prediction models. The inclusion of studies from diverse global populations enhances the applicability and generalisability of the findings. Since c-statistics are influenced by the composition of the study population (eg, age distribution), we only extracted c-statistics from studies where laboratory and non-laboratory CVD risk equations were compared within the same population.71 This study focuses on CVD equations that have been externally validated, thereby providing limited insight into the comparison of laboratory and non-laboratory CVD risk equations that have not been externally validated. Many CVD risk equations are recalibrated before application in new populations to align predicted risks with observed outcomes. While this recalibration improves model performance, it may obscure differences in the original calibration of the equations. Therefore, our review focused on non-recalibrated equations to evaluate their baseline predictive abilities between laboratory-based and non-laboratory-based models. However, in practice, many CVD risk equations are recalibrated for the specific population they are intended for, which may result in better calibration than what was observed in this study.77 Most of the studies in this review were conducted in high-income countries, while non-laboratory CVD risk equations are most applicable in LMICs, emphasising the urgent need for prospective cohort studies in LMICs to assess their CVD risk profiles and inform the derivation and external validation of context-specific equations.
Conclusion
The discrimination and calibration measures between laboratory-based and non-laboratory-based models show minimal differences, demonstrating the insensitivity of c-statistics and calibration metrics to the inclusion of additional predictors. However, in most reviewed studies, the HRs for these additional predictors were substantial, significantly altering predicted risk, particularly for individuals with higher or lower levels of these predictors compared with the average.
Contributors: YMA conceptualised the study, developed the protocol, conducted the systematic review, performed the data curation and formal analysis and drafted the original manuscript. SMA contributed to the conceptualisation, method and data curation, assisted in formal analysis and reviewed and edited the manuscript. NB, KW and DC provided supervision, refined the method and offered critical guidance throughout the study. NB, KW and DC also critically reviewed and edited the manuscript for important intellectual content. All authors critically reviewed and approved the final manuscript for submission. YMA is the guarantor.
Funding: KW is funded by Australian National Health and Medical Research Council Investigator Grants (2008697).
Competing interests: None declared.
Patient and public involvement statement: Patients and public were not involved in the design of this study.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
Data are available on reasonable request. Extracted data used in the analysis are available from the corresponding author on reasonable request.
Ethics statements
Patient consent for publication:
Not applicable.
Ethics approval:
This study is a systematic review, which typically analyses and synthesises existing research studies rather than collecting new data from participants, and ethical approval is not applicable.
Acknowledgements
The Australia National University Research Scholarship (International) supports YMA.
Vos T, Lim SS, Abbafati C, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. The Lancet2020; 396:1204–22.
Vaduganathan M, Mensah GA, Turco JV, et al. The Global Burden of Cardiovascular Diseases and Risk: A Compass for Future Health. J Am Coll Cardiol2022; 80:2361–71.
Ueda P, Woodward M, Lu Y, et al. Laboratory-based and office-based risk scores and charts to predict 10-year risk of cardiovascular disease in 182 countries: a pooled analysis of prospective cohorts and health surveys. Lancet Diabetes Endocrinol2017; 5:196–213.
Rezaei F, Seif M, Gandomkar A, et al. Comparison of laboratory-based and non-laboratory-based WHO cardiovascular disease risk charts: a population-based study. J Transl Med2022; 20.
Gaziano TA, Pandya A, Steyn K, et al. Comparative assessment of absolute cardiovascular disease risk characterization from non-laboratory-based risk assessment in South African populations. BMC Med2013; 11.
Gaziano TA, Young CR, Fitzmaurice G, et al. Laboratory-based versus non-laboratory-based method for assessment of cardiovascular disease risk: the NHANES I Follow-up Study cohort. The Lancet2008; 371:923–31.
D’Agostino RB, Vasan RS, Pencina MJ, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation2008; 117:743–53.
Schiborn C, Kühn T, Mühlenbruch K, et al. A newly developed and externally validated non-clinical score accurately predicts 10-year cardiovascular disease risk in the general adult population. Sci Rep2021; 11.
Joseph P, Yusuf S, Lee SF, et al. Prognostic validation of a non-laboratory and a laboratory based cardiovascular disease risk score in multiple regions of the world. Heart2018; 104:581–7.
Kaptoge S, Pennells L, De Bacquer D, et al. World Health Organization cardiovascular disease risk charts: revised models to estimate risk in 21 global regions. Lancet Glob Health2019; 7:e1332–45.
Dehghan A, Rayatinejad A, Khezri R, et al. Laboratory-based versus non-laboratory-based World Health Organization risk equations for assessment of cardiovascular disease risk. BMC Med Res Methodol2023; 23.
Ahmad N, Boutron I, Dechartres A, et al. Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review. Trials2010; 11.
Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology2010; 21:128–38.
Moons KGM, Wolff RF, Riley RD, et al. PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med2019; 170:W1–33.
Wolff RF, Moons KGM, Riley RD, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med2019; 170:51.
Pencina MJ, D’Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med2004; 23:2109–23.
Caetano SJ, Sonpavde G, Pond GR, et al. C-statistic: A brief explanation of its construction, interpretation and limitations. Eur J Cancer2018; 90:130–2.
Lin JS, Evans CV, Johnson E, et al. Nontraditional Risk Factors in Cardiovascular Disease Risk Assessment: Updated Evidence Report and Systematic Review for the US Preventive Services Task Force. JAMA2018; 320:281–97.
Hua X, McDermott R, Lung T, et al. Validation and recalibration of the Framingham cardiovascular disease risk models in an Australian Indigenous cohort. Eur J Prev Cardiol2017; 24:1660–9.
D’Agostino RB, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA2001; 286:180–7.
Riley RD, Ensor J, Snell KIE, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ2016; 353.
Snell KI, Ensor J, Debray TP, et al. Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res2018; 27:3505–22.
Albarqouni L, Doust JA, Magliano D, et al. External validation and comparison of four cardiovascular risk prediction models with data from the Australian Diabetes, Obesity and Lifestyle study. Med J Aust2019; 210:161–7.
Al-Shamsi S, Govender RD, King J, et al. External validation and clinical usefulness of three commonly used cardiovascular risk prediction scores in an Emirati population: a retrospective longitudinal cohort study. BMJ Open2020; 10.
Kariuki JK, Stuart-Shor EM, Leveille SG, et al. Validation of the nonlaboratory-based Framingham cardiovascular disease risk assessment algorithm in the Atherosclerosis Risk in Communities dataset. J Cardiovasc Med (Hagerstown)2017; 18:936–45.
Li J, Liu F, Yang X, et al. Validating World Health Organization cardiovascular disease risk charts and optimizing risk assessment in China. Lancet Reg Health West Pac2021; 8.
Gaziano TA, Pandya A, Steyn K, et al. Comparative assessment of absolute cardiovascular disease risk characterization from non-laboratory-based risk assessment in South African populations. BMC Med2013; 11:1–11.
Pandya A, Weinstein MC, Gaziano TA, et al. A comparative assessment of non-laboratory-based versus commonly used laboratory-based cardiovascular disease risk scores in the NHANES III population. PLoS ONE2011; 6.
Gaziano TA, Abrahams-Gessel S, Alam S, et al. Comparison of Nonblood-Based and Blood-Based Total CV Risk Scores in Global Populations. Glob Heart2016; 11:37–46.
Kumar S. Cost effectiveness of non-laboratory cvd screening in uzbekistan. yale university: elischolar – a digital platform for scholarly publishing at yale. 2014;
Guzman-Vilca WC, Quispe-Villegas GA, Váscones Román FF, et al. Agreement between the laboratory-based and non-laboratory-based WHO cardiovascular risk charts: a cross-sectional analysis of a national health survey in Peru. BMJ Open2022; 12.
Rezaei F, Seif M, Gandomkar A, et al. Agreement between laboratory-based and non-laboratory-based Framingham risk score in Southern Iran. Sci Rep2021; 11:10767.
Jahangiry L, Dehghan A, Farjam M, et al. Laboratory-based and office-based Globorisk scores to predict 10-year risk of cardiovascular diseases among Iranians: results from the Fasa PERSIAN cohort. BMC Med Res Methodol2022; 22:305.
Ananda Selva Das P, Dubey M, Kaur R, et al. WHO Non-Lab-Based CVD Risk Assessment: A Reliable Measure in a North Indian Population. Glob Heart2022; 17.
Boateng D, Agyemang C, Beune E, et al. Cardiovascular disease risk prediction in sub-Saharan African populations — Comparative analysis of risk algorithms in the RODAM study. Int J Cardiol2018; 254:310–5.
Wekesah FM, Mutua MK, Boateng D, et al. Comparative performance of pooled cohort equations and Framingham risk scores in cardiovascular disease risk classification in a slum setting in Nairobi Kenya. Int J Cardiol Heart Vasc2020; 28.
Muiru AN, Bibangambah P, Hemphill L, et al. Distribution and Performance of Cardiovascular Risk Scores in a Mixed Population of HIV-Infected and Community-Based HIV-Uninfected Individuals in Uganda. J Acquir Immune Defic Syndr2018; 78:458–64.
Sivanantham P, Kar SS, Lakshminarayanan S, et al. Performance of WHO updated cardiovascular disease risk prediction charts in a low-resource setting – Findings from a community-based survey in Puducherry, India. Nutr Metab Cardiovasc Dis2022; 32:2129–36.
Enriquez R, Ssekubugu R, Ndyanabo A, et al. Prevalence of cardiovascular risk factors by HIV status in a population‐based cohort in South Central Uganda: a cross‐sectional survey. J Int AIDS Soc2022; 25.
Jones CA, Ross L, Surani N, et al. Framingham ten-year general cardiovascular disease risk: agreement between BMI-based and cholesterol-based estimates in a South Asian convenience sample. PLoS ONE2015; 10.
Berchmans Niyibizi J, Kufre Joseph O, Naomi L, et al. The Comparability of Lipid-based and Body Mass Index-based Cardiovascular Disease Risk Scores: Using the Rwanda 2012-2013 Non-communicable Diseases Risk Factors Survey Data. RJMHS2021; 4:166–84.
Deori TJ, Agarwal M, Masood J, et al. Estimation of cardiovascular risk in a rural population of Lucknow district using WHO/ISH risk prediction charts. J Family Med Prim Care2020; 9:4853–60.
Fatema K, Zwar NA, Milton AH, et al. Application of two versions of the WHO/international society of hypertension absolute cardiovascular risk assessment tools in a rural Bangladeshi population. BMJ Open2015; 5.
Nordet P, Mendis S, Dueñas A, et al. Total cardiovascular risk assessment and management using two prediction tools, with and without blood cholesterol. MEDICC Rev2013; 15:36–40.
Wagner RG, Crowther NJ, Micklesfield LK, et al. Estimating the burden of cardiovascular risk in community dwellers over 40 years old in South Africa, Kenya, Burkina Faso and Ghana. BMJ Glob Health2021; 6.
Green BB, Anderson ML, Cook AJ, et al. Using Body Mass Index Data in the Electronic Health Record to Calculate Cardiovascular Risk. Am J Prev Med2012; 42:342–7.
Sarrafzadegan N, Hassannejad R, Marateb HR, et al. PARS risk charts: A 10-year study of risk assessment for cardiovascular diseases in Eastern Mediterranean Region. PLoS ONE2017; 12.
McGorrian C, Yusuf S, Islam S, et al. Estimating modifiable coronary heart disease risk in multiple regions of the world: the INTERHEART Modifiable Risk Score. Eur Heart J2011; 32:581–9.
Damen JA, Pajouheshnia R, Heus P, et al. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med2019; 17.
Kerr KF, Wang Z, Janes H, et al. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology2014; 25:114–21.
Pencina MJ, D’Agostino RB, D’Agostino RB, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med2008; 27:157–72.
Pylypchuk R, Wells S, Kerr A, et al. Cardiovascular disease risk prediction equations in 400 000 primary care patients in New Zealand: a derivation and validation study. Lancet2018; 391:1897–907.
Rodríguez-Ariza CD, Cabrera-Villamizar A, Rodríguez-Pulido AL, et al. External validation of the ACC/AHA ASCVD risk score in a Colombian population cohort. Sci Rep2023; 13.
Pennells L, Kaptoge S, Wood A, et al. Equalization of four cardiovascular risk algorithms after systematic recalibration: individual-participant meta-analysis of 86 prospective studies. Eur Heart J2019; 40:621–31.
Laukkanen JA, Kunutsor SK. Is ‘re-calibration’ of standard cardiovascular disease (CVD) risk algorithms the panacea to improved CVD risk prediction and prevention? Eur Heart J2019; 40:632–4.
Chen L, Tonkin AM, Moon L, et al. Recalibration and validation of the SCORE risk chart in the Australian population: the AusSCORE chart. Eur J Cardiovasc Prev Rehabil2009; 16:562–70.
Ogero M, Ndiritu J, Sarguta R, et al. Recalibrating prognostic models to improve predictions of in-hospital child mortality in resource-limited settings. Paediatr Perinat Epidemiol2023; 37:313–21.
Davis SE, Lasko TA, Chen G, et al. Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc2017; 24:1052–61.
Van Calster B, Nieboer D, Vergouwe Y, et al. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol2016; 74:167–76.
