Journal Pre-proof Derivation and external validation of the SIMPLICITY Score as a simple immune-based risk score to predict infection in kidney transplant recipients.

Existing approaches for infection risk stratification in kidney transplant recipients are suboptimal. Here, we aimed to develop and validate a weighted score integrating non-pathogen-specific immune parameters and clinical variables to predict the occurrence of post-transplant infectious complications. To this end, we retrospectively analyzed a single-center derivation cohort of 410 patients undergoing kidney transplantation in 2008-2013 in Madrid. Peripheral blood lymphocyte subpopulations, serum immunoglobulin and complement levels were measured at one-month post-transplant. The primary and secondary outcomes were overall and bacterial infection through month six. A point score was derived from a logistic regression model and prospectively applied on a validation cohort of 522 patients undergoing kidney transplantation at 16 centers throughout Spain in 2014-2015. The SIMPLICITY score consisted of the following variables measured at month one after transplantation: C3 level, CD4 + T-cell count, CD8 + T-cell count, IgG level, glomerular filtration rate, recipient age, and infection within the first month. The discrimination capacity in the derivation and validation cohorts was good for overall (areas under the receiver operating curve of 0.774 and 0.730) and bacterial infection (0.767 and 0.734, respectively). The cumulative incidence of overall infection significantly increased across risk categories in the derivation (low-risk 13.7%; intermediate-risk, 35.9%; high-risk77.6%) and validation datasets (10.2%, 28.9% and 50.4%, respectively). Thus, the SIMPLICITY score, based on easily available immune parameters, allows for stratification of kidney transplant recipients at month one according to their expected risk of subsequent infection.

The occurrence of infectious complications is one of the major drawbacks with current clinical practices in solid organ transplantation (SOT). 1 Despite its restrictive pharmacokinetic nature, therapeutic drug monitoring of immunosuppressive drugs constitutes the only approach widely used to investigate the state of post-transplant immunosuppression. 2,3 An increasing number of immunological monitoring strategies have been proposed over the last years, ranging from the enumeration of non-pathogen-specific parameters (e.g. peripheral blood lymphocyte subpopulations [PBLSs]) to labor-consuming functional assays for pathogen-specific (e.g. cytomegalovirus [CMV]) cell-mediated immunity. 4,5 Our group has previously demonstrated the value of post-transplant hypogammaglobulinemia (HGG), C3 hypocomplementemia (HCC) and low PBLS counts to identify kidney transplant (KT) recipients at increased risk of post-transplant infection. [6][7][8] Clinicians still face challenges in translating such biomarkers, which usually explore nonoverlapping effector immune mechanisms in a compartmentalized fashion, into daily practice.
The integration of various easily available parameters covering innate and adaptive responses into a single risk score might provide a valuable support to the clinical decision-making process.
Nevertheless, previous attempts to develop an immune-based score aimed at assessing the risk of infection after SOT have been of limited applicability due to the single-center design or small sizes of studies. [9][10][11][12][13][14][15][16] Moreover, some of them incorporate complex assay procedures (such as T-cell proliferative responses) not always accessible to clinical laboratories. 10,14 More importantly, none of these scores have been externally validated and, since the same datasets were usually used to derive prediction rules and to calculate diagnostic proprieties, overestimation of predictive capacity cannot be ruled out. 17 With these gaps in mind, our study was aimed at developing a weighted risk score based on simple non-pathogen-specific immune parameters and clinical variables to predict infection among KT recipients recruited in a single-center derivation cohort. We next assembled a large cohort at 16 Spanish centers to externally validate the predictive accuracy of the score.

Characteristics of derivation and validation cohorts
Overall, 489 and 570 KT recipients were potentially eligible for inclusion in the derivation and validation cohorts. After screening assessment, 410 and 522 patients were finally included in each cohort, respectively (Figure S1 inhibitors), and duration of anti-CMV prophylaxis ( Table 1).
One-year patient and death-censored graft survival in the derivation cohort were 91.4% and 93.6%. The corresponding rates for the validation cohort were 97.6% and 97.1%, respectively.
Three (0.7%) and 4 (0.8%) patients were lost to follow-up in each cohort before completing the 6-month post-transplant period.

Post-transplant infection in derivation and validation cohorts
In the derivation cohort, 133 ( (Figure 1b). Causative agents and clinical syndromes are also detailed in Supporting Material (Table S2).

Model derivation
As expected from previous studies, [6][7][8] patients in the derivation cohort that subsequently developed infection had lower PBLS counts and IgG and complement levels at month 1 compared to those who remained free from infection. Significant differences were found for CD3 + (P-value = 0.007), CD4 + (P-value = 0.025) and CD8 + T-cell counts (P-value = 0.004), and serum C3 levels (P-value = 0.0026) ( Figure S2). Although IgG levels measured as a continuous variable did not significantly differ between patients with or without infection, a dose-response gradient in the cumulative incidence at month 6 was found across increasing degrees of IgG HGG ( Figure S3).
Since strong collinearity was found between the CD3 + T-cell count and the other PBLSs  Table 3.
The excess risk of overall infection associated with higher scores was confirmed after adjusting for the occurrence of BPAR through post-transplant month 6 as a time-dependent covariate (hazard ratio [HR] per one-point increment: 1.18; 95% CI: 1.50 -1.23; P-value <0.0001), as well as in a set of sensitivity analyses (Figure 4). These results also applied to post-transplant bacterial infection (secondary outcome) ( Figure S4).

Performance of the risk score in the validation cohort
Similarly to that observed in the derivation cohort, PBLS counts (CD3 + , CD4 + and CD8 + T-cells) and serum C3 levels at post-transplant month 1 were significantly lower among patients with overall infection through month 6 compared to those without ( Figure S5).

Calibration of the risk score
Hazard ratios for study outcomes across increasing categories of the SIMPLICITY score were in the same order of magnitude in both the derivation and validation cohorts (for example, 8.188 and 6.922, respectively, for high-risk [score ≥10] versus low-risk [score 0-3] strata), indicating a robust capacity for risk stratification ( Table S6).
The calibration plot for predicting the primary outcome ( Figure S8a) revealed a calibration intercept (β 0 ) of -0.669 (95% bootstrap CI: -0.985 --0.363), whereas the calibration slope was 0.752 (95% bootstrap CI: 0.539 -1.018) ( Table S7). The intercept relates to calibration-in-thelarge, which compares the mean of all predicted risks with the mean observed risk. 19 Since the 95% CI of this estimate was negative, it could be deduced that the predicted probabilities were systematically too high. On the other hand, the calibration slope -which is related to the average strength of the predictor effects-did not significantly differ from 1.00, indicating that there was no evidence of average stronger or weaker effects in the derivation model.
To investigate the cause of this suboptimal calibration, we calculated the mean and the SD of the linear predictor (LP) of the original model in both datasets. The LP results from the logit transformation of the predicted risks in logistic regression, with an increased (or decreased) variability of LP indicating more (or less) case-mix heterogeneity between derivation and validation cohorts. Conversely, the comparison of means of the LP reveals differences in outcome frequencies. 20 Mean LP was higher in the derivation (-0.897) than in the validation dataset (-1.141), with similar SDs (1.177 and 1.114, respectively), which suggests that the calibration-in-the-large of the model in the external validation dataset would be mostly affected by differences in case mix-severity rather than by other sources of heterogeneity. 20 To overcome this circumstance, we explored to which extent an "updated" score would adjust better to differences in outcome incidence observed in the validation cohort. It has been shown that a prediction model may be updated (i.e. adjusted) to the new dataset obtained from a setting different from that in which it was originally developed. 21,22 The updated model is simultaneously constructed on both the derivation and the validation data, yielding better risk estimates. After adjusting the intercept of the model, we obtained an alternative point assignment (Table S8). Predictive performance and calibration of this updated score are available as Supporting Material ( Figure S8b and Table S7).

Imputation of missing data
Finally, imputation of missing values for the continuous parameters included in the score (eGFR, CD4 + and CD8 + T-cell counts, and IgG and C3 levels) was performed in the derivation cohort, obtaining similar β regression coefficients ( Table S9). The resulting model was then validated in four different imputed datasets, with auROC values for predicting the primary study outcome very close to that obtained in the original analysis with no missing data imputation (Table S10).

Discussion
Given the deleterious impact on graft and patient outcomes attributable to post-transplant infection, the development and validation of prediction rules able to effectively stratify the KT population according to individual susceptibility constitutes a crucial unmet clinical need. Herein, we propose the SIMPLICITY score, which integrates four non-pathogen-specific quantitative immune biomarkers (CD4 + and CD8 + T-cell counts and serum IgG and C3 levels) and three simple clinical variables (recipient age, prior infection and graft function). In contrast to previous efforts, [9][10][11][12][13][14][15][16] our score has two major advantages. Firstly, the immune parameters included are broadly available in routine practice with a short turnaround time and no specialized laboratory equipment. Secondly, we have been able to validate the discriminative capacity and diagnostic accuracy of the score in an independent cohort recruited in 16 Spanish centers. to the demonstrated decreaed risk of viral infection with these agents. 23 By collapsing the SIMPLICITY score into three strata, we can define a low-risk population (score 0-3) in which the expected cumulative incidence of overall infection through month 6 would be below 15% (13.7% and 10.2% in the derivation and validation cohorts). Alternatively, KT recipients in the high-risk segment (score ≥10) face a cumulative risk of infection that exceeds 50%. The decision to categorize the score according to these thresholds was based on the low number of patients with very high values. In accordance with this observation, the score did not exhibit a normal distribution, with a median of 5 points and 3 points for the derivation and validation cohorts, respectively. Future studies should estimate the equivalent to the "number necessary to treat", or how many patients above the different score thresholds should receive extended prophylaxis or immunoglobulin therapy to prevent one additional episode of infection.
The observed rates of infection during the post-transplant period were lower in the validation than in the derivation cohort and, therefore, the PPV for scores ≥10 differed across datasets. In view of the different recruitment periods (2008-2013 versus 2014-2015), such a discrepancy may be explained by long-term improvements in surgical procedures, immunosuppression and prophylaxis regimens. Even in the more contemporary period, in which a secular trend towards a sustained reduction in the incidence of infectious complications after KT has been suggested, 24 the score was still able to identify a subgroup of recipients exposed to an unacceptable risk of severe, potentially life-threatening infection. By simply updating the intercept of the original model for differences in outcome frequency (a methodological approach increasingly used in clinical research 21,22 ), we also propose an alternative "updated" score with improved calibration that could be applicable in settings with low a priori infection rates.  13 However, no information on score performance was provided, and we were not able to externally validate the discriminative capacity of this immune risk phenotype. 25  including CMV disease and P. jirovecii pneumonia. 26,27 Post-transplant HGG is a common and often neglected complication, with mild-to-severe forms occurring in as many as 39% and 15% of SOT recipients during the first year. 28 Since the humoral response is responsible for the clearance of encapsulated bacteria through opsonization and complement activation, posttransplant IgG HGG serves as a good predictor for bacterial infection. 29 Finally, the C3 component plays a pivotal role in the complement activation cascade to form the C5 convertase and to assemble the membrane attack complex. 30 Therefore, the assessment of C3 HCC by nephelometry may advantageously replace more complex in vitro haemolytic assays to explore its functionality.

Study population and setting
The present observational derivation-validation study was performed in two non-overlapping cohorts. The first one (derivation cohort) comprised adult patients (≥18 years) with ESRD who Specific infectious syndromes were diagnosed on the basis of commonly accepted criteria. [33][34][35][36] CMV disease included viral syndrome and end-organ disease defined as per the American Society of Transplantation criteria. 37 Proven or probable invasive fungal infection was defined according to the European Organization on Research and Treatment in Cancer and the Mycoses Study Group criteria. 38 The eGFR was assessed by the 4-variable Modification of Diet in Renal Disease equation. 39 The occurrence of BPAR was suspected in case of an otherwise unexplained rise in serum creatinine and diagnosed by histological examination. 40

Statistical analysis
Quantitative data were shown as the mean ± standard deviation or the median with IQR.  Continuous variables were dichotomized according to the optimal cut-off value, as determined by the Youden's index. 18 The Hosmer-Lemeshow test was used to assess the goodness-of-fit of the models. Discriminative capacity was estimated by the auROC. Multicollinearity among explanatory variables was analyzed using the VIF, with values <3 being considered acceptable. Time-to-event curves were plotted by the Kaplan-Meier method and inter-group differences were compared with the log-rank test to analyze the ability of the score for stratification across increasing risk categories. Univariate and multivariate Cox regression models were constructed to investigate whether the associations found between score risk categories and outcomes remained significant after adjusting for clinical covariates, with associations expressed HRs and 95% CIs. To show robustness of the score, a set of sensitivity analyses restricted to specific causes of ESRD, D/R CMV serostatus, or immunosuppression or prophylaxis regimens was also performed. All these validation analyses were also carried out for post-transplant bacterial infection (secondary outcome).
The resulting score was next applied to the independent external validation cohort to assess its discriminative capacity (auROC), diagnostic accuracy and calibration. Calibration indicates how closely predicted probabilities match observed frequencies of occurrence, and was graphically assessed by means of calibration plots.

Funding sources
This study was partially supported by the Spanish Ministry of Science, Innovation and     o Table S4. Derivation cohort: Variance inflation factors assessed to control for multicollinearity among the explanatory variables included in the predictive model.
o Table S5. Validation cohort: Diagnostic accuracy of the SIMPLICITY score at month 1 for predicting overall infection between post-transplant months 1 and 6 (primary outcome).
o Table S6. Hazard ratios across SIMPLICITY score risk categories for overall and bacterial infection between post-transplant months 1 and 6 in the derivation and validation cohorts.
o Table S7. Predictive performance and calibration parameters of the SIMPLICITY score.
o Table S8. Updated SIMPLICITY score: the intercept (β 0 ) of the original model has been updated according to the dataset derived from the validation cohort, resulting in alternative point assignment.
o Table S9. Supplementary information is available on Kidney International's web site.