ORIGINAL ARTICLE

JOP. J Pancreas (Online) 2005; 6(2):231-237.

Estimating the Diagnostic Accuracy of Procalcitonin as a Marker of the Severity of Acute Pancreatitis: A Meta-Analytic Approach

Nusrat Shafiq¹, Samir Malhotra¹, Deepak K Bhasin², Surinder Rana², Shabir Siddhu¹, Promila Pandhi¹

Departments of ¹Pharmacology and ²Gastroenterology, Post Graduate Institute of Medical Education and Research. Chandigarh, India

Context Approximately 15-20% of cases of acute pancreatitis are categorized as severe. There is a lack of accurate predictors of disease severity. Several studies have evaluated the usefulness of procalcitonin as a marker of severe disease. Reports regarding the diagnostic accuracy of procalcitonin are conflicting.

Objective The present meta-analysis was carried out to evaluate the relevance of procalcitonin as a predictor of disease severity.

Methods Two investigators working independently attempted to locate eligible studies by electronic and manual means. Studies in which at least one of the markers of disease severity was procalcitonin were included for analysis. For all the studies included, the following parameters were calculated: true positive, false negative, false positive and true negative. A summary receiver operating characteristic (SROC) curve was generated from these parameters.

Results Four studies were finally included in the analysis. The unweighted regression line parameters b and i were 3.633 and 1.399, respectively. The values for b and i for weighted regression line were 3.637 and 1.428. The SROC curve generated demonstrated that procalcitonin is not a good predictor of the severity of acute pancreatitis.

Conclusion The available data indicates that procalcitonin cannot be considered a good marker for assessing the severity of pancreatitis.

INTRODUCTION

Acute pancreatitis is usually a mild disease with minimal organ dysfunction. However, 15-20% of all cases demonstrate severe acute pancreatitis [1, 2]. In acute pancreatitis, early assessment of the patient which can lead to an accurate prediction of the severity is useful for several reasons. The first well-established step is the need to categorize patients at risk for complications for appropriate stratification in clinical trials. Furthermore, it is important to identify the patients who are at risk for developing complications in order to be able to initiate effective management before those complications develop.

The lack of accurate predictors of disease severity makes such categorization difficult. Several biochemical parameters [3], contrast-enhanced computed tomography [4, 5], and multiple clinico-biochemical scores [6, 7] have been used to assess the severity of acute pancreatitis. An ideal prognostic method should be simple, inexpensive, routinely available and highly accurate. Such a method is however not yet available.

Procalcitonin is a 116-amino acid propeptide of calcitonin with a molecular weight of 13 kDa [8]. It has been introduced as an early marker of severe infection and inflammation [9, 10]. Several studies [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] have evaluated the usefulness of procalcitonin as a predictor of severity and the development of infected necrosis in acute pancreatitis.

In view of the conflicting reports of the diagnostic accuracy of procalcitonin in predicting the severity of pancreatitis, we aimed at adopting a meta-analytic approach for arriving at a conclusion regarding this test as a predictor of severity.

We systematically searched MEDLINE and EMBASE for all relevant articles until November 2004. We first researched medical subject heading (MeSH) terms and textwords for "Markers" AND "Acute Pancreatitis". Secondly, we researched MeSH terms and textwords for "Procalcitonin" AND "Acute Pancreatitis" AND "Severity". We then combined the two searches and retrieved all the relevant articles found by either search. The manual search was carried out by looking at the reference lists of the retrieved articles and the Index Medicus. All the relevant articles thus obtained were combined with those obtained from the electronic search.

Two investigators conducted the search independently. Studies in which at least one of the markers for predicting the severity of pancreatitis using procalcitonin were included. Studies had to present the sensitivity and specificity of the procalcitonin test as a predictor of the severity of pancreatitis to be considered for inclusion. Otherwise, the study had to present enough data to allow them to be calculated. Studies conducted exclusively in patients with post-ERCP pancreatitis, and those evaluating outcomes other than the severity of pancreatitis, such as the development of infected necrosis or multiple organ failure only, were excluded from the evaluation.

A summary receiver operating curve (SROC) as described previously [21] was generated. Briefly, in tests of diagnostic accuracy, sensitivity and specificity are calculated using a particular threshold. For the determination of sensitivity and specificity, a threshold value is generally decided a priori. Several studies done to evaluate the effectiveness of a particular diagnostic test in predicting some outcomes may show different sensitivity and specificity values because of different thresholds. SROC is a method of pooling the results of different studies to judge the predictive value of a test. The following parameters were calculated from the studies: true positive (TP), false negative (FN), false positive (FP) and true negative (TN). The true positive rates (TPR) and false positive rates (FPR) were converted to their logistic transformations by using the formulae given below:

Logit (TPR) = ln (TPR / (1 - TPR))
Logit (FPR) = ln (FPR / (1 - FPR))

The sum (S) and difference (D) of the two transformations were then calculated:

S = Logit (TPR) + Logit (FPR)
D = Logit (TPR) - Logit (FPR)

D is equivalent to the diagnostic log-odds ratio (ln(OR)), which conveys the accuracy of the test in discriminating cases from non-cases. S can be interpreted as a measure of the diagnostic threshold, with high values corresponding to liberal inclusion criteria for cases. S = 0 when TPR= 1 - FPR, that is, on the anti-diagonal from the top-left to bottom-right corners of the SROC space [22].

Next, in order to estimate the relationship between D and S, the two were adapted to a linear model:

D = b S + i

The coefficient b represents the dependence of the test accuracy on the threshold; if b is near 0, then the studies are homogeneous and can by summarized by an overall OR, noting that i = ln(OR). If b is not equal to 0, then the studies are heterogeneous with respect to OR. In this case, i can be thought of as the value of ln(OR) when S = 0 [22].

Weights were employed to reflect inter-study heterogeneity with respect to the sample variance of D [22]. The weighting parameter ‘w’ was defined as:

w = 1 / (1 / (TP + 0.5) + 1 / (FN + 0.5) + 1 / (TN + 0.5) + 1 / (FP + 0.5))

Both weighted and unweighted regression line parameters b and i were obtained.

These values were then utilized to return to the transformed values for TPR as given by the following formula:

TPR = 1 / (1 + (1 / (e ⁱ / (1 - b).(FPR / (1 - FPR)) ^{(1 + b)/(1 - b)})))

A plot between TPR thus obtained and FPR gave the SROC.

Estimates were tested vs. 0 by the t-test and were reported together with their standard errors (SE) and 95% confidence intervals (CI). The chi-squared test and linear regression were applied. The statistical analyses were performed by running the SPSS (version 8.0 for Windows) using a personal computer.

Ten studies were identified [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] out of which four [11, 12, 13, 14] were finally included in the analysis (Figure 1). The details of the studies included are given in Table 1. These studies included a total of 313 patients with acute pancreatitis. These patients had been categorized into mild and severe cases irrespective of the etiology of the pancreatitis. The TP, FP, FN, TN values, their logit transformed values and the 95% confidence intervals are given in Table 2.

Figure 1. Flowchart of studies evaluated for inclusion in the meta-analysis.

The parameters b and i (±SE) obtained from these data by unweighted regression analysis (r=0.968; P=0.016) were 3.584±0.381 (95% CI: 1.946-5.222; P=0.011) and 1.297±0.166 (95% CI: 0.580-2.013; P=0.016), respectively. The values for b and i for weighted regression line (r=0.964; P<0.001) were 3.601±0.142 (95% CI: 3.280-3.922; P<0.001) and 1.332±0.085 (95% CI: 1.139-1.524; P<0.001), respectively. Both the estimated b values were significantly different from 0 and, therefore, the studies resulted heterogeneous with respect to OR.

The unweighted SROC generated from the reverse extrapolated values of TPR plotted against FPR demonstrates that procalcitonin is not a good predictor of the severity of acute pancreatitis (Figure 2). Out of the 4 studies included in our analysis, two showed very low sensitivities [12, 14], and paradoxically, higher false positive rates than the other two studies [11, 13]. To rule out random error as being a cause of such a finding [23], TPR values were confirmed to be non-homogeneously distributed among the 4 studies (P<0.001), while no significant differences were obtained for FPR values (P=0.750).

Figure 2. Summary receiver operating (SROC) curve for procalcitonin as a marker of the severity of acute pancreatitis.

Moreover, the regression coefficient for TPR and FPR was negative (r = -0.894) when all the studies were considered, suggesting that, basically, a valid SROC curve cannot be found using the present data, and therefore, a single summary curve should not be constructed [23].

In our study, we have used a relatively new approach of meta-analyzing diagnostic studies, namely the SROC method for assessing the utility of procalcitonin as a predictor of the severity of acute pancreatitis. Reports on diagnostic tests when being evaluated in the initial stages, may show a large discrepancy. Such a situation was also observed for ‘procalcitonin’ as a marker for the severity of acute pancreatitis. A plot between TPR and FPR at various thresholds, the ROC plot, is commonly used in presenting the report of a single study. SROC curves give a good overview of pooled results of several studies [24].

The results of our meta-analysis show that procalcitonin may not be a useful marker for estimating the severity of acute pancreatitis. Several elements of this analysis merit further discussion.

First, two studies [12, 14] showed very low sensitivity, and paradoxically, higher false positive rates than the other two studies [11, 13]. In view of this lack of a monotonically increasing relationship between TPR and FPR in two of the studies included in our meta-analysis, we needed to rule out random error as being a cause of such a finding [23]. This was done showing that the TPRs were not found to be homogenous. Because of this lack of homogeneity among the studies, pooling of such data may be problematic. However, exclusion of negative studies from systematic reviews and meta-analyses may not be the best approach [24] since they already suffer the drawback of publication bias (negative studies are less likely to be published). Therefore, we decided to include all eligible studies in our meta-analysis.

There may be several reasons for the heterogeneity observed. First, studies showing procalcitonin to be a poor marker of the severity of pancreatitis had a greater percentage (70% versus 35%) of patients with pancreatitis of biliary origin [12, 14] and it has been observed that procalcitonin may not be a sensitive marker for this type of pancreatitis [14, 18]. This could have contributed to the heterogeneity observed. Secondly, different assay techniques (manual methods or kits) were used in different studies. Also, the coefficient of variation was not reported by any of the studies. These factors may also have contributed to the heterogeneity. Thirdly, the time of blood sampling with respect to the onset of symptoms may also lead to some discrepancy in test results. For example, in one study [11] showing good correlation of procalcitonin with severity, the procalcitonin was measured even after 4 days of the onset of the symptoms and in another [12] showing poor correlation, the measurement was done early (within the first 24 hours of the onset of symptoms). Since the markers of severity may be highly time-sensitive [12], such differences can also account for the heterogeneity among the studies.

It is well-known that the closer the ROC and the SROC curves are to the upper left-hand quadrant, the more accurate they are, because the TPR is 1 and the FPR is 0. The SROC curve obtained by us does not lie in the upper left quadrant as would be desired. Other than the explanations given above, a small sample size (with respect to both the number of studies available for analysis as well as the number of patients included in each study) may be another possible explanation for our SROC curve not occupying the upper left-hand corner [22]. The total sample size in our study was 313 patients, with two studies [12, 14] having sample sizes less than fifty. However, the position of the graphical points obtained in the subgroup analysis shown in Figure 2 may be expected to result in an optimal SROC curve provided a larger number of studies with similar results are available.

Other studies [15, 18] have evaluated procalcitonin as a predictor of the development of complications such as pancreatic necrosis, multiple dysfunction syndrome, but not for the severity of the disease per se. Our exclusion criteria did not permit us to incorporate these studies in our analysis.

In a review on biological markers for assessing the severity of acute pancreatitis [3], procalcitonin was considered to be a good marker in predicting disease severity early on and it was assigned to category ‘A’. However, in this review, studies [12, 14] showing low sensitivity of procalcitonin as a marker for severity were not mentioned.

In conclusion, a valid SROC curve cannot be found in our data and this result indicates that, from the data available so far, procalcitonin cannot be considered a good marker for assessing the severity of acute pancreatitis. Studies with larger numbers of patients, with more homogenous patient populations, and better correlation between the onset of symptoms and blood sampling and more similarity in the assay techniques are required in order to resolve the issue.

References

Received January 26^th, 2005 - Accepted March 24^th, 2005

Keywords Biological Markers; Calcitonin; Meta-Analysis; ROC Curve; Pancreatitis, Acute Necrotizing; Sensitivity and Specificity

Abbreviations FN: false negative; FP: false positive; FPR: false positive rate MeSH: medical subjects heading; SROC: summary receiver operating curve; TN: true negative TP: true positive; TPR: true positive rate

Acknowledgements Ashok Saluja, UMASS, and the Editor of Hepatogastroenterology for providing us some of the articles for our meta-analysis

Correspondence
Samir Malhotra
Department of Pharmacology
Post Graduate Institute of Medical Education and Research
Chandigarh
India-160012
Phone: +91-172.275.5243
Fax: +91-172.274.4401
E-mail: samirmalhotra345@yahoo.com