Skip Navigation

JNCI Journal of the National Cancer Institute 2006 98(11):765-774; doi:10.1093/jnci/djj209
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (50)
Right arrow Request Permissions
Google Scholar
Right arrow Articles by Ronco, G.
Right arrow Articles by Cuzick, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Ronco, G.
Right arrow Articles by Cuzick, J.
Social Bookmarking
 Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2006. Published by Oxford University Press.

ARTICLE

Human Papillomavirus Testing and Liquid-Based Cytology: Results at Recruitment From the New Technologies for Cervical Cancer Randomized Controlled Trial

Guglielmo Ronco, Nereo Segnan, Paolo Giorgi-Rossi, Marco Zappa, Gian Piero Casadei, Francesca Carozzi, Paolo Dalla Palma, Annarosa Del Mistro, Stefania Folicaldi, Anna Gillio-Tos, Gaetano Nardo, Carlo Naldoni, Patrizia Schincaglia, Manuel Zorzi, Massimo Confortini, Jack Cuzick
For the New Technologies for Cervical Cancer Working Group

Affiliations of authors: Unit of Cancer Epidemiology, Centro per la Prevenzione Oncologica, Turin, Italy (GR, NS); Agency for Public Health, Lazio Region, Rome, Italy (PGR); Centro per lo Studio e la Prevenzione Oncologica, Florence, Italy (MZ, FC, MC); Unit of Pathology, Ospedale Maggiore, AUSL Bologna, Italy (GPC); Unit of Pathology, Ospedale di Trento, Trento, Italy (PDP); Service of Immunology and Cancer Molecular Diagnostics, Azienda Ospedaliera di Padova, Padua, Italy (ADM); Unit of Pathology, Presidio Ospedaliero, AUSL Imola, Imola, Italy (SF); Unit of Cancer Epidemiology and CPO, CERMS, University of Turin, Turin, Italy (AGT); Preventive Gyneacological Oncology, Ospedale Civile Maggiore di Verona, Verona, Italy (GN); Centro di riferimento screening—Assessorato alla Sanità—Regione Emilia-Romagna, Bologna, Italy (CN); Centro Prevenzione Oncologica, AUSL Ravenna, Ravenna, Italy (PS); Venetian Tumour Registry, Azienda Ospedaliera di Padova, Padua, Italy (MZ); Queen Mary's School of Medicine and Dentistry and Cancer Research UK, London, U.K. (JC)

Correspondence to: Guglielmo Ronco, MD, PhD, Unit of Cancer Epidemiology, Centro per la Prevenzione Oncologica Piemonte, via San Francesco da Paola 31, 10123 Torino, Italy (e-mail: guglielmo.ronco{at}cpo.it).


    ABSTRACT
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Background: Although testing for human papillomavirus (HPV) has higher sensitivity and lower specificity than cytology alone for detecting cervical intraepithelial neoplasia (CIN), studies comparing conventional and liquid-based cytology have had conflicting results. Methods: In the first phase of a two-phase multicenter randomized controlled trial, women aged 35–60 years in the conventional arm (n = 16 658) were screened using conventional cytology, and women in the experimental arm (n = 16 706) had liquid-based cytology and were tested for high-risk HPV types using the Hybrid Capture 2 assay. Women in the conventional arm were referred to colposcopy with atypical cells of undetermined significance (ASCUS) or higher and those in the experimental arm were referred with ASCUS or higher cytology or with a positive (≥1 pg/mL) HPV test. Sensitivity and positive predictive value (PPV) for detection of cervical intraepithelial neoplasia grade 2 or higher (CIN2+) were calculated. Results: The screening methods and referral criterion applied in the experimental arm had higher sensitivity than that in the conventional arm (relative sensitivity = 1.47; 95% confidence interval [CI] = 1.03 to 2.09) but a lower PPV (relative PPV = 0.40; 95% CI = 0.23 to 0.66). With HPV testing alone at ≥1 pg/mL and at ≥2 pg/mL, the gain in sensitivity compared with the conventional arm remained similar (relative sensitivity = 1.43, 95% CI = 1.00 to 2.04 and relative sensitivity = 1.41, 95% CI = 0.98 to 2.01, respectively) but PPV progressively improved (relative PPV = 0.58, 95% CI = 0.33 to 0.98 and relative PPV = 0.75, 95% CI = 0.45 and 1.27, respectively). Referral based on liquid-based cytology alone did not increase sensitivity compared with conventional cytology (relative sensitivity = 1.06; 95% CI = 0.72 to 1.55) but reduced PPV (relative PPV = 0.57; 95% CI = 0.39 to 0.82). Conclusions: HPV testing alone was more sensitive than conventional cytology among women 35–60 years old. Adding liquid-based cytology improved sensitivity only marginally but increased false-positives. HPV testing using Hybrid Capture 2 with a 2 pg/mL cutoff may be more appropriate than a 1 pg/mL cutoff for primary cervical cancer screening.



    INTRODUCTION
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Several studies have demonstrated that human papillomavirus (HPV) testing has a greater sensitivity but lower specificity than cytology for detecting high-grade cervical intraepithelial neoplasia (CIN) (113). All of these studies have employed a split-sample, or two-sample, design, in which each woman is her own control. Although this is a suitable design for examining the cross-sectional sensitivity and false-positive rate for different tests, it does not permit long-term evaluation of different management strategies because usually all women with abnormal results from either test are subjected to further assessment and, possibly, treatment.

Liquid-based cytology has been widely introduced, especially in the last 5 years. However, evaluations of its sensitivity and specificity have severe limitations (14), because only a few studies computed relative sensitivity and false-positives in a primary screening setting using high-grade histology as the endpoint (1519). In addition, either comparisons were made between nonrandomized populations (16,17) or a split-sample design was used (15,18). The latter approach may compromise the accuracy of liquid-based cytology that is performed on the second sample because the diagnostic cells can be removed when taking the first sample.

We designed a population randomized controlled trial, the New Technologies for Cervical Cancer (NTCC) study, to compare the effectiveness, acceptability, and cost of HPV testing and conventional cytology for primary screening for cervical cancer. In this study, we randomly assigned women to either conventional cytology or to the experimental arm. In the first phase of the trial, women in the experimental arm were screened using thin-layer cytology and HPV testing of samples that were collected in liquid medium, and in the second phase women were screened using HPV testing alone. Thus, different groups of women were tested and underwent assessment and treatment according to different screening strategies. This design was chosen to permit the evaluation of subsequent long-term rates of disease (CIN and cancer) associated with each strategy. In addition, the study was conducted on a very large population within the routine activity of organized screening programs. Therefore, it evaluates the impact of introducing HPV testing in routine screening practice.

We present here data from the first phase of the trial on cross-sectional sensitivity and specificity at the first screening examination, which was conducted at recruitment. We focus on the effect of using different criteria for referral to colposcopy (concerning the combined use of HPV and liquid-based cytology and the cutoff used for HPV testing). These criteria are essential for defining the best modalities of screening by HPV testing before it can be adopted routinely.

Because more women who are younger than 35 years of age are HPV-positive than women 35 years and older (20), we applied a different protocol in the two age groups. Women younger than 35 years who were HPV-positive in the absence of cytologic abnormalities were advised to have repeat cytology and HPV testing, and women aged 35 years and older were referred directly to colposcopy if they were HPV-positive, independent of cytology. Because a different presentation of data is needed with the two phases, this report focuses only on the first phase, among women aged 35 years and older.


    SUBJECTS AND METHODS
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
Study Design

A randomized controlled trial was conducted in nine organized cervical screening programs in Italy that routinely actively invite women aged 25–64 years for screening (Table 1). Women aged 25–60 years who were attending a new routine cervical screening episode were asked to participate in the trial. Women who were pregnant, had undergone hysterectomy, or had been treated for CIN within the last 5 years were excluded. The persons acquiring the cervical samples, who had attended specific training, were required to obtain written informed consent from participants they recruited. Women were then randomly assigned to the conventional or to the experimental arm at a 1 : 1 ratio. In the Turin and Viterbo centers, computers were used. In the other centers, sealed numbered envelopes containing the random allocation were prepared by the local coordinating center, provided to each unit, and opened in sequence. The results of the assignment were then communicated to consenting women. The study was approved by the local ethics committees of the participating centers. The randomized clinical trial registration number is ISRCTN81678807 [controlled-trials.com] . Here we report on the first phase of the randomized trial among women aged 35 years and older.


View this table:
[in this window]
[in a new window]
 
Table 1.  Features of participating centers

 
The cervical cell samples were obtained by using a plastic Ayre's spatula and a cytobrush. A conventional smear was prepared for women who were randomly assigned to the conventional arm. During the first phase of the trial, cervical cells of women who were randomly assigned to the experimental arm were put in PreservCyt solution (ThinPrep; Cytyc Corporation, Boxborough, MA) and were used for both liquid-based cytology preparation and HPV testing.

Women assigned to the experimental arm were referred to colposcopy if cytology indicated atypical cells of undetermined significance (ASCUS) or higher. Women in this arm who were HPV-positive, independent of cytology results, were also referred to colposcopy.

Women assigned to the conventional arm underwent conventional cytology screening and were managed according to the protocol already in use for routine screening activity in each center. They were always referred to colposcopy if cytology was low-grade squamous intraepithelial lesion (LSIL) or higher. Depending on the protocol already in use in each center, women with ASCUS cytology were either directly referred to colposcopy (72%) or were recommended for repeat screening (28%) and were referred for colposcopy if repeat cytology results were LSIL or higher (Table 1).

The endpoint of the present analysis was histology-confirmed cervical intraepithelial neoplasia grade 2 (CIN2) or higher or cervical cancer detected as a result of the screening test performed at recruitment and of its assessment. Results of tests performed after a recommendation to repeat at the standard Italian screening interval (3 years) were not considered. We included lesions that were detected up to 1 year after the first referral to colposcopy.

Cytology

Liquid-based cytology was performed by using the ThinPrep system (Cytyc Corporation). One slide per woman was prepared according to the manufacturer's instructions. Both conventional cytology and liquid-based cytology were performed, without knowledge of HPV results, in the 14 laboratories (six in the Turin center) that routinely interpret cytology in regular screening programs. The same cytologists were assigned to liquid-based and conventional cytology. Abnormal slides were reviewed by a local supervisor (or, in Florence, by a panel of cytologists) before they reported the results to the women. This diagnosis was used both for the woman's management and for the study analysis. Two laboratories had no previous experience in interpreting liquid-based cytology, seven had previous experience with the ThinPrep system (500–10 000 slides read per laboratory), and five laboratories had previous experience with another liquid-based system (approximately 1000 slides read per laboratory). Cytologists from all laboratories attended a training course that was provided by the manufacturer before the start of the study.

Cytology was classified according to the Bethesda 1991 guidelines (TBS 1991). The ASCUS subcategories recommended in the TBS 1991 were not applied. We used the TBS 1991 guidelines to avoid problems due to a switch to the 2001 classification, which was introduced just before the start of the study.

All centers routinely conducted activities to improve the sensitivity and specificity of cytologists. These included monitoring the distribution of diagnoses and positive predictive values (PPVs) and the circulation of Pap smears and discussion within and among laboratories. These activities were continued during the study period. In addition, as an exercise to improve consistency between centers, a set of 30 liquid-based cytology slides that were difficult to classify was circulated to each center during the study. The slides were read blindly at each center, and slides with discrepant diagnoses were discussed among the representatives of each center.

HPV Testing

HPV testing was done, blind to cytology results, in seven laboratories by using the Hybrid Capture 2 assay (HC2; Digene Corporation, Gaithersburg, MD). Only the group of probes designed to detect high-risk HPV types 16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 59, and 68 was used. After preparation of one cytology slide, 4 mL of the remaining PreservCyt sample was processed with the Sample Conversion Kit (Digene) followed by HC2 assay, according to the manufacturer's instructions. When less than 4 mL of sample was available (for 310 women), women were recalled for another sample. HC2 results were expressed as the ratio of the specimen's light emission to that of three concurrently tested 1 pg/mL HPV DNA controls (Relative Light Units [RLU]). Therefore, RLU is an (indirect) measure of the specimen's viral concentration relative to 1 pg/mL. Before the start of the study, technicians in all centers attended a training course that was provided by Digene. A set of quality assurance procedures was implemented, including the use of controls from the manufacturer with known HPV DNA content and the circulation of clinical samples prepared by one of the participating laboratories. These procedures showed high accuracy and reproducibility of HPV detection—only three of 1024 target samples containing purified concentration-defined HPV DNA were incorrectly scored in a positive versus negative classification, and the multiple-rater k scores were 0.91, 0.60, and 0.69 for HPV negative, HPV low-positive, and HPV high-positive, respectively (21).

Colposcopy and Histology

The same colposcopists who had access to the participants' cytology and HPV status were used in the experimental and conventional arms of the study. Suspicious areas identified by colposcopy were biopsied. Women were referred for repeat colposcopy using routine protocols according to the colposcopist's judgment. The main reason for repeating the colposcopy was a lack of histology-confirmed CIN in the presence of clearly abnormal cytology. Histology was read locally by pathologists who were not blinded to cytology and HPV results but was reviewed centrally by investigators who were blinded to study arm, to the original diagnosis, and to the cytology and HPV results.

Histology Review

All specimens of women who had histologically determined CIN1+ within 1 year after referral to colposcopy were reviewed blindly and independently. For each woman, all histologic slides were provided together, and the most severe diagnosis was used. If the relevant material could not be retrieved (22 of 650 women), the most severe initial diagnosis was considered.

Each patient's specimens were reviewed blindly and independently by one or two pathologists. If a pathologist did not agree with the original diagnosis regarding the presence of CIN2+, the information was discussed by a group of pathologists and a consensus diagnosis was reached. This consensus diagnosis was then used in the analysis.

For most women who were diagnosed with CIN2/3 and for 12% of those with CIN1, each patient's specimens were independently reviewed by two pathologists who were randomly selected from a pool of nine, i.e., one per center. Consensus discussions included all nine pathologists when necessary. The remaining cases (88% of CIN1, with five of CIN2/3 also included to ensure that reviewers were blinded to the original diagnosis) were reviewed by one pathologist who was selected among the three most expert, and, if needed, consensus discussions included all three pathologists. Overall, considering all age groups, 4.2% of women with CIN1 were upgraded to CIN2 (and none was upgraded to CIN3), with no difference by method of review (P = .98); 11.3% of women with CIN2/3 were downgraded to CIN1 or to no CIN.

Statistical Analysis

Sensitivity and specificity of different combinations of HPV and liquid-based cytology were computed within the experimental arm. Exact confidence intervals were calculated. Only women who had a valid relevant test were included in the analyses. The sensitivity and specificity of different approaches among subjects with valid results for both tests were compared by the McNemar test (22).

The relative sensitivity of different combinations of liquid-based cytology and HPV versus conventional cytology was estimated using CIN2+ and CIN3+ detection rates relative to the conventional arm. All randomly assigned eligible women were included in the analyses. The relative PPV versus that of conventional cytology was computed by including only women who actually underwent colposcopy. If a center effect on relative sensitivity or relative PPV was present, the confidence interval was calculated by using a multilevel random effects model with random intercept and random slope (23). All P values are two-sided, and P<.05 was considered statistically significant.

Performing random colposcopies on a sample of women who were negative for both tests was judged as unfeasible for the expected strong negative impact on recruitment. Therefore, absolute sensitivity and specificity estimates are not corrected for verification bias.

The study size of the entire trial (that includes two phases, one with HPV plus liquid-based cytology and one with HPV only in the experimental arm) was determined to have sufficient statistical power to show a reduction in the detection rate of CIN2+ in the experimental versus conventional arms at the next screening examination. Assuming a loss to follow-up of approximately 30% and a detection rate of 3.0 CIN per 1000 women in the conventional arm, the target study size of approximately 100 000 women provides greater than 80% power to detect a 32% reduction of detection rate in the experimental versus conventional arms as statistically significant at the 5% level using a two-sided test.


    RESULTS
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
We randomly assigned 45 307 women in the first phase of recruitment (Fig. 1). Of them, 133 (81 from the conventional and 52 from the experimental arm) were excluded because they had been randomly assigned but were not eligible. Therefore, a total of 22 466 eligible women were randomly assigned to the conventional arm and 22 708 to the experimental arm.


Figure 1
View larger version (12K):
[in this window]
[in a new window]
 
Fig. 1. Trial profile. ASCUS = atypical squamous cells of undetermined significance; HPV = human papillomavirus; SIL = squamous intraepithelial lesion.

 
Further analyses are based on the 16 658 and 16 706 women aged 35 years and older who were randomly assigned to the conventional and experimental arm, respectively. Median age was 45 years in both arms (P = .97). More than half of the women in the conventional (53.0%) and the experimental arm (53.8%), respectively, had been screened for cervical cancer in an organized program within 4 years of enrollment (P = .14).

Women were managed according to the intended protocol, with a few exceptions due to local clinical decisions (Fig. 1). At least one colposcopy was received by 93.0% of referred women (90.8% in the conventional arm and 93.7% in the experimental arm). Among women who received colposcopy, the mean number of colposcopies per woman was 1.32 in the conventional and 1.21 in the experimental arm. Two of 51 women were diagnosed with CIN2+ as a result of repeat colposcopies in the conventional arm versus five of 75 in the experimental arm.

Liquid-Based Cytology and Conventional Cytology Results

A statistically significantly lower proportion of women in the experimental arm (liquid-based cytology) than in the conventional arm had at least one unsatisfactory Pap smear (n = 418 [2.5%] versus n = 620 [3.7%], P<.001). In particular, the frequency of women with unsatisfactory cytology because of obscuring inflammation was much lower in the experimental arm than in the conventional arm (n = 71 [0.42%] versus n = 325 [2.0%], P<.001), but the frequency of women with unsatisfactory cytology for other reasons was slightly higher in the experimental arm than in the conventional arm (n = 347 [2.1%] versus n = 295 [1.8%]).

Cytology results and reviewed histologic results by study arm are summarized in Table 2. More women in the experimental arm than in the conventional arm were classified as having ASCUS and LSIL cytology (for ASCUS, n = 553 [3.3%] versus n = 382 [2.3%], P<.001; for LSIL, n = 297 [1.8%] versus n = 170 [1.0%], P<.001). High-grade squamous intraepithelial lesion (HSIL) or more severe cytology was detected in 54 (0.32%) of women in the experimental arm and in 42 (0.25%) in the conventional arm (P = .23). Overall, 594 women in the conventional arm (3.6%) had ASCUS or more severe cytology versus 904 in the experimental arm (5.4%). The proportion of women with ASCUS+ cytology decreased with age in both arms but more steeply in the experimental arm (Fig. 2).


View this table:
[in this window]
[in a new window]
 
Table 2.  Cytology, human papillomavirus (HPV) status, and reviewed histologic outcome by study arm*

 

Figure 2
View larger version (7K):
[in this window]
[in a new window]
 
Fig. 2. The percentage of women who were human papillomavirus (HPV)-positive and with atypical squamous cells of undetermined significance or higher (ASCUS+) cytology by age group; point estimates and 95% confidence intervals are shown. HPV-positive (diamonds) and ASCUS+ cytology in the experimental arm (squares); ASCUS+ cytology in the conventional arm (triangles). Percent HPV-positive was determined for women with valid HPV tests (n = 16410). Percent ASCUS+ was determined for women with valid cytology who were randomly assigned to the conventional (n = 16 255) and experimental arms (n = 16 383), respectively.

 
HPV Positivity Rate

Overall, 7.1% of women in the experimental arm were HPV-positive (Table 2); the rate of HPV-positive women decreased with age (Fig. 2). Among women with normal cytology, 5.4% were HPV-positive at 1 pg/mL cut-off versus 24.1%, 42.1%, and 77.8% of women with ASCUS, LSIL, and HSIL+ cytology, respectively (Table 2). Overall, therefore, 10.6% of all women in the experimental arm were either HPV-positive or had ASCUS+ cytology, whereas 1.80% were both HPV-positive and had ASCUS+ cytology. The proportion of women who were HPV-positive was reduced to 5.45% using a cutoff of 2 pg/mL and to 3.45% using a cutoff of 10 pg/mL (Table 3).


View this table:
[in this window]
[in a new window]
 
Table 3.  Number of women in the experimental arm by human papillomavirus (HPV) value and histologic outcome*

 
Sensitivity and Specificity of HPV Versus Liquid-Based Cytology and Effect of Different Cutoffs for HPV Testing on Referral to Colposcopy

Overall, 75 women with CIN2 or more severe histology were identified in the experimental arm. Among them, 54 had ASCUS or more severe cytology and 73 were HPV-positive (Table 2). Among those who were HPV-positive, 68 had a RLU ≥10.0 and 72 had a RLU ≥2.0, whereas only one woman with CIN3 (none with CIN2) had a RLU between 1.0 and 2.0, thus a viral concentration between 1 and 2 pg/mL (Table 3).

HPV testing with a 1 pg/mL cutoff was more sensitive but less specific than liquid-based cytology with an ASCUS cutoff, considering both CIN2+ and CIN3+ histology endpoints. HPV testing with a cutoff of 2 pg/mL was also more sensitive for CIN2+ than ASCUS+ liquid-based cytology, with a similar specificity for both CIN2+ and CIN3+ (Table 4).


View this table:
[in this window]
[in a new window]
 
Table 4.  Sensitivity and specificity of liquid-based cytology and human papillomavirus (HPV) testing in the experimental arm*

 
Comparison of Different Strategies Versus Regular Practice With Conventional Cytology

In the conventional arm, 51 women with CIN2+ histology were identified (Table 2). When compared with conventional cytology (in which ASCUS was the cutoff for colposcopy), the relative sensitivity for CIN2+ histology in the experimental arm (either ASCUS+ from liquid-based cytology or HPV ≥1 pg/mL) was 1.47 (95% CI = 1.03 to 2.09) (Table 5). However, this resulted in a decrease of PPV from 11.4% to 4.5% (relative PPV = 0.40, 95% CI = 0.23 to 0.66). With HPV ≥1 pg/mL alone as a criterion for colposcopy, the sensitivity relative to conventional cytology was still increased (relative sensitivity = 1.43, 95% CI = 1.00 to 2.04), whereas PPV was 6.6% (relative PPV = 0.58, 95% CI = 0.33 to 0.98). With HPV ≥2 pg/mL alone as a criterion for colposcopy, an increase in sensitivity was suggested compared with conventional cytology (relative sensitivity = 1.41, 95% CI = 0.98 to 2.01), and PPV was 8.5% (relative PPV = 0.75, 95% CI = 0.45 to 1.27).


View this table:
[in this window]
[in a new window]
 
Table 5.  Detection rate, positive predictive value (PPV), relative sensitivity, and relative PPV for histology-confirmed CIN2+ and CIN3+ of different screening strategies in the experimental arm versus conventional cytology ≥ASCUS*

 
By using liquid-based cytology with ASCUS+ as the only cutoff for referral to colposcopy, the sensitivity was similar to that obtained in the conventional arm with conventional cytology (relative sensitivity = 1.06), but PPV was decreased to 6.5% (relative PPV = 0.57, 95% CI = 0.39 to 0.82). The relative sensitivity and relative PPV for CIN2+ of liquid-based ASCUS+ cytology versus conventional ASCUS+ cytology were 1.03 and 0.67, respectively, when the analysis was restricted to the seven centers that referred to colposcopy all women with ASCUS cytology in both arms. Relative sensitivity and PPV for HPV testing alone and for HPV testing with liquid-based cytology also changed only slightly when only these seven centers were considered (data not shown), and there was no evidence of heterogeneity in relative sensitivity and relative PPV between these centers and the centers that referred women in the conventional arm with ASCUS cytology for repeat testing.

No statistically significant variation of relative sensitivity and relative PPV by age group was observed. There was also no effect of center on relative sensitivity. However, there was heterogeneity between centers regarding the relative PPV versus conventional cytology of HPV at 1 pg/mL (P = .02); the same was true for HPV at 2 pg/mL (P = .028) and for liquid-based cytology with HPV (P = .042) but not for ASCUS+ liquid-based cytology alone (P = .13).

With CIN3+ histology as the endpoint, relative sensitivity and PPV results were in the same direction as with CIN2+. In particular, the relative sensitivity versus that in the conventional arm was 1.25 with either ASCUS+ liquid-based cytology or HPV >1 pg/mL as a criterion for colposcopy and was 1.22 with HPV ≥1 pg/mL alone, although none of these differences reached statistical significance.


    DISCUSSION
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
This study presents the first results from a randomized trial that compares conventional cytology and the combined use of HPV testing and liquid-based cytology. It also allowed HPV testing and liquid-based cytology to be compared in the same women. With respect to conventional cytology, HPV testing with a 1 pg/mL cutoff supplemented by liquid-based cytology led to a substantial (47%) increase in sensitivity for CIN2+ but also to a 60% loss in PPV. Increases in sensitivity were still obtained with HPV alone at 1 and 2 pg/mL cutoffs (43% and 41%, respectively, compared with conventional cytology), and these approaches reduced the losses in PPV (42% and 25%, respectively, compared with conventional cytology). Liquid-based cytology did not cause an increase in sensitivity compared with conventional cytology but led to a substantial (43%) reduction in PPV.

The study was conducted within organized screening programs on a large population, a situation very similar to routine application. Nearly 80% of eligible women were enrolled. Therefore, our results are representative of those that would be expected in routine application. The referral rate, detection rate of CIN2+, and PPV observed in the conventional arm were similar to those observed in the Italian organized cervical screening programs among women aged 25–64 years in 2000–2002 [2.7–3.0%, 2.8–3.0 per 1000, and 11.4–15.4%, respectively (24)]. The detection rate (and consequently PPV) was low compared with that in other countries, even considering the age distribution, reflecting the low risk of cervical neoplasms in Italy (25). A very low incidence of interval cancers was previously observed during the regular screening activity of the largest center (26), strongly suggesting that a low detection rate does not result from failure to identify progressive lesions. Individual randomization of this large population guarantees that a difference in the detection rate of histologically confirmed lesions between arms can be attributed to test sensitivity and not to a difference in baseline risk of the populations.

Liquid-based and conventional cytology showed similar sensitivity, but the PPV was greatly reduced with liquid-based cytology, even when restricting the analysis to the centers that used the same criteria of referral to colposcopy with both methods. This reduction was the result of a larger proportion of women being classified as having ASCUS+ cytology with liquid-based cytology but of no increase in the detection of histology-confirmed lesions. In previous studies without split sampling (direct-to-vial), a higher proportion of LSIL was generally reported with liquid-based cytology than with conventional cytology (16,17,2733), whereas an increase in HSIL cytology was frequently (16,2731) but not always (19,32) reported. Regarding ASCUS, some studies found lower detection rates (16,2729,34) whereas others found higher rates (3133,35). The few studies, each with less than 10 000 women, which computed relative sensitivity for biopsy-proven high-grade CIN in a primary screening setting provided conflicting results (1518). In most of these studies (15,16,18), the false-positive rate was higher with liquid-based cytology than with conventional cytology. The main advantage of liquid-based cytology was an overall reduction of unsatisfactory slides, which has been consistently observed in previous direct-to-vial studies (16,2833,35).

HPV testing for high-risk types was more sensitive than both conventional (by approximately 40%) and liquid-based (by approximately 30%) cytology. This was true both with CIN2+ and with CIN3+, the more proximal precursor of cancer, as the endpoint. Previous studies comparing HPV testing with conventional (19,1113) and liquid-based (4,10) cytology consistently found higher sensitivity with HPV testing. Many of them, however, were based on self-referred women (2,4,5,8,10,12,13). In addition, some were conducted in populations at high risk (6,7,9) or women who were younger than the overall spectrum of the screened population (1,5,10). Results of these previous studies are consistent with our findings in a large, population-based randomized study addressing a low-risk population.

Only two of the 75 women in the experimental arm with CIN2+ were HPV-negative, and little was gained by performing liquid-based cytology on every woman. However, doing this reduced specificity and led to many unnecessary referrals. Thus, our data strongly suggest that supplementing HPV testing with cytology provides little advantage and mainly increases costs and anxiety. Based on these results, a second phase of the trial, in which HPV testing alone is compared with conventional cytology is now being undertaken.

Moreover, no high-grade lesion was found among 845 HPV-negative women with ASCUS cytology. Therefore, this result strongly supports the use of HPV in triaging ASCUS when cytology is performed first, as previously reported (11,3638).

HPV testing alone using a 1 pg/mL cutoff reduced PPV by approximately 40% compared with conventional cytology (Table 5). Using a 2 pg/mL cutoff provided almost the same sensitivity but strongly reduced the number of women to be referred for colposcopy. The 2 pg/mL cutoff led to a PPV only 25% lower than that obtained with conventional cytology, suggesting this is more appropriate for population screening at least when liquid-based cytology samples are used. A similar result was obtained in the HPV in Addition to Routine Testing (HART) study (11). However, in a high-risk population in Costa Rica the optimal cutoff was 1 pg/mL (7). It should, however, be noted that relative PPV for HPV versus conventional cytology varied statistically significantly between centers. An explanation could be different progression from infection to high-grade CIN, possibly depending on a different HPV type and variant mix in different centers.

The study has a few potential limitations. Because colposcopies were performed only on women who were positive for either test, estimates of absolute values of sensitivity and specificity could not be corrected for verification bias. In addition, although cytologists were blinded to HPV results, they knew that women having liquid-based cytology were also tested for HPV. This situation could have unconsciously caused the application of broader criteria of interpretation.

The HART study (11) showed that surveillance of HPV-positive but cytologically normal women by repeat HPV testing and cytology at 12 months led to the same detection rates of CIN2+ as direct referral for colposcopy. A possible strategy is, therefore, to test women for HPV, triage by cytology those who are positive, and directly refer women who are positive for both tests for colposcopy. Women who are HPV-positive but cytologically normal could repeat tests at 1 year. In our study, 1.80% of women in the experimental arm were positive both for high-risk HPV and had ASCUS+ cytology. Another 5.29% were HPV-positive but cytologically normal. Assuming a clearance of HPV after 1 year to be approximately 50%, as observed in most published studies (11,39), the overall referral rate would be approximately 4.5%, compared with 3% in the conventional arm. When a cutoff of 2 pg/mL is used, the same strategy would lead to a referral rate of approximately 3.5%. However, a high rate of compliance to repeat testing is essential for this approach to work, and immediate referral should be considered for HPV-positive women unlikely to comply with repeat testing.

The follow-up phase of this study will compare the detection rate of CIN2+ after recruitment and up to and including the next regular screening invitation at 3 years in the two arms. Women with normal cytology in the conventional arm will be compared with those negative for both tests in the experimental arm. A very low detection rate in the experimental arm would prove that longer intervals could be used with HPV testing. The study has adequate power to investigate a reduction of CIN2+ at follow-up, but it is also the pilot study for a larger proposed study to examine reduction in cancer incidence as an endpoint.


    NOTES
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 
This work was supported by the European Union (Europe against Cancer contracts SI.2.327046 and SPC.2002475), by the Italian Ministry of Health (Progetto Speciale "Valutazione di nuove tecnologie per lo screening del cervicocarcinoma"), by the Special Project "Oncology," Compagnia di S. Paolo FIRMS, and by the participating Italian regions. The sponsors had no role in the study design, the collection of the data, analysis, interpretation of the results, or the writing of the manuscript.

We thank all the staff who assisted in running the study. We also thank the thousands of women who have participated into this study.

The following are contributing members of the NTCC Working Group: Turin: R. Rizzolo and D. Mari (CPO Piemonte), L. De Marco (Unit of Cancer Epidemiology and CPO, CERMS, University of Turin), B. Ghiringhello and F. Parisio (Unit of Pathology, OIRM S. Anna), R. Volante (Centre for Early Cancer Diagnosis and Treatment, OIRM S. Anna), E. Berardengo (Unit of Pathology, Ospedale S. Giovanni AS), A. Andrion (Unit of Pathology, Ospedale Martini), S. Coverlizza (Unit of Pathology, Ospedale Giovanni Bosco), S. Taraglio (Unit of Pathology, Ospedale Maria Vittoria), M. G. Accinellli (Unit of Pathology,Universityof Turin); Trento: E. Polla, A. Pojer, S. Girlando, and D. Aldovini (Unit of Pathology, Ospedale di Trento); Veneto: M. Vettorazzi (Venetian Tumour Registry, Azienda Ospedaliera di Padova), D. Minucci, and M. Matteucci (Unity of Ginaecology, Azienda Ospedaliera di Padova), L. Onnis and E. Insacco (Department of Pathology, University of Padua), M. Lestani (Department of Pathology, University of Verona) and A. Vignato (Servizio di Citologia, Ostetricia e Ginecologia, Ospedale di Soave); Emilia-Romagna: M. Manfredi (Centro screening, AUSL Bologna), P. Pierotti (Unit of Pathology, Ospedale Maggiore, AUSL Bologna), G. Collina (Unit of Pathology Ospedale Bellaria, AUSL Bologna) M. Serafini (Centro Prevenzione Oncologica, AUSL Ravenna), C. Sintoni (Unit of Pathology, Presidio Ospedaliero di Ravenna, AUSL Ravenna), M. Aldi (Unit of Pathology, Presidio Ospedaliero di Faenza, AUSL Ravenna), A. Bondi and G. Galanti (Unit of Pathology, Presidio Ospedaliero, AUSL di Imola); Florence: A. Iossa, S. Ciatto, M. P. Cariaggi, S. Cecchini, C. Sani (CSPO Firenze), and G. L. Taddei (Unit of Pathology University of Florence); Lazio: S. Brezzi, P. Raggi, and E. Gomes (Local Health Unit, Viterbo, Italy), A. Pellegrini and M. L. Schiboni (Ospedale S. Giovanni, Rome).


    REFERENCES
 Top
 Notes
 Abstract
 Introduction
 Subjects and methods
 Results
 Discussion
 References
 

(1) Cuzick J, Szarewski A, Terry G, Ho L, Hanby A, Maddox P, et al. Human papillomavirus testing in primary cervical screening. Lancet 1995;345:1533–6.[CrossRef][ISI][Medline]

(2) Clavel C, Masure M, Bory JP, Putaud I, Mangeonjean C, Lorenzato M, et al. Hybrid Capture II-based human papillomavirus detection, a sensitive test to detect in routine high-grade cervical lesions: a preliminary study on 1518 women. Br J Cancer 1999;80:1306–11.[CrossRef][ISI][Medline]

(3) Cuzick J, Beverley E, Ho L, Terry G, Sapper H, Mielzynska I, et al. HPV testing in primary screening of older women. Br J Cancer 1999;81:554–8.[CrossRef][ISI][Medline]

(4) Clavel C, Masure M, Bory JP, Putaud I, Mangeonjean C, Lorenzato M, et al. Human papillomavirus testing in primary screening for the detection of high-grade cervical lesions: a study of 7932 women. Br J Cancer 2001;84:1616–23.[CrossRef][ISI][Medline]

(5) Ratnam S, Franco EL, Ferenczy A. Human papillomavirus testing for primary screening of cervical cancer precursors. Cancer Epidemiol Biomarkers Prev 2000;9:945–51.[Abstract/Free Full Text]

(6) Kuhn L, Denny L, Pollack A, Lorincz A, Richart RM, Wright TC. Human papillomavirus DNA testing for cervical cancer screening in low-resource settings. J Natl Cancer Inst 2000;92:818–25.[Abstract/Free Full Text]

(7) Schiffman M, Herrero R, Hildesheim A, Sherman ME, Bratti M, Wacholder S, et al. HPV DNA testing in cervical cancer screening: results from women in a high-risk province of Costa Rica. JAMA 2000;283:87–93.[Abstract/Free Full Text]

(8) Schneider A, Hoyer H, Lotz B, Leistritza S, Kuhne-Heid R, Nindl I, et al. Screening for high-grade cervical intra-epithelial neoplasia and cancer by testing for high-risk HPV, routine cytology or colposcopy. Int J Cancer 2000;89:529–34.[CrossRef][ISI][Medline]

(9) Denny L, Kuhn L, Pollack A, Wainwright H, Wright TC. Evaluation of alternative methods of cervical cancer screening for resource poor settings. Cancer 2000;89:826–33.[CrossRef][ISI][Medline]

(10) Kulasingam SL, Hughes JP, Kiviat NB, Mao C, Weiss NS, Kuypers JM, et al. Evaluation of human papillomavirus testing in primary screening for cervical abnormalities: comparison of sensitivity, specificity, and frequency of referral. JAMA 2002;288:1749–57.[Abstract/Free Full Text]

(11) Cuzick J, Szarewski A, Cubie H, Hulman G, Kitchener H, Luesley D, et al. Management of women who test positive for high-risk types of human papillomavirus: the HART study. Lancet 2003;362:1871–6.[CrossRef][ISI][Medline]

(12) Petry KU, Menton S, Menton M, van Loenen-Frosch F, de Carvalho Gomes H, et al. Inclusion of HPV testing in routine cervical cancer screening for women above 29 years in Germany: results for 8466 patients. Br J Cancer 2003;88:1570–7.[CrossRef][ISI][Medline]

(13) Salmeron J, Lazcano-Ponce E, Lorincz A, Hernandez M, Hernandez P, Leyva A, et al. Comparison of HPV-based assays with Papanicolaou smears for cervical cancer screening in Morelos State, Mexico. Cancer Causes Control 2003;14:505–12.[CrossRef][Medline]

(14) IARC Working Group on the Evaluation of Cancer Preventive Strategies. Cervix cancer screening. IARC Handbooks of Cancer Prevention No 10. Lyon (France): IARC Press; 2005.

(15) Hutchinson ML, Zahniser DJ, Sherman ME, Herrero R, Alfaro M, Bratti M, et al. Utility of liquid-based cytology for cervical carcinoma screening: results of a population-based study conducted in a region of Costa Rica with a high incidence of cervical carcinoma. Cancer 1999;87:48–55.[CrossRef][ISI][Medline]

(16) Carpenter AB, Davey DD. ThinPrep Pap Test: performance and biopsy follow-up in a university hospital. Cancer 1999;87:105–12.[Medline]

(17) Baker JJ. Conventional and liquid-based cervicovaginal cytology: a comparison study with clinical and histologic follow-up. Diagn Cytopathol 2002;27:185–8.[Medline]

(18) Coste J, Cochand-Priollet B, De Cremoux P, Le Glès C, Cartier I, Molinié V, et al. Cross sectional study of conventional cervical smear, monolayer cytology, and human papillomavirus DNA testing for cervical cancer screening. BMJ 2003;326:733.[Abstract/Free Full Text]

(19) Obwegeser JH, Brack S. Does liquid-based technology really improve detection of cervical neoplasia? A prospective, randomized trial comparing the ThinPrep Pap Test with the conventional Pap test, including follow-up of HSIL cases. Acta Cytol 2001;45:709–14.[ISI][Medline]

(20) Ronco G, Ghisetti V, Segnan N, Snijders PFJ, Gillio-Tos A, Meijer CJLM, et al. Prevalence of human papillomavirus infection in women in Turin, Italy. Eur J Cancer 2005;41:297–305.[CrossRef][Medline]

(21) Carozzi F, Del Mistro A, Confortini M, Sani C, Puliti D, Trevisan R, et al. Reproducibility of HPV DNA testing by Hybrid Capture 2 in a screening setting: intralaboratory and interlaboratory quality control in seven laboratories participating in the same clinical trial. Am J Clin Pathol 2005;124:716–21.[Medline]

(22) Agresti A. Categorical data analysis. New York (NY): Wiley, 1990.

(23) Rabe-Heskett S, Skrondal A, Pickles A. Reliable estimation of generalised linear mixed models using adaptive quadrature. Stata J 2001;2:1–21.

(24) Ronco G, Giubilato P, Naldoni C, Zorzi M, Anghinoni E, Scalisi A, et al. Level of activation and process indicators of organized cervical cancer screening programs in Italy [in Italian]. Osservatorio Nazionale per la Prevenzione dei Tumori Femminili. 2004 Terzo rapporto.

(25) Parkin DM, Whelan SL, Ferlay J, Teppo L, Thomas DB, editors. Cancer incidence in five continents Vol VII. IARC Scientific Publications No 155. Lyon (France): IARC Press; 2003.

(26) Ronco G, Pilutti S, Patriarca S, Montanari G, Ghiringhello B, Volante R, et al. Impact of the introduction of organised screening for cervical cancer in Turin, Italy: cancer incidence by screening history 1992–98. Br J Cancer 2005;93:376–8.[CrossRef][ISI][Medline]

(27) Diaz-Rosario L, Kabawat SE. Performance of a fluid-based, thin-layer Papanicolaou smear method in the clinical setting of a independent laboratory and an out-patient screening population in New England. Arch Pathol Lab Med 1999;123:817–21.[ISI][Medline]

(28) Guidos BJ, Selvaggi SM. Use of ThinPrep Pap Test in clinical practice. Diagn Cytopathol 1999;20:70–3.[CrossRef][Medline]

(29) Vassilakos P, Saurel J, Rondez R. Direct-to-vial use of the AutoCyte PREP liquid-based preparation for cervical-vaginal specimens in three European laboratories. Acta Cytol 1998;43:65–8.

(30) Tench W. Preliminary assessment of the AutoCyte PREP: direct-to-vial performance. J Reprod Med 2000;45:912–6.[Medline]

(31) Weintraub J, Morabia A. Efficacy of a liquid-based thin layer method for cervical cancer screening in a population with a low incidence of cerviacal cancer. Diagn Cytopathol 2000;22:52–9.[CrossRef][ISI][Medline]

(32) Cheung ANY, Szeto EF, Leung BSY, Khoo US, Ng AWY. Liquid-based cytology and conventional smears: a comparison study in Asian screening population. Cancer 2003;99:331–5.[Medline]

(33) Colgan TJ, Machlin M, Collerchio M, Howlett R, Thompson F, Seidenfeld A, Mai V. Results of the implementation of liquid-based cytology—SurePath in the Ontario Screening Program. Cancer 2004;102:362–7.[Medline]

(34) Confortini M, Bulgaresi P, Cariaggi MP, Carozzi FM, Cecchini S, Cipparone I, et al. Comparing conventional and liquid-based smears from a consecutive series of 297 subjects referred to colposcopy assessment. Cytopathology 2004;15:168–70.[Medline]

(35) Fremont-Smith M, Marino J, Griffin B, Spencer L, Bolick D. Comparison on the SurePath liquid-based Papanicolau smear with the conventional Papanicolaou smear in a multisite direct-to-vial study. Cancer 2004;102:269–79.[Medline]

(36) Manos MM, Kinney WK, Hurely LB, Sherman ME, Shieh-Ngai J, Kurman RJ, et al. Identifying women with cervical neoplasia: using human papillomavirus DNA testing for equivocal Papanicolaou results. JAMA 1999;281:1605–10.[Abstract/Free Full Text]

(37) Solomon D, Schiffman M, Tarone R; ALTS Study Group. Comparison of three management strategies for patients with atypical squamous cells of undetermined significance: baseline results from a randomized trial. J Natl Cancer Inst 2001;93:293–9.[Abstract/Free Full Text]

(38) Arbyn M, Buntinx F, Van Ranst M, Paraskevaidis E, Martin-Hirsch P, et al. Virologic versus cytologic triage of women with equivocal Pap smears: a meta-analysis of the accuracy to detect high-grade intraepithelial neoplasia. J Natl Cancer Inst 2004;96:280–93.[Abstract/Free Full Text]

(39) Cuzick J, Sasieni P, Davies P, Adams J, Normand C, Frater A, et al. A systematic review of the role of human papillomavirus testing within a cervical screening programme. Health Technol Assess 1999;3:i–iv, 1–196.[Medline]

Manuscript received June 17, 2005; revised March 15, 2006; accepted April 14, 2006.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?


This article has been cited by other articles:


Home page
Cancer Epidemiol. Biomarkers Prev.Home page
P. E. Gravitt, M. Schiffman, D. Solomon, C. M. Wheeler, and P. E. Castle
A Comparison of Linear Array and Hybrid Capture 2 for Detection of Carcinogenic Human Papillomavirus and Cervical Precancer in ASCUS-LSIL Triage Study
Cancer Epidemiol. Biomarkers Prev., May 1, 2008; 17(5): 1248 - 1254.
[Abstract] [Full Text] [PDF]


Home page
J Am Board Fam MedHome page
J. E. Wilkinson and M. C. Cerreto
Primary Care for Women with Intellectual Disabilities
J Am Board Fam Med, May 1, 2008; 21(3): 215 - 222.
[Abstract] [Full Text] [PDF]


Home page
JNCI J Natl Cancer InstHome page
G. Ronco, P. Giorgi-Rossi, F. Carozzi, M. Confortini, P. D. Palma, A. Del Mistro, A. Gillio-Tos, D. Minucci, C. Naldoni, R. Rizzolo, et al.
Results at Recruitment From a Randomized Controlled Trial Comparing Human Papillomavirus Testing Alone With Conventional Cytology as the Primary Cervical Cancer Screening Test
J Natl Cancer Inst, April 2, 2008; 100(7): 492 - 501.
[Abstract] [Full Text] [PDF]


Home page
ANN INTERN MEDHome page
G. F. Sawaya
Adding Human Papillomavirus Testing to Cytology for Primary Cervical Cancer Screening: Shooting First and Asking Questions Later
Ann Intern Med, April 1, 2008; 148(7): 557 - 559.
[Full Text] [PDF]


Home page
Obstet GynecolHome page
Evidence-Based Medicine Versus Liquid-Based Cytology
Obstet. Gynecol., January 1, 2008; 111(1): 2 - 3.
[Full Text] [PDF]


Home page
J. Clin. Microbiol.Home page
P. E. Castle, P. E. Gravitt, D. Solomon, C. M. Wheeler, and M. Schiffman
Comparison of Linear Array and Line Blot Assay for Detection of Human Papillomavirus and Diagnosis of Cervical Precancer and Cancer in the Atypical Squamous Cell of Undetermined Significance and Low-Grade Squamous Intraepithelial Lesion Triage Study
J. Clin. Microbiol., January 1, 2008; 46(1): 109 - 117.
[Abstract] [Full Text] [PDF]


Home page
NEJMHome page
M.-H. Mayrand, E. Duarte-Franco, I. Rodrigues, S. D. Walter, J. Hanley, A. Ferenczy, S. Ratnam, F. Coutlee, E. L. Franco, and the Canadian Cervical Cancer Screening Trial Study
Human Papillomavirus DNA versus Papanicolaou Screening Tests for Cervical Cancer
N. Engl. J. Med., October 18, 2007; 357(16): 1579 - 1588.
[Abstract] [Full Text] [PDF]


Home page
NEJMHome page
P. Naucler, W. Ryd, S. Tornberg, A. Strand, G. Wadell, K. Elfgren, T. Radberg, B. Strander, O. Forslund, B.-G. Hansson, et al.
Human Papillomavirus and Papanicolaou Tests to Screen for Cervical Cancer
N. Engl. J. Med., October 18, 2007; 357(16): 1589 - 1597.
[Abstract] [Full Text] [PDF]


Home page
BMJHome page
G. Ronco, J. Cuzick, P. Pierotti, M. P. Cariaggi, P. D. Palma, C. Naldoni, B. Ghiringhello, P. Giorgi-Rossi, D. Minucci, F. Parisio, et al.
Accuracy of liquid based versus conventional cytology: overall results of new technologies for cervical cancer screening: randomised controlled trial
BMJ, July 7, 2007; 335(7609): 28 - 28.
[Abstract] [Full Text] [PDF]


Home page
J. Clin. Microbiol.Home page
F. Carozzi, S. Bisanzi, C. Sani, M. Zappa, S. Cecchini, S. Ciatto, and M. Confortini
Agreement between the AMPLICOR Human Papillomavirus Test and the Hybrid Capture 2 Assay in Detection of High-Risk Human Papillomavirus and Diagnosis of Biopsy-Confirmed High-Grade Cervical Disease
J. Clin. Microbiol., February 1, 2007; 45(2): 364 - 369.
[Abstract] [Full Text] [PDF]


This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in: