QJM Advance Access originally published online on December 17, 2006
QJM 2007 100(1):29-35; doi:10.1093/qjmed/hcl121
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The predictive value of ultrasound findings in the management of thyroid nodules
Department of Medical and Surgical Sciences, Internal Medicine and Endocrinology Unit, University of Brescia, Brescia, Italy
Address correspondence to Dr C. Cappelli, Department of Medical and Surgical Sciences, Internal Medicine and Endocrinology Unit, University of Brescia, c/o 2 Medicina Spedali Civili di Brescia, Piazzale Spedali Civili no. 1, 25100 Brescia, Italy. e-mail: cappelli{at}unibs.med.it
Received 7 April 2006 and in revised form 12 July 2006
 |
Summary |
|---|
 Top Summary IntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
Background: Thyroid nodules are a common medical problem, but whether fine-needle aspiration cytology (FNAC) is mandated for smaller non-palpable nodules, is controversial.
Aim: To evaluate whether ultrasonographic features of thyroid nodules are associated with histological malignancy, and to identify useful criteria for clinical decision-making.
Design: Prospective observational study.
Methods: From January 1991 to September 2004, 5198 patients were referred to our hospital for ultrasound evaluation of thyroid nodules. Overall, 7455 nodules (diameters from 6 to 100 mm) were examined; 2865 (38.4%) were <10 mm in diameter. All patients with suspicious or malignant cytology underwent surgery.
Results: Of the 349 nodules undergoing surgery, 284 were malignant. The prevalence of cancer was slightly higher in nodules 
10 mm in diameter vs. those <10 mm (5.5% vs. 3.0%, p < 0.01). However, histological type and local aggressiveness were largely independent of nodule size. Microcalcifications (72.2% vs. 28.7%; p < 0.001; OR 6.4, 95%CI 4.9–8.4), blurred margins (52.8% vs. 18.8%; p < 0.001; OR 4.8, 95%CI 3.8–6.1), solid hypoechoic appearance (80.6% vs. 52.4%; p < 0.001; OR 3.8, 95%CI 2.8–5.1), size 10 mm (77.4% vs. 64.9%; p < 0.001; OR 1.9, 95%CI 1.4–2.5) and intranodular vascular pattern type 2 (61.6% vs. 49.7%; p < 0.001; OR 1.6, 95%CI 1.2–2.0) were all significantly more frequent in malignant than in benign nodules. These associations were similar when large and small nodules were analysed separately.
Discussion: No single parameter, including nodule size, satisfactorily identifies a subset of patients to be electively investigated by FNAC, although several may be useful in this regard.
|
Introduction |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
Thyroid nodules are a common medical problem.1 Although they are traditionally found as palpable masses at neck examination in patients with or without suspected thyroid disease, the apparent prevalence of non-palpable thyroid nodules (i.e. <1 cm in diameter) in the general population has recently increased, probably as a consequence of the increasing application of ultrasound.2–5 One or more additional non-palpable thyroid nodules may be found in about 50% of patients with a clinically palpable solitary nodule, but they are also incidentally detected by imaging studies performed for various reasons.5
The prevalence of such lesions appears to be high in the general population and to increase in areas with iodine deficiency; some authors report that up to 70% of nodules of <1 cm in size are found among individuals with no history of thyroid disease.6 In addition, nodular thyroid disease is more common in the elderly, a population subgroup that is steadily increasing.7
Fine-needle aspiration cytology (FNAC) is considered the most reliable test for the diagnosis of thyroid nodules.8 FNAC is recommended for palpable nodules, but the indication for this procedure in non-palpable nodules is a matter of controversy;5 some clinicians recommend ultrasonography-guided FNAC (US-FNAC),9,10 whereas others consider that a clinical follow-up (neck palpation) is sufficient in absence of history of familiar thyroid cancer or head/neck irradiation.5
Such uncertainty mostly results from our limited knowledge of the prevalence of cancer in non-palpable nodules (so-called ‘microcarcinomas‘),9,11 and their clinical behaviour. Although in most reports incidentally-discovered carcinomas of the thyroid gland follow an indolent course,12,13 in a few, a significant number of microcarcinomas are associated with extracapsular growth and nodal or distant metastases.14–16
We aimed to evaluate the predictive value of ultrasound findings in the diagnosis of malignant thyroid nodules, and to identify features that might be useful for making practical decisions about their management.
|
Methods |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
From January 1991 to September 2004, 5198 patients with thyroid nodules were referred for US-FNAC in our hospital. Overall, 7455 nodules were examined, of which 2865 (38.4%) were <1 cm in diameter.
The nodules had been diagnosed by ultrasound investigation of the neck region performed for various reasons (Table 1). No patient included in this series had a previous diagnosis of thyroid malignancy. The study was conducted according to the principles of the Helsinki declaration and the guidelines of the institutional ethical committee.
|
Ultrasound investigations used an ultrasonographic scanner (Siemens Elegra or ATL 5000) equipped with a 10–12 MHz linear transducer for morphological studies and a 4.7 MHz transducer for colour-Doppler evaluation. Examinations were conducted and recorded by two skilled sonographers according to a standard procedure. The following ultrasound parameters were assessed in all nodules: (i) nodule size (maximum diameter as evaluated by sagittal and transverse scans); (ii) echographic structure (solid, mixed or cystic); (iii) echogenicity (iso-, hyper- or hypoechoic); (iv) presence/absence of calcification; (v) lesion margins well-defined or blurred; (vi) vascular pattern (along the maximum diameter of the nodule: type 0, absence of flow signals; type 1, vascular images in peripheral position; type 2, intranodular flow with multiple vascular images).
Totally echo-free nodules (anechoic) were assumed to be cystic, and were not considered further in this study; complex (cystic and solid) nodules were assumed to be mixed nodules, and FNAC sampling was directed to the solid portion of the lesion. FNAC was performed in all patients with solitary nodules with diameter 6 mm, while in patients with more than one nodule, FNAC was performed on all nodules 10 mm in diameter and/or up to four nodules of 6–10 mm each. Two skilled endocrinologists performed all the FNAC using 25-gauge needles and capillarity action, using a freehand technique. Cytological specimens were smeared according to the Papanicolaou technique, and evaluated by experienced cytopathologists. When the smear was inadequate (<6 clusters with 10 cells each),17 FNAC was repeated once; only technically satisfactory results were considered.
Adequate cytological material was classified as benign (colloid nodules, lymphocytic thyroiditis, cystic goitres), malignant (papillary carcinoma, medullary carcinoma, anaplastic carcinoma) or suspicious (including follicular or Hurthle cell neoplasms). Cases with benign cytology (or repeated inadequate smears) underwent clinical and ultrasound control after 6 months; in case of nodule growth and/or changes in ultrasound features, a second (third) US-FNAC was performed to rule out overlooked malignancies. All patients with suspicious or malignant cytology underwent surgery. Total thyroidectomy with removal of level VI lymph nodes was performed on all patients when a histological diagnosis of cancer was confirmed; in the presence of nodal metastasis, the surgeon performed a radical lymph node dissection ipsilateral to the nodule. The surgical samples were cut before fixation for macroscopic examination. Evaluation used multiple 5 mm slices of the whole thyroid gland to accurately define the dimensions of malignant lesions and their multifocality. The diagnosis of cancer was based on the final histological diagnosis on surgical specimens and carcinomas were staged according to TNM (Tumour, Node, Metastasis) classification.18 Neoplastic lesions outside the nodule examined by FNAC were considered incidental and not included in the analysis.
Statistical analysis
The size of the study was powered to detect an odds ratio of 1.5 in the prevalence of malignant lesions between large and small thyroid nodules (diameter 10 vs. <10 mm). A total of 5600 nodules (with a ratio of large to small nodules of 2:1, according to literature data) was requested for a two-group 
2 test with a p = 0.05 two-sided significance level, to achieve 80% statistical power.
Clinical, ultrasound, cytological and histological findings were separately recorded and blind-processed for statistical evaluation. Comparison of frequency distributions used the 2 test. Univariate and multivariate (logistic regression analysis) odds ratios (OR) with 95%CIs were calculated to assess the relationships between ultrasound criteria and histological outcomes.
The diagnostic value of ultrasound criteria was also assessed in terms of sensitivity, specificity, likelihood ratio, positive/negative predictive value and efficiency.19
|
Results |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
After a second round of FNAC, 703 patients (1320/7455 nodules, 17.7%) still had inadequate cytological specimens, and were excluded from further analysis. Inadequate samples were significantly more common for ‘small nodules’ (<10 mm in diameter) than for 'large nodules' (diameter
10 mm) (26.1% vs. 12.5%, p < 0.001). Thus a total of 6135 nodules with valid cytological specimens were obtained from 4495 patients (3118 females, 1377 males). A single nodule was examined in 3144 patients; two, three, four or five were biopsied in 1118, 182, 46 and 5 patients, respectively. Nodule size (as evaluated by ultrasound) ranged from 6 to 100 mm (mean ± SD 15.5 ± 9.0 mm); 4015 were classified as large nodules and the remaining 2120 as small.
Cytological and histological features
FNAC showed suspicious or overtly malignant results in 349 nodules (5.7%); all underwent surgery. One additional nodule with benign cytology was surgically removed because of a size increase over a six-month follow-up period. A diagnosis of carcinoma was histologically confirmed in 284 (4.6%): 242 papillary, 37 follicular and 5 medullary. The remaining 66 were all adenomatous, except the one with benign cytology, which showed only hyperplasia. The prevalence of cancer was lower in small vs. large nodules (3.0% vs. 5.5%, p < 0.001). However, follicular and medullary histotypes were more common in small than in large nodules (20.8 and 1.9% vs. 6.7 and 1.2%, respectively, p = 0.01).
Malignant features at histological examination were slightly more frequent in nodules from multinodular goitres (147/2770, 5.3%) than in solitary nodules (137/3365, 4.1%) at univariate analysis, with borderline statistical significance (p = 0.022). Neoplastic lesions were not related to sex (5.0% in males, 4.5% in females; p = NS).
Ultrasound investigations
The relationships between ultrasound findings and cytological and histological features are shown in Table 2. Microcalcifications were more common in histologically-confirmed malignant lesions than in benign nodules (72.2% vs. 28.7%; p < 0.001; OR 6.4, 95%CI 4.9–8.4), as were blurred margins (52.8% vs. 18.8%; p < 0.001; OR 4.8, 95%CI 3.8–6.1), solid hypoechoic appearance (80.6% vs. 52.4%; p < 0.001; OR 3.8, 95%CI 2.8–5.1), nodule size 10 mm (77.4% vs. 64.9%; p < 0.001; OR 1.9, 95%CI 1.4–2.5) and intranodular vascular pattern type 2 (61.6% vs. 49.7%; p < 0.001; OR 1.6, 95%CI 1.2–2.0). Under logistic regression analysis, size and all the above ultrasound criteria were significantly associated with a histological diagnosis of malignancy (Table 3). These associations of ultrasound features with malignancy remained similar when large and small nodules were analysed separately. The analysis also showed that age, gender and number of lesions were not significant predictors of cancer.
|
|
The predictive value of ultrasound features is summarized in Table 4. No single parameter, including nodule size, satisfactorily identified a subset of patients to be electively investigated by FNAC; some 9–22% of histologically confirmed carcinomas would be missed, depending on the parameter.
|
According to our results, the best compromise between risk of missing carcinomas of potential clinical aggressiveness and the need to avoid unnecessary procedures is probably to use FNAC on nodules with at least one of: microcalcifications, blurred margins or hypoechoic pattern.
TNM staging
Local extension of thyroid neoplasia with extracapsular growth (pT4 at TNM staging) was present in 26/284 carcinomas (9.2%), whereas nodal involvement was found in 67 (23.6%). Preliminary ultrasound evaluation predicted extracapsular growth in 7/26 carcinomas (26.9%) and nodal involvement in 7/67 (10.4%). Surprisingly, the size of carcinoma nodules was not significantly associated with the occurrence of extracapsular growth (large nodules 10.5%, small nodules 4.9%; p = NS) or lymph-node metastasis (large nodules 23.6%, small nodules 25.0%; p = NS).
|
Discussion |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
Prevalence and clinical relevance of cancer lesions according to nodule size
The overall percentage of malignancy in our series is lower (4.6%) than previously reported in studies of smaller size, but similar to that recently observed by Lin et al.20 in a very large series of >20 000 patients (3.6%). This suggests that we have not missed very many malignant nodules.
The frequency of histologically malignant lesions was slightly less in small than in large nodules, but clinically relevant in absolute terms. Follicular and medullary carcinomas, (histological types with a less favourable prognosis) were more frequent in small nodules. Extracapsular growth and nodal involvement at surgical staging were similar in small and large nodules.
The discrepancy between the low prevalence of clinically evident thyroid neoplasia compared with the high rate of cancers under 10–15 mm at autopsy21 prompts many physicians to believe that in most cases occult thyroid cancer has no clinical relevance. In addition, several reports have suggested that incidentally discovered thyroid papillary carcinomas follow an indolent course.12,13,16,22 On the other hand, other studies support our finding that evidence of extrathyroidal spread and/or nodal metastasis is equally common in large and small nodules,14–16 and that no valid dimensional cut-off for neoplastic aggressive behaviour can be identified.
Ultrasonographic findings as predictors of nodule malignancy
Many studies have investigated whether the ultrasonographic characteristics of thyroid nodules are useful indicators of histological malignancy. Overall, these investigations suggest a few ultrasonographic features that are significantly more frequent in malignant than in benign thyroid nodules, and some have tried to define a set of characteristics that identify nodules at higher risk of malignancy.
There is almost unanimous agreement that the presence of microcalcifications within a nodule is associated with thyroid cancer. Recently, two retrospective studies with 799 and 1475 nodules, respectively, have suggested that this is the only ultrasonographic finding predicting histological malignancy.7,23 Our data also indicate that intrinsic microcalcification is the strongest criterion for cancer (multivariate OR 10.8).
Blurred margins, hypoechoic pattern and intranodular vascular flow have also been associated with malignant lesions in some (but not all) investigations.9,10,16,23–27 Our results confirm the predictive value of these features, with a stronger association (OR 8.3) for blurred margins.
Decision-making
This large series of thyroid nodules prospectively investigated by US-FNAC for potential malignancy clearly indicates that nodule size is only a weak predictor of histological malignancy, as suggested by us a few years ago in a interim analysis of a subset of the present data26 and subsequently confirmed by other authors.10,25 Consequently, the decision to proceed to FNAC simply because a nodule is ‘palpable‘, or >10 mm, cannot be considered a proper strategy.
In terms of risk to benefit ratio for the individual patient, US-FNAC should be probably considered not only for palpable nodules, but also for nodules <10 mm in diameter, taking into account the prevalence of malignant lesions in small nodules, the frequency of nodal metastasis, extracapsular growth and aggressive histological types found in microcarcinomas and the limited incidence of minor side-effects related to the procedure. However, cost considerations have always been important and have recently become even more relevant in many countries.
Papini et al.10 suggest that FNAC on non-palpable nodules characterized by a hypoechoic appearance and at least one of three additional ultrasound features (intranodular vascular images, blurred margins and microcalcifications) would miss only 13% of carcinomas, but perform FNAC only in 31% of patients. The same criteria applied to our series would require FNAC in 39%, but miss 23% of carcinomas. However, we prospectively included nodules of any size, whereas Papini selected non-palpable nodules;10 in this respect, our data may be a more relevant test-bed for ultrasound criteria.
According to our results, the best compromise is probably to use FNAC on nodules with at least one of: microcalcifications, blurred margins or hypoechoic pattern. Compared to the strategy of using FNAC only on nodules 10 mm, this strategy requires a similar number of procedures but reduces the percentage of missed carcinomas from 20% to 2%.
If economical and/or organizational reasons restrict the number of FNACs that can be done, then including those nodules with at least two of: (size 10 mm, hypoechoic pattern, blurred margins and microcalcifications) would limit the number of procedures to 57%, and missed carcinomas to about 7%.
Study limitations
Our percentage of non-diagnostic specimens at US-FNAC was 26% in small nodules, about twice that in large nodules; similarly high percentages of inadequate cytology have been previously reported for nodules <10 mm.5,10,11 It is unclear whether (and how) this difference in diagnostic evaluation between small and large nodules affects the interpretation of our data, since follow-up was limited to a 6 months in all patients. However, we do not think that our main finding of a rather high prevalence of malignant lesions in small nodules is invalidated by this limitation.
|
Conclusions |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
The prevalence of malignant lesions in thyroid nodules referred for US-FNAC is not substantially different in nodules above 1 cm vs. below 1 cm in size. Histological type and local aggressiveness are not more favourable in ‘microcarcinomas’ than in larger thyroid cancers. A cost-effective approach to the use of FNAC should not depend solely upon nodule size; specific ultrasound patterns (microcalcifications, blurred margins and hypoechoic appearance) appear to be useful indicators.
|
Notes |
|---|
* Drs Cappelli and Castellano contributed equally to this paper.
|
References |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsReferences |
|---|
1. Mandel SJ. (2004) A 64-year-old woman with a thyroid nodule. JAMA 292 2632–42.
2. Castro MR and Gharib H. (2005) Continuing controversies in the management of thyroid nodules. Ann Intern Med 142 926–31.
3. Mazzaferri EL. (1993) Management of solitary thyroid nodule. N Engl J Med 328 553–9.
4. Hegedus L. (2004) The Thyroid Nodule. N Engl J Med 351 1764–71.
5. Tan GH and Gharib H. (1997) Thyroid incidentalomas: management approaches to non-palpable nodules discovered incidentally on thyroid imaging. Ann Intern Med 126 226–31.
6. Rasmussen LB, Ovesen L, Bulow I, Jorgensen T, Knuddsen N, Lauberg P, Perrild H. (2002) Relations between various measures of iodine intake and thyroid volume, thyroid nodularity, and serum thyroglobulin. Am J Clin Nutr 76 1069–76.
7. Lyshchik A, Higashi T, Asato R, Tanaka S, Ito J, Mai JJ, Pellot-Barakat C, Insana MF, Brill AB, Saga T, Hiraoka M, Togashi K. (2005) Thyroid gland tumor diagnosis at US Elastography. Radiology 237 202–11.
8. Gharib H. (1994) Fine-needle aspiration biopsy of the thyroid nodules: advantages, limitations, and effect. Mayo Clin Proc 69 44–9.[ISI][Medline]
9. Lin JD, Huang BY, Weng HF, Jeng LB, Hsueh C. (1997) Thyroid ultrasonography with fine-needle aspiration cytology for the diagnosis of thyroid cancer. J Clin Ultrasound 25 111–18.[CrossRef][ISI][Medline]
10. Papini E, Guglielmi R, Bianchini A, Crescenzi A, Taccogna S, Nardi F, Panunzi C, Rinaldi R, Toscano V, Pacella CM. (2002) Risk of malignancy in nonpalpable thyroid nodules: predictive value of ultrasound and color-doppler features. J Clin Endocrinol Metab 87 1941–6.
11. Leenhardt L, Hejblum G, Franc B, Fediaevsky LD, Delbot T, Le Guillouzic D, Menegaux F, Guillausseau C, Hoang C, Turpin G, Aurengo A. (1999) Indications and Limits of Ultrasound-Guided Cytology in the Management of Nonpalpable Thyroid Nodules. J Clin Endocrinol Metab 84 24–8.
12. Hay ID, Grant CS, van Heerden JA, Goelbner JR, Ebersold JR, Bergstralh EJ. (1994) Papillary thyroid microcarcinoma: a study of 535 cases observed in a 50-year period. Surgery 112 1139–47.
13. Rosaj J, Carcangiu ML, De Lellis RA. (1996) Papillary Carcinoma. Atlas of tumour pathology. Tumors of the thyroid gland Washington pp. 96–100.
14. Yamashita H, Noguchi S, Murakami N, Toda M, Uchino S, Watanabe S, Kawamoto H. (1999) Extracapsular invasion of lymph node metastasis. A good indicator of disease recurrence and poor prognosis in patients with thyroid microcarcinoma. Cancer 86 842–9.[CrossRef][ISI][Medline]
15. Lupoli G, Vitale G, Caraglia M. (1999) Familial papillary thyroid microcarcinoma: a new clinical entity. Lancet 353 637–9.[CrossRef][ISI][Medline]
16. Sugitani I and Fujimoto Y. (1999) Symptomatic versus asymptomatic papillary thyroid microcarcinoma: a retrospective analysis of surgical outcome and prognostic factors. Endocr J 46 209–16.[ISI][Medline]
17. Guidelines of the Papanicolaou Society of Cytopathology for the Examination of Fine-Needle Aspiration Specimens from Thyroid Nodules. (1996) The Papanicolaou Society of Cytopathology Task Force on Standards of Practice. Mod Pathol 9 710–15.[ISI]
18. In Hermanek P, Hutter R, Sobin L, Wagner G, Wittekind C (Eds.). TNM Atlas (1997) Berlin-Heidelberg Springer-Verlag.
19. Deeks JJ and Altman DG. (2004) Diagnostic test 4: Likelihood ratios. Br Med J 329 168–9.
20. Lin JD, Chao TC, Huang BY, Chen ST, Chang HY, Hsueh C. (2005) Thyroid cancer in the thyroid nodules evaluated by ultrasonography and fine-needle aspiration cytology. Thyroid 15 708–17.[CrossRef][ISI][Medline]
21. Wang C and Crapo L. (1997) The epidemiology of thyroid disease and the implications for screening. Endocrinol Metab Clin North Am 26 189–98.[CrossRef][ISI][Medline]
22. Ito Y, Ubruno T, Nakano K, Takamura Y, Miya A, Kobayashi K, Yokozawa T, Matsuzuka F, Kuma S, Kuma K, Miyauchi A. (2003) An observation trial without surgical treatment in patients with papillary microcarcinoma of the thyroid. Thyroid 13 381–7.[CrossRef][ISI][Medline]
23. Kang HW, No H, Chung JH, Min YKI, Lee MS, Lee MK, Yang JH, Kim KW. (2004) Prevalence, clinical and ultrasonographic characteristics of thyroid incidentalomas. Thyroid 14 29–33.[CrossRef][ISI][Medline]
24. Chan BK, Desser TS, McDougal R, Weigel RJ, Brooke JR Jr. (2003) Common and Uncommon Sonographic Features of Papillary Thyroid Carcinoma. J Ultrasound Med 22 1083–90.
25. Hegedus L. (2001) Thyroid ultrasound. Endocrinol Metab Clin North Am 30 1941–6.
26. Cappelli C, Braga M, Agosti B, Cerudelli B, Giubbini R, Salvi A, Agabiti Rosei E. (2001) Management of thyroid nodules with ultrasonographic hypoechoic pattern and size less than one centimetre. J Endocrinol Invest 24 98 (abstract).[ISI][Medline]
27. Tritton RL, Gervais DA, Boland GW, Maher MM, Mueller PR. (2003) Sonography and sonographically guided fine-needle aspiration biopsy of the thyroid gland: indications and techniques, pearls and pitfalls. Am J Radiol 14 267–71.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. Y. Kim, C. H. Lee, S. Y. Kim, W. K. Jeon, J. H. Kang, S. K. An, and W. S. Jun Radiologic and Pathologic Findings of Nonpalpable Thyroid Carcinomas Detected by Ultrasonography in a Medical Screening Center J. Ultrasound Med., February 1, 2008; 27(2): 215 - 223. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C Cappelli, I Pirola, M Castellano, E Gandossi, E De Martino, A Delbarba, B Agosti, A Tironi, and E A. Rosei Fine needle cytology of complex thyroid nodules Eur. J. Endocrinol., October 1, 2007; 157(4): 529 - 532. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R Jones, R Spendiff, S Fareedi, and P S Richards The role of ultrasound in the management of nodular thyroid disease Imaging, March 1, 2007; 19(1): 28 - 38. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||