QJM Advance Access originally published online on October 15, 2008
QJM 2009 102(1):43-49; doi:10.1093/qjmed/hcn139
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
A recognition tool for transient ischaemic attack
From the Division of Cardiovascular and Medical Sciences, Western Infirmary Hospital, University of Glasgow, UK
Address correspondence to Jesse Dawson, Lecturer in Clinical Pharmacology and Medicine, Western Infirmary Hospital, Dumbarton Road, Glasgow, G11 6NT, UK. email: j.dawson{at}clinmed.gla.ac.uk
Received 19 June 2008 and in revised form 15 October 2008
 |
Summary |
|---|
 Top Summary IntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Background: Scoring systems exist to assist rapid identification of acute stroke but not for the more challenging diagnosis of transient ischaemic attack (TIA).
Aim: To develop a clinical scoring system to assist with diagnosis of TIA.
Methods: We developed and validated a clinical scoring system for identification of TIA patients. Logistic regression analysis was employed.
Results: Our development cohort comprised 3216 patients. The scoring system included nine clinically useful predictive variables. After adjustment to reflect the greater seriousness of missing true TIA patients (a 2:1 cost ratio), 97% of TIA and 24% of non-TIA patients were accurately identified. Our results were confirmed during prospective validation.
Conclusions: This simple scoring system performs well and could be used to facilitate accurate detection of TIA.
|
Introduction |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Patients with a recent transient ischaemic attack (TIA) or minor stroke merit rapid investigation and treatment initiation to minimize their risk of future vascular events; the seven-day risk of stroke following TIA may exceed 30% in the highest risk groups1 and risk is highest in the first 48 h.2 Accordingly, guidelines recommend that patients are assessed as soon as possible (and at least within 1 week)3,4 and that fast track or rapid access neurovascular clinics are established.5
Such clinics can significantly reduce early stroke risk6 but immediate or same day assessment of those with suspected TIA will be difficult to achieve nationwide and measures to optimize efficiency of assessment would be welcome. Difficulties in accurate diagnosis of patients with suspected TIA are well documented7–9; interobserver agreement has been shown to be as low as 50%,10 and between 31% and 62% of patients referred with suspected TIA are deemed to have a non-cerebrovascular diagnosis by a stroke specialist.9,11–13 Improvements in diagnostic accuracy of TIA and a reduction in rates of non-cerebrovascular referrals from general practitioners (GPs) would reduce the burden on TIA clinics and facilitate the rapid management of those with genuine TIA.
Diagnostic algorithms have been successfully used in acute stroke14–18 but no equivalent system exists for suspected TIA. Risk stratification instruments such as the ABCD and ABCD2 scores are available but were developed on populations with confirmed TIA1,2 and while diagnostic algorithms are employed in clinical trials,19,20 they are not widely used in clinical practice. We hypothesized that a diagnostic algorithm and clinical scoring system could be developed to aid accurate diagnosis of TIA and minor stroke, that its use would reduce non-cerebrovascular referrals and that this would reduce clinic waiting times and facilitate rapid assessment.
|
Methods |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Our algorithm was developed using data contained in the West Glasgow Stroke Registry and was tested on an independent prospective data set from the same source. The impact of the projected reduction in non-cerebrovascular referral rate was then assessed using real data on referral rates and clinic availability during the prospective validation study.
The Western Infirmary serves a catchment population of approximately 225 000 people and receives approximately 500–600 outpatient referrals per year [predominantly (>95%) from GPs]. The fast track TIA clinic is held twice weekly. Baseline demographic data, a history of presenting complaint, relevant examination findings and diagnosis and management plans are prospectively recorded at the time of clinic review. All patients are discussed with a Consultant Stroke Physician (with at least 10 years experience). Data are entered into an electronic database (the West Glasgow Stroke Registry) which currently includes all patients who attended the Fast Track clinic between March 1992 and January 2005. At the follow-up visit, data regarding investigation results, final diagnosis and treatment plans are gathered.
Development of the diagnostic tool
We used diagnosis (‘cerebrovascular’ vs. ‘non-cerebrovascular’) as determined at clinic visit as the reference standard in the study. This included patients with TIA and those with minor stroke symptoms lasting 24 h who had not been referred to the in-patient service. We identified in advance variables likely to be useful in the diagnosis of TIA, including those thought to be suggestive of an alternative diagnosis (Table 1). We included symptoms used in stroke diagnostic algorithms and those previously shown to predict a stroke diagnosis.14–18,21 Logistic regression models were used to identify discriminatory variables and to develop a clinical scoring system.
|
Statistical analysis
Analyses were performed using S Plus version 6.2. First, univariate analysis was used to identify variables predictive of diagnosis. Logistic regression models were used to identify independently discriminatory variables. Stepwise selection procedures (both forward and backward) were employed to identify significant explanatory variables using Akaike's Information Criterion.22 In a forward stepwise selection procedure, the initial model involves no explanatory variables and the most significant variable of all the explanatory variables is added to the model at each step until all significant variables are included. In backward selection, the initial model involves all possible explanatory variables and at each step the least significant variable is omitted from the model until the final model selected involved only significant variables. Two-way interaction variables were also considered. The final model was internally validated using 3-fold cross-validation. During 3-fold cross-validation, the data were split into three groups of equal size and the model was fitted to the data of two of the groups to predict the class of the remaining group. This was repeated for all group combinations and for two further random splits of the data.
Variables that showed discriminatory power were considered for inclusion in a clinical scoring system. Non-weighted (where explanatory variables were assigned a value of 1) and weighted scoring systems (based upon the regression coefficient and rounded to one decimal place) were then developed. Receiver operating characteristic curves (ROC) were used to determine optimal cut-off scores; and sensitivity, specificity and positive and negative predictive values were calculated. Following this, two ‘costs of misclassification’ models were developed to reflect the presumed greater importance of failing to identify cerebrovascular events compared to incorrectly labelling mimics. A cost of 2:1 is considered when it is assumed that the cost of misclassifying a cerebrovascular patient as non-cerebrovascular is twice as much as the cost of misclassifying in the opposite direction. Similarly, a cost of 3:1 is where the cost of misclassifying a cerebrovascular patient as non-cerebrovascular is deemed to be three times as great. Essentially, it is a penalty assigned for making a mistake, taking into account that a failure to treat a life-threatening condition is more serious than undertaking unnecessary but generally safe investigations. By raising or lowering the cost of a misclassification, decisions are biased in different directions, as if there were more or fewer cases in a given class. These ratios were arbitrarily chosen. The Hosmer–Le Cessie test was used to evaluate fit of the model.
We have not performed a formal power calculation for our multivariable logistic regression models; this is notoriously complex. However, we have a large number of outcomes for each variable included [35 for the least common variable (seizure) with others in the hundreds]. Further, the small standard errors, small P-values and large sample size give further evidence that power was sufficient for us to proceed to multivariate modelling.
Prospective validation of the diagnostic tool
Data on all referrals to the Fast Track TIA clinic were gathered from October 2005 to June 2006 (by J.D.). The clinical scoring system scores were not used to aid clinical diagnosis and were later calculated by independent observers who were not involved with the patients care or development of the scoring system (M.H., M.J.V.).
Assessment of the impact of the tool
Delays to clinic appointment during the prospective validation phase were calculated (expressed as median and inter-quartile range). The effect of the projected reduction in non-cerebrovascular referral rates was then established via a model based upon the actual number of referrals and clinic availability during the study period. During early modelling, it was predicted that the scoring system would reduce the number of non-cerebrovascular referrals by 
50% or 25% with the unadjusted weighted system and 2:1 costs model, respectively. We therefore recalculated median delay to clinic appointment first with every second then every fourth non-cerebrovascular patient being removed.
The advice of a Multi-centre Research Ethics Committee was sought and formal ethics committee approval and formal informed consent were deemed unnecessary.
|
Results |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Development of the scoring system
The development cohort included 3230 patients. Mean age was 65 years (SD 12.8). Other baseline characteristics are shown in Table 2. Sufficient data were available for 3216 patients, of whom 2215 (69%) had a diagnosis of TIA or minor stroke.
|
Variables predictive of diagnosis on univariate analysis are shown in Table 1. Three risk factors (history of stroke or TIA, hypertension and diabetes) and 17 clinical features were significantly predictive of clinician diagnosis on logistic regression analysis. The predictive clinical features were headache, vomiting, loss of consciousness, seizure (all predictive of non-cerebrovascular diagnosis), age, duration of symptoms, visual loss, diplopia, ataxia, speech disorder, dysphasia, unilateral arm or leg weakness, unilateral facial weakness, unilateral sensory disturbance, other pattern of weakness and other pattern of sensory disturbance (predictive of a cerebrovascular diagnosis). During stepwise selection, 8 variables were rejected from the model leaving 12 explanatory variables. These were history of stroke or TIA, headache, diplopia, loss of consciousness, seizure, age, duration of symptoms, speech disorder, unilateral leg weakness, unilateral upper motor neuron (UMN) facial weakness, unilateral lower motor neuron (LMN) facial weakness and other weakness. No interaction terms remained in the model.
Three variables were removed as it was felt that they would not be useful in the scoring system; LMN facial weakness was removed as there were few cases and each occurrence was associated with a diagnosis of Bell's palsy (a rare occurrence in the TIA clinic). UMN facial weakness was then renamed as unilateral facial weakness. Duration of symptoms was removed. This was recorded as <1 h, 1–24 h, 1–3 days or >3 days in the database and it was felt that none of these boundaries was sensitive enough to influence decision making; many TIAs last <1 h, while on the other hand, those lasting longer convey greater risk.1,2’Other weakness’ was also removed. This was essentially defined as non-unilateral weakness, so was already being accounted for in the model and it was felt this may lead to confusion during use of the score. Unilateral leg weakness was included in the final model but unilateral arm weakness was not; this was because the vast majority of patients with unilateral arm weakness had unilateral leg weakness and vice versa meaning one of these variables was removed during stepwise regression. The variable unilateral leg weakness was therefore replaced by unilateral limb weakness. This left nine predictive variables: six positive indicators of cerebrovascular disease and three indicators of a non-cerebrovascular diagnosis. The final regression coefficients (when only these nine variables were included and rounded to one decimal place) are shown in Table 3 and all had P-values of <0.001. Removal of the unsuitable variables did not affect the performance of the final model (assessed using linear discriminant analysis).
|
ROC curves identified a score of >6.1 as the optimal cut-off for prediction of cerebrovascular diagnosis using the weighted scoring system (Table 3). This accurately identified 84% of cerebrovascular diagnoses and 60% of non-cerebrovascular diagnoses with a positive predictive value of 82% and NPV of 62%. With adjustment to reflect the greater seriousness of missing true cerebrovascular patients (a 2:1 cost ratio), an optimal cut-off score of >5.4 was used (Figure 1) and 97% of TIA and 24% of non-TIA patients were accurately identified with a PPV 73% and NPV 78% (Table 4). The Hosmer–Le Cessie test gave a P-value of 0.02 for fit of this model. The 3:1 costs model gave a sensitivity of 99% but reduced specificity to only 12% and so was abandoned. The unweighted scoring system also performed less well and was also abandoned (sensitivity of 82% and specificity of 60%).
|
|
The prospective validation set
Two hundred and thirty-seven patients were included of whom 143 (60.3%) had a diagnosis of TIA or minor stroke. Baseline characteristics are shown in Table 2.
The weighted scoring system correctly identified 85% of patients with a cerebrovascular diagnosis and 54% of those with a non-cerebrovascular diagnosis with a PPV of 74% and an NPV of 70%. Using the 2:1 misclassification score, 93% of patients with a cerebrovascular diagnosis and 34% of those with a non-cerebrovascular diagnosis were correctly identified with a PPV of 68% and an NPV of 76% (Table 4). The confidence intervals for all parameters greatly overlap those found during the development phase.
Impact on clinic waiting times
During the period of the prospective validation set, the median waiting time from referral date to clinic attendance was 15 days (IQR 8–23). Removal of every fourth and every second non-cerebrovascular patient had potential to reduce the median waiting time to 8 and 7 days, respectively.
|
Discussion |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Our results show that a clinical scoring system could be utilized to reduce the number of non-cerebrovascular referrals to a fast track TIA service. A scoring system that employed a 2:1 cost of misclassification adjustment would have little adverse impact on recognition rates of bona fide TIA and reduces the number of non-cerebrovascular referrals by a clinically meaningful amount.
TIA is poorly managed in many countries, including in the United Kingdom. Recent reports from the UK National Audit Office and the House of Commons Public Accounts Committee23 found that approximately half of patients were seen within 14 days, 58% had a scan outside an effective time window and the majority waited 12 weeks or more for a carotid ultrasound scan. This must be improved as more rapid management is effective in reducing stroke risk.6
The UK consensus guidelines suggest that patients with suspected TIA are assessed and investigated within 1 week,3,4 while European guidelines appropriately suggest assessment ‘without delay’.24 Fulfilling these aims will be costly and will present challenges. For example, our clinic was established (and able) to assess patients within days but experienced a progressive increase in non-cerebrovascular referrals to 50%.11 This ‘overloaded’ the system, such that the median delay to clinic assessment in the prospective validation period of this study was an unsatisfactory 15 days. We believe that reducing the number of non-cerebrovascular referrals is feasible and will free up existing resources and facilitate urgent assessment of those with TIA.
However, accurate identification of stroke and TIA patients is notoriously difficult9,10 and a variety of conditions, such as seizure, migraine or systemic upset can mimic TIA.21 Assessment algorithms exist but are rarely employed outwith the clinical trial setting19,20 and in practical terms represent little more than descriptions of a typical TIA which give little practical guidance to aid decision making by non-specialists. A recent systematic review of the predictive value of various symptoms and clinical signs21 and the available stroke assessment tools14–18 show that symptoms such as unilateral weakness and a language disorder suggest that stroke has occurred, while loss of consciousness or seizure activity point towards an alternative diagnosis. Diplopia, vertigo and sensory loss are also, but more weakly, consistent with stroke. Unsurprisingly, we also found similar variables to be predictive of TIA.
Unilateral limb weakness, unilateral facial weakness, speech disorder, diplopia, history of stroke or TIA and increasing age were predictive of TIA. There are obvious deficiencies of a scoring system involving only these variables. Patients with amaurosis fugax, sensory lacunar and some posterior circulation events would be missed. It is unrealistic, given the heterogeneity of TIA symptoms, to expect that a clinical scoring system could detect all types of event. Amaurosis fugax for example has an entirely different set of differential diagnoses than other types of TIA. It is imperative that such systems are introduced in conjunction with improved user education, with clear direction as to their limitations and clear advice that clinician concern and acumen override the score.
The recently developed ABCD and ABCD2 scores are risk stratification instruments that deserve comment.1,2 They seem able to identify those with a high risk of stroke early after TIA but were developed on data from patients with a confirmed diagnosis of TIA and are thus unproven as diagnostic instruments. They would also suffer similar, or perhaps greater, weaknesses as our score in terms of failure to detect amaurosis fugax, sensory lacunar and posterior circulation events, but whether our score would add further to the increasingly used ABCD scores is unclear and under investigation.
There are limitations to our study. We chose to use the clinic diagnosis rather than final diagnosis (such as that supported by brain imaging) as the reference standard. Essentially, therefore we have assessed whether a scoring system can distinguish those whom a specialist feels require further investigation for cerebrovascular disease from those who do not. We feel this is an appropriate aim for a diagnostic tool, in particular one to aid the clinical diagnosis of TIA. However, we have not assessed levels of inter-observer agreement in our unit but hope that our standardized review process, the large number of subjects and limited number of experienced observers used will limit this. Further, only three patients with suspected stroke had a final diagnosis which differed from the original clinic diagnosis during the prospective validation phase. Although it was reassuringly consistent with the development sample, our prospective validation sample was small and cannot replace external validation. The Hosmer–Le Cessie test yielded a P-value of 0.02 suggesting that the favoured model is not a good fit to the data (P-values of >0.05 suggest a good fit). The fit of the model could perhaps have been improved by selecting different weights for the scores involved in the system or by selecting a different cut-off point for the model, but we believe this renders the tool less clinically useful; we assigned a greater importance to sensitivity in order to ensure that the maximum number of true TIA cases would be referred for specialist for assessment.
We hope our tool could be used by our colleagues in primary care. However, it was developed using data generated during assessment by stroke specialists. It is not certain that GPs would reach the same conclusions regarding what constitutes a particular symptom, for example, pre-syncope or unilateral limb weakness. A sensitivity of 85%, as seen with the weighted scoring system, represents a failure rate of TIA detection to great for safe use in clinical practice. The 2:1 misclassification scoring system yielded a sensitivity of 97% and specificity of 24% during the development phase and 93% and 34%, respectively, during the prospective validation phase. We found that this reduction in non-cerebrovascular referral rate to be clinically meaningful; it would greatly reduce delay to assessment in our service. The weighted system performed best, although the non-weighted system would be easiest to use in clinical practice. However, many centres are moving towards an electronic referral process which would allow scores to be calculated automatically from information given, without the need for manual calculation. An example of this can be found at http://www.stams.strath.ac.uk/~karenl/tia/. This allows maximum information and discriminatory ability to be retained, and in particular to avoid dichotomizing continuous variables such as age.
The false negatives or ‘missed TIAs’ are of interest. In the prospective validation phase, there were 10 patients with a presumed cerebrovascular diagnosis who were not identified by the 2:1 costs scoring system. Of these, three were felt at final clinic review not to have had a TIA or minor stroke and only one of the remainder had brain imaging supportive of ischaemia in the relevant territory. This patient presented with a pure haemianopic visual disturbance. Two had possible transient ataxia, one had a transient language disorder and the remainder pure sensory symptoms. We hope these patients would still be identified if the score was used in conjunction with basic clinical acumen; use of the score itself will improve knowledge of presentation of stroke and TIA and may thus increase detection of TIA at the population level.
In summary, our study suggests that a reduction in the rate of referral of non-cerebrovascular diagnoses could significantly improve performance of TIA services and that a simple clinical scoring system could be used to achieve this. Further work to evaluate use of the score in clinical practice is ongoing. Independent external validation and perhaps comparison to the ABCD scores would be of use.
|
Acknowledgements |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We wish to acknowledge Marie McIver and Pamela MacKenzie for their hard work inputting the data to registry.
Conflict of interest: None declared.
|
References |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
1. Rothwell PM, Giles MF, Flossmann E, Lovelock CE, Redgrave JNE, Warlow CP, et al. A simple score (ABCD) to identify individuals at high early risk of stroke after transient ischaemic attack. Lancet (2005) 366:29–36.[CrossRef][Web of Science][Medline]
2. Johnston SC, Rothwell PM, Nguyen-Huynh MN, Giles MF, Elkins JS, Bernstein AL, et al. Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet (2007) 369:283–92.[CrossRef][Web of Science][Medline]
3. The Intercollegiate Working Party for Stroke. National Clinical Guidelines For Stroke. (2004) London: RCPL.
4. Wolf PA, Clagett GP, Easton JD, Goldstein LB, Gorelick PB, Kelly-Hayes M, et al. Preventing ischemic stroke in patients with prior stroke and transient ischemic attack - a statement for healthcare professionals from the Stroke Council of the American Heart Association. Stroke (1999) 30:1991–4.
5. Department of Health. National Service Framework for Older People, Standard Five (2001) London: DoH. 61–75.
6. Rothwell PM, Giles MF, Chandratheva A, Alexander FC. Effect of urgent treatment of transient ischaemic attack and minor stroke on early recurrent stroke (EXPRESS study): a prospective population-based sequential comparison (Lancet 2007; 370:1432), Lancet (2008) 371:386.[Medline]
7. Libman RB, Wirkowski E, Alvir J, Rao TH. Conditions that mimic stroke in the emergency department - implications for acute stroke trials. Arch Neurol (1995) 52:1119–22.
8. Kothari RU, Brott T, Broderick JP, Hamilton CA. Emergency physicians - accuracy in the diagnosis of stroke. Stroke (1995) 26:2238–41.
9. Ferro JM, Pinto AN, Falcao I, Rodrigues G, Ferreira J, Falcao F, et al. Diagnosis of stroke by the nonneurologist - a validation study. Stroke (1998) 29:1106–9.
10. Tomasello F, Mariani F, Fieschi C, Argentino C, Bono G, Dezanche L, et al. Assessment of inter-observer differences in the Italian multi-center study on reversible cerebral-ischemia. Stroke (1982) 13:32–5.
11. Murray S, Bashir K, Lees KR, Muir K, MacAlpine C, Roberts M, et al. Epidemiological aspects of referral to TIA clinics in Glasgow. Scottish Med. J (2007) 52:4–8.
12. Dennis MS, Bamford JM, Sandercock PAG, Warlow CP. Incidence of transient ischemic attacks in Oxfordshire, England. Stroke (1989) 20:333–9.
13. Sempere AP, Duarte J, Cabezas C, Claveria LE. Incidence of transient ischemic attacks and minor ischemic strokes in Segovia, Spain. Stroke (1996) 27:667–71.
14. Kidwell CS, Starkman S, Eckstein M, Weems K, Saver JL. Identifying stroke in the field - prospective validation of the Los Angeles Prehospital Stroke Screen (LAPSS). Stroke (2000) 31:71–6.
15. Kothari RU, Pancioli A, Liu T, Brott T, Broderick J. Cincinnati prehospital stroke scale: reproducibility and validity. Ann Emerg Med (1999) 33:373–8.[CrossRef][Web of Science][Medline]
16. Harbison J, Hossain O, Jenkinson D, Davis J, Louw SJ, Ford GA. Diagnostic accuracy of stroke referrals from primary care, emergency room physicians, and ambulance staff using the face arm speech test. Stroke (2003) 34:71–6.
17. Bray JE, Martin J, Cooper G, Barger B, Bernard S, Bladin C. Paramedic identification of stroke: community validation of the Melbourne Ambulance Stroke Screen. Cerebrovasc Dis (2005) 20:28–33.[CrossRef][Web of Science][Medline]
18. Nor AM, Davis J, Sen B, Shipsey D, Louw SJ, Dyker AG, et al. The Recognition of Stroke in the Emergency Room (ROSIER) scale: development and validation of a stroke recognition instrument. Lancet Neurol (2005) 4:727–34.[CrossRef][Web of Science][Medline]
19. Koudstaal PJ, Vangijn J, Staal A, Duivenvoorden HJ, Gerritsma JGM, Kraaijeveld CL. Diagnosis of transient ischemic attacks - improvement of interobserver agreement by a checklist in ordinary language. Stroke (1986) 17:723–8.
20. Karanjia PN, Nelson JJ, Lefkowitz DS, Dick AR, Toole JF, Chambless LE, et al. Validation of the ACAS TIA/stroke algorithm. Neurology (1997) 48:346–51.
21. Goldstein LB, Simel DL. Is this patient having a stroke? JAMA (2005) 293:2391–402.
22. Akaike H. Information theory and an extension of the maximum likelihood principle. 2nd International Symposium on Information Theory, Academia K1400, Budapest (1973) 267–281.
23. National Audit Office. Reducing brain damage: faster access to better stroke care. [http://www.nao.org.uk/publications/nao_reports/05-06/0506452.pdf.] Accessed 20 July 2006.
24. The European Stroke Organisation (ESO) Executive Committee and the ESO Writing Committee. Guidelines for the management of ischaemic stroke and transient ischaemic attack 2008. Cerebrovasc Dis (2008) 25:457–507.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D S Lasserson Initial management of suspected transient cerebral ischaemia and stroke in primary care: implications of recent research Postgrad. Med. J., August 1, 2009; 85(1006): 422 - 427. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. J. Quinn, A. C. Cameron, J. Dawson, M. R. Walters, and K. R. Lees Response to Letter by Ray et al Stroke, June 1, 2009; 40(6): e468 - e468. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||