Rheumatology Advance Access originally published online on November 15, 2005
Rheumatology 2006 45(4):459-464; doi:10.1093/rheumatology/kei185
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Competence-based assessment of specialist registrars: evaluation of a new assessment of out-patient consultations
Staffordshire Rheumatology Centre, Haywood Hospital, Stoke-on-Trent and West Midlands Deanery, Birmingham Research Park, Birmingham, UK.
Correspondence to: C. Dowson, Staffordshire Rheumatology Centre, Haywood Hospital, High Lane, Burslem, Stoke-on-Trent ST6 7AG, UK. E-mail: Caitlyn.Dowson{at}uhns.nhs.uk
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Objective. To implement and evaluate formal assessment of the performance of West Midlands rheumatology specialist registrars (SpRs) in conducting out-patient consultations.
Methods. SpRs conducted a full out-patient clinic while being observed by one or two consultants who scored each SpR's performance on a structured pro forma. The assessment was conducted within the examining consultant's hospital trust (in which the SpR was not working). The process was evaluated by trainees and trainers by means of a questionnaire.
Results. Sixteen SpRs were assessed. No trainee failed to reach the required standard. Six trainees scored ‘borderline’ for management planning and four had a borderline performance recorded for examination skills. Overall the process was valued by trainers and trainees, although consensus was that it would be more practicable for the trainee to be examined within the hospital in which they were working by a visiting consultant examiner. It was also felt that a broader range of scoring options would be preferable.
Conclusion. Formal assessment, by direct observation, of the skills of SpRs in conducting an out-patient clinic is practicable and offers information which is useful to trainer and trainee. A culture of direct observation of performance can be achieved within the context of rheumatology higher medical training.
KEY WORDS: Competence, Assessment, Specialist registrar, Out-patient, Consultation
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Specialist registrar (SpR) training has continued to evolve since Calman first introduced the concept of streamlined training in 1997 [1]. Rather than simply training for a set period of time, SpRs are now required to meet appropriate standards to progress to their next year of training. Training committees are responsible for determining and documenting that these standards have been met. SpRs complete training records providing details of conditions they have seen, procedures they have performed, audit and research with which they have been involved and courses they have attended. These log books predominantly demonstrate training received rather than levels of knowledge or clinical competence attained [2].
In January 2003, the Joint Committee on Higher Medical Training (JCHMT) introduced competence-based curricula for many medical specialties, including rheumatology. These outline the knowledge, skills and attitudes to be acquired by SpRs before they may be awarded the Certificate of Completion of Specialist Training (CCST) [3]. Against this background there is a clear need to develop robust mechanisms of demonstrating SpR competence. Yet within rheumatology, and indeed most of the other medical subspecialties nationally, there has been little if any formal assessment of SpR competence beyond the educational supervisor's annual report.
In the West Midlands, rheumatology SpRs have undergone annual assessments since the training grade was established. These assessments have been designed, implemented and evaluated by the West Midlands Rheumatology Services and Training Committee (WMRSTC) and have included Objective Structured Clinical Examinations (OSCEs) [4], research presentations, debates and mock consultant interviews. The assessments are carried out prior to the annual Record of In Training Assessment (RITA) so that the results can be taken into account when determining if each individual SpR has reached the standards required to progress to the next year of training.
In 2003, the training committee decided to pilot and evaluate a competence-based assessment of SpRs’ performance in conducting routine out-patient consultations. The aims were to assess a range of clinical and managerial skills in a normal clinical setting and to evaluate this novel assessment method. We report the results of this pilot, which involved 16 rheumatology SpRs in the West Midlands.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Candidates and examiners
Candidates consisted of all SpRs in rheumatology training in the West Midlands. The examiners were consultants in rheumatology in Trusts within the West Midlands. The majority, but not all, were educational supervisors for rheumatology SpRs.
The rheumatology SpRs and consultants in the region were given information regarding the aims, format and scoring criteria for the compulsory out-patient clinic assessment. Each trainee was required to set a date for the examination with their allocated consultant assessor, within whose Trust they had not previously worked as an SpR.
Clinic setting and patients
Each trainee was examined in the consultant assessor's out-patient clinic. Each consultant invited consenting new and follow-up patients, with a variety of rheumatological problems, to attend the clinic. The consultant maintained overall responsibility for the patients.
Trainees performed a normal out-patient clinic, seeing the allocated patients in the presence of the consultant examiner. Trainees took a history, performed a relevant examination, requested investigations and agreed a management plan with the patient. The trainee could seek advice from the examiner if in doubt as to patient management or local facilities, e.g. for investigations. The trainee also dictated a letter to the patient's general practitioner, again in the presence of the consultant examiner.
Assessment, feedback and evaluation
The SpRs were able to familiarize themselves with the clinic setting, notes, computers and request forms, etc. They were encouraged to conduct the clinic as they normally would do. The assessors were required to closely observe and score each consultation without interrupting, unless any of the participants (i.e. patient, trainee or consultant) felt that consultant input was required. A scoring pro forma was completed for each consultation (Table 1) and a summary sheet for the entire assessment. Constructive feedback was given during and/or at the end of the assessment. SpRs and consultants completed evaluation forms. The evaluation forms consisted of a series of statements and a Likert scale on which the respondent expressed their degree of agreement or disagreement (Table 2). For each statement the respondent was also invited to comment.
|
|
Assessment pro forma
The assessment pro forma was designed by one of the authors (A.H.) specifically for this assessment (Table 1). The scoring system and criteria were based on systems used previously, such as the rheumatology SpR OSCEs [4], and was developed with the aim of identifying less than satisfactory performance rather than to rank trainees. The scoring criteria were shown to trainees and assessors at least 1 month before the assessment. It comprised seven domains against which the trainee was scored as ‘good’, ‘borderline’ or ‘does not reach required standard’. Descriptors for each domain score were provided. Examples are shown in Table 3. The seven domains comprised history taking, examination, diagnostic skills, communication skills, management planning, letter dictation and examiner's overall impression. Assessors completed one pro forma for each patient seen and a summary pro forma at the end of the clinic. A copy of the summary pro forma was given to the trainee and also kept within the trainee's file which was made available at the time of the RITA.
|
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The out-patient assessment
Sixteen SpRs were assessed during a 3-h clinic by either one consultant throughout or by two consultants consecutively (16 consultants in total). Each SpR saw between four and seven patients comprising new and follow-up visits. The case mix was typical of a general rheumatology clinic and consisted of a mixture of patients and problems seen in everyday practice, including at least two follow-up patients with inflammatory arthropathies and at least one new patient.
Candidates' scores
Ten SpRs scored ‘good’ overall for at least six out of the seven domains. No-one ‘failed to reach the required standard’ (Fig. 1). Borderline scores occurred most frequently for management planning and examination skills (Fig. 2).
|
|
There was no clear correlation between year of training and overall scores as the majority of candidates were at about the same stage of their training (four in year 2, 10 in year 3 and two in year 4).
Evaluation
Evaluation forms were completed by 12 SpRs and 13 consultant examiners. Overall 12 of the consultants and all responding SpRs were broadly positive about the process.
Competence
Twelve trainees and twelve examiners agreed that this was a worthwhile exercise for evaluating important clinical and consultation skills. One consultant disagreed, stating ‘Observing the process altered trainee behaviour’. Two consultants suggested the assessment provided useful information regarding the SpRs’ attitudes and one stated that it was ‘very informative’. Two SpRs commented that the unfamiliar environment and notes made the clinic more difficult and time-consuming. Two SpRs questioned how true to life the clinic was. One SpR stated that it was a good formative assessment whilst two commented on the failure of the examiner to use Pendleton's rules of feedback. All consultants and all but one SpR disagreed with the statement that ‘the skills/knowledge/attitudes which the assessment is placed to evaluate are not that important’.
Strengths of the assessment
The majority of SpRs and consultants commented that this assessment was strong at assessing history taking, communication and examination skills. Three SpRs also suggested management planning skills whilst two SpRs and two consultants said attitudes. Two consultants stated the assessment was close to real-life situations. One consultant, less impressed with the assessment method, stated that it was ‘strong at assessing map-reading skills’, presumably referring to trainees finding the unfamiliar hospital.
Weakness of the assessment
Trainees and consultants felt this assessment was weak at assessing depth of knowledge. In addition, SpRs were concerned that consultations took longer in the strange environment and management plans were more difficult to make because of differing practices in different units. Several SpRs and consultants felt that the scoring system was too restricted and did not allow them to give a score any higher than ‘good’. A few suggested that the ‘borderline’ score sounded too negative and should be replaced by ‘satisfactory’. However, when asked specifically if they were confident the assessment would identify both weak and strong candidates, all bar one consultant and four SpRs agreed.
Demands of the assessment
Two trainees and four consultants felt the assessment was too demanding of the trainee, particularly because of the strange environment. One trainee and two consultants felt that the assessment was too demanding of the consultant. One consultant felt uncomfortable about being a ‘dumb observer’ and the other stated it was ‘definitely harder than seeing patients myself’.
Two SpRs and three consultants agreed that the assessment was too consuming of resources for the information it offered. The reasons given included, ‘Planned 3 h session took 4.5 h’, ‘I attempted to sort out a temporary contract—time consuming!’, ‘I had to move a clinic at late notice’ and ‘It was a very big effort for everyone across the region.’ However, the majority of those involved disagreed with the statement that it was too demanding and two consultants commented that it was probably less consuming of resources than an OSCE.
Future assessments
Almost all trainees and consultants agreed that this in-training assessment should be used again. Only one consultant said no, stating his/her preference was for an OSCE.
In response to the question regarding what assessment strategies should be used by the WMRSTC to inform the RITA, the SpRs stated that a varied approach should continue with a mixture of formative and summative assessments throughout the training programme. Suggestions included combinations of this assessment, OSCE, 360-degree assessment, decision-making exercises, presentation skills and even a written examination early in the programme. Consultants gave similar suggestions and in addition stated the importance of the logbook, attendance record and patients’ views.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Since Calman first introduced the streamlined SpR training scheme, the WMRSTC have recognized the need to implement a variety of summative and formative assessments in order to guide the RITA process and to enable trainees in the region to become competent consultants in the future. The General Medical Council (GMC), JCHMT and government have recently declared their intention to ensure effective, transparent assessment aimed at determining clinical competence in a range of settings. Thus, the development of effective work-based assessments is a major challenge facing those with responsibility for postgraduate training in medicine.
To complement competence-based curricula, the JCHMT intends to introduce a menu of competence-based assessments to be adopted as a regular form of SpR assessment [5]. The JCHMT and the Education Department of the Royal College of Physicians are studying the validity of the MiniCEX (Mini Clinical Evaluation Exercise) [6], Directly Observed Procedural Skill (DOPS) and 360-degree assessment for this purpose [7]. In addition to these measures, the JCHMT plans to revise the training record such that it more closely reflects competencies. The Department of Health is equally concerned that specialists must be seen to have reached the high standards expected of them. In a letter to the British Medical Association (BMA) and NHS Confederation, Mr Alan Milburn, the then Secretary of State for Health, declared his intention to work with the Medical Royal Colleges on ‘modernising training and introducing more competence based assessment’ [8]. He also stated that under new systems doctors should perform ‘more operations without direct supervision as soon as they have been accredited as having the right skills’. The Postgraduate Medical Education and Training Board is to be responsible for ensuring appropriate training standards in medical education and training [9].
The learning outcomes of a SpR training programme in rheumatology have recently been identified within a competency-based framework published by the JCHMT [3]. As well as defined clinical skills and knowledge, they encompass educational, managerial and problem-solving skills and include important attitudinal components. We believe the assessment described in this paper is well placed to assess competence in aspects of clinical skills (including communication skills), and, to a degree, knowledge and attitudes.
The attractiveness of this model of assessment is that it is clear that it will test the skills used in routine outpatient consultations, i.e. it has face validity [10]. Rheumatology is becoming a predominantly out-patient-based speciality and most SpRs in the region take part in four clinics per week. Although they seek help with difficult cases, they are rarely observed in action in a clinic setting. Video-recorded consultations are a well-established means of assessing general practice trainees, particularly in communication skills [11, 12], and may merit investigation in the setting of rheumatology training, but few rheumatology departments are currently equipped for this. OSCEs offer the ability to sample a range of clinical skills in a reliable fashion, using a number of examiners [13]. However, they are relatively expensive in their consumption of resources [4]. In addition, they may reflect competence rather than performance. The mini-clinical evaluation exercise (Mini-CEX) described by Norcini et al. [6, 14] offers considerable potential as a means of work-based assessment and is the subject of a recent Royal College of Physicians pilot study in the context of higher medical training. This approach relies on structured assessment, by a trainee's educational supervisor, of everyday clinical encounters during routine practice. The Mini-CEX takes about 25 min, was acceptable to participants and has the potential to be reliable [14]. However, the trainee chooses which encounters are assessed and so there may be issues of validity.
We adopted the approach of enlisting consultant assessors who were willing to examine SpRs by direct observation during a clinic. It is accepted that the presence of a supervising consultant might alter the performance of the SpR. Attempts were made to reduce this effect by ensuring that the SpR had not previously worked with the assessor, by observing several consultations during the 3-h clinic and by the assessor being as unobtrusive as possible. In contrast to an OSCE, this approach relies on the assessment by only one or two examiners, thus there is also potential for significant inter-observer variability. Our main strategy to minimize this was the provision of instructions and the use of descriptors to define trainee performance. Reliability could be further enhanced by increasing the number of examiners of a given candidate, and by holding training sessions for examiners.
Another aspect of the assessment's validity which might be questioned is its content validity [9], which in this context was the case mix of patients seen within the assessment. Apart from recommending that a mixture of new and follow-up patients be used, the case mix was not regulated. To ensure that patients with a breadth of disorders were seen and consistency for all the assessments across the region was achieved, the cases to be seen could be stipulated, although this would increase the logistical difficulties.
Despite these weaknesses, we believe that our pilot results support the need for further work in this area. The information gained from observing the SpR in clinic was thought to be of value. Individual scores influenced discussions at the RITA regarding SpRs’ progress and areas worth improving. Overall results highlighted areas requiring greater emphasis in the training programme but, on the whole, were encouraging and implied that the SpRs are generally competent at leading consultations in clinic.
Although the main aim of this criterion-referenced assessment was to detect any failing SpRs so that appropriate remedial action could be taken, some assessors and trainees stated that they wanted more options so that they could rank SpRs’ performances as, for example, good, very good or excellent. A few assessors felt uncomfortable giving a borderline score, which ‘sounded rather negative’, and instead preferred to give a good score. The number of borderline scores given may therefore have been reduced and the perceived need for improvement diminished. A prior, small pilot study of the scoring system may have highlighted such issues and would be worth undertaking during the development of future assessments. Multiple assessments of a number of trainees were not undertaken at the time of the out-patient assessment but would be valuable in order to assess the reproducibility of the process.
Of particular interest, however, was the finding that the number of borderline scores was highest for management planning skills and examination skills. Management plans are clearly more difficult to make in an unfamiliar hospital where the SpR is uncertain of the different treatment or admission protocols, available facilities such as hydrotherapy, joint injection clinics, combined clinics and drug monitoring systems, or appropriate consultants for referrals to other specialities. Assessment of SpRs in their own hospital, by a visiting external examiner or local consultant, may be more appropriate in future. However, some allowance was made for these difficulties and there may, therefore, be a real need to ensure that SpRs improve their ability to make short- and long-term management plans for new and follow-up patients and avoid postponing important decisions until the patient is seen by someone else at their next visit. Management plans for in-patients and out-patients should be discussed regularly and incorporated into training days. Similarly, trainees may benefit from more practical sessions teaching examination skills. As assessment drives learning and proof of competence is increasingly required, future assessments should include these competencies.
The majority of trainees and consultants involved felt that this assessment of the varied and important out-patient skills was worthwhile and should be repeated. The consensus of opinion is that such assessments should be carried out in the familiar setting of the SpR's hospital. Whole clinics may be assessed with results standing alone or as part of a portfolio. Individual consultations, in a variety of settings, may be observed as a Mini-CEX or referred to by patients, nurses or colleagues during a 360-degree assessment. Video-recorded consultations may play an increasing role in the assessment of rheumatology SpRs. The OSCE remains remarkably popular with SpRs and consultants in this region despite the extra resources required to run it.
Whilst we acknowledge that, if looked at in isolation, there are weaknesses in terms of the validity of our approach, we would argue that the assessment described should be viewed as one of a selection of assessments utilized by the WMRSTC. Evidence clearly supports the validity and reliability of utilizing multiple assessments and a need to include a programme of integrated assessments within a training scheme [15]. Thus, during a 4-yr training period, SpRs in the West Midlands will have been assessed by at least one OSCE, one mock interview (essentially a viva voce examination), one assessed presentation and the out-patient assessment described here, as well as informally by educational supervisors and via their log book. Results will inform the RITA process and reflect not only individual trainee performance but also the appropriateness of the assessment and the quality of the training provided. Within the West Midlands, we believe that perhaps the most important step we have made in rheumatology SpR assessment has been to establish the culture of observed practice of trainees.
West Midlands trainees’ ongoing experience of competence-based assessments should equip them with the skills required to perform well in GMC-imposed assessments, exit examinations and consultant appraisals and revalidation in the future. However they are scrutinized, out-patient clinic competencies will continue to be essential skills in the armoury of a consultant rheumatologist.
|
|
Acknowledgments |
|---|
The following were the members of the WMRSTC at the time of the assessment described: I. Rowe, M. Allen, D. Mulherin, C. Buckley, S. Rigby, K. Grindulis. Trainee representative: K. Douglas.
No conflict of interest has been declared by the authors.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
- Calman KC, Temple JG, Naysmith R, Cairncross RG, Bennet SJ. Reforming higher specialist training in the United Kingdom – a step along the continuum of medical education. Med Educ 1999;33:28–33.[CrossRef][Web of Science][Medline]
- Tunbridge M, Dickinson D, Swan P. Conference of Postgraduate Deans of the United Kingdom. Outcomes of assessments of registrars in the medical specialities. Clin Med 2004;4:66–8.[Medline]
- Curriculum for rheumatology. Joint Committee on Higher Medical Training, January 2003 (http://www.jchmt.org.uk/rheum/curr_rheumatology.pdf).
- Hassell AB. Assessment of specialist registrars in rheumatology: experience of an objective structured clinical examination (OSCE). Rheumatology 2002;41:1323–8.
[Abstract/Free Full Text] - Wragg A, Wade W, Fuller G, Lowan G, Mills P; Royal College of Physicians, London. Assessing the performance of specialist registrars. Clin Med 2003;3:131–4.[Web of Science][Medline]
- Norcini JJ, Blank LL, Arnold GK, Kimball HR. The Mini-CEX (Clinical Evaluation Exercise): a preliminary investigation. Ann Intern Med 1995;123:795–9.
[Abstract/Free Full Text] - Potter TB, Palmer RG. 360-Degree assessment in a multidisciplinary team setting. Rheumatology 2003;42:1404–7.
[Abstract/Free Full Text] - Milburn A. Improving rewards for NHS consultants and modernising medical careers. London: Department of Health 2003 (http://www.dh.gov.uk/assetRoot/04/08/20/84/04082084.pdf).
- Postgraduate Medical Education and Training Board. Vision, values and aims. Department of Health, 2004.
- Cohen L, Manion L, Morrison K, eds. Validity and reliability in research methods in education. 5th edn. London: Routledge Falmer, 2001.
- Tate P, Foulkes J, Neighbour R, Campion P, Field S. Assessing physicians interpersonal skills via videotaped encounters: a new approach for the MRCGP. J Health Commun 1999;4:143–52.[CrossRef][Web of Science][Medline]
- Campion P, Foulkes J, Neighbour R, Tate P. Patient centredness in the MRCGP video examination: analysis of a large cohort. BMJ 2002;325:691–2.
[Free Full Text] - Harden RMcG, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J 1975;1:447–51.
[Abstract/Free Full Text] - Norcini JJ, Blank LL, Duffy FD, Fortna GS. The Mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003;138:476–81.
[Abstract/Free Full Text] - van der Vleuten CP, Schuwirth, LW. Assessing professional competence: from methods to programmes. Med Educ 2005;39:309–17.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||