How to go about doing an audit: stages in the cycle

auditcycle

This cycle equates roughly with the following questions:

  • What am I trying to do?
  • Am I doing it?
  • Why am I doing it?
  • What can I do to make things better?
  • Have I made things better?

Stage 1. Defining the problem and the purpose of the audit: what can I audit?

You can audit any aspect of your practice. You may well already have an issue in mind, which is why you are considering an audit. If not, then the suggested starting point is a systematic examination of your existing practice procedures. This could cover the premises, patient referral pathways, initial contact and booking, patient demographics and presenting conditions, recording, patient management, diagnostic information, monitoring or treatment, adverse events, treatment outcomes, treatment end points, onward referral – in fact any area of practice. Within any of these areas look for things that you think could, or should, be done better. The British Acupuncture Council’s (BAcC) ‘The Standards of Practice for Acupuncture’(SPA)  would provide a useful framework for this. You should consider these points when making your choice:

  • Benefit for patients
  • Benefit for your practice/yourself
  • Potential for improvement
  • Will hold your interest
  • Something that’s under your control; hence it’s feasible for you to take steps to change it
  • It can be measured/recorded without too much trouble

Having identified a problem/topic you should state explicitly what are the aims of the audit: this will drive the rest of the stages

Stage 2. Criteria and standards

Criteria are definitions of good practice. Standards state how high the bar will be set in meeting each criterion. The more vital it is that a criterion is met, the higher the standard, so they help to prioritise your actions. The idea is that you will compare standards achieved on these criteria for your own practice vs those defined as good/best practice: but where are such standards to be found? Specifying what should be done and how it should be done does not sit well with independent practitioners following their own path. Defined standards are much more likely to be found in institutional healthcare settings where there are set procedures and targets.

If good practice standards don’t already exist for acupuncture then can you develop  your own? Without established quality standards you may have to rely largely on your own experience and common sense, but you could improve considerably on that by discussing it with colleagues (especially esteemed and experienced ones), even doing this as a group activity.

The Health Quality Improvement Partnership (Dixon and Pearce 2011)  describes four different tools to use for developing standards:
Nominal group process: to identify priorities among a list of ideas agreed by consensus by a team of people
Delphi process: very similar to the nominal group
Critical appraisal: read and evaluate existing evidence
Benchmarking: analyse the performance of a leader in the field

These all have serious difficulties for most acupuncturists, who do not normally operate in teams and may find it hard to locate any relevant evidence or persuade master practitioners to divulge their own performance data (even if they had any).
Rees (1997) adopts a much more relaxed approach, suited to CAM professions that are not set up (burdened, one may say) with explicit standards for all the various processes involved in delivering healthcare. She suggests the literature, colleagues, professional bodies and your own training knowledge as sources of possible information on good practice in respect of the chosen issue. She does not focus on standards but does state that there should be explicit criteria of good practice: these form the basis for deciding whether or not you’ve actually got a problem and need to take action. This simple diagram describes her approach:

reesapproach


The BAcC has drawn up a list of practice standards for its members [link to SPA document on website]. These are intended to help characterise traditional acupuncture and to provide members with best practice guidelines and ideas for reflective practice and audit. For the most part they are not defined specifically enough to be used as audit criteria, nor quantitatively enough to determine audit standards. However, they provide the necessary substrate from which practitioners can choose their own area(s) of interest and then develop their own best practice criteria.

The research base may indicate that acupuncture is effective for a given illness but it rarely helps with saying what specific approach is best for a specific clinical situation.

At the end of the day you will be the one deciding on the criteria and standards, for it is your particular situation and issue that are being addressed. However, taking soundings from colleagues is certainly worthwhile and some may already have collected their own data. We know that some acupuncturists have done one-off analyses of their records and others may do this more routinely. The latter may be doing this for marketing purposes but most likely also for internal consumption and practice improvement.

Stage 3.  Collect data from your practice

Most audits use survey methods. This commonly involves retrospective analysis of case notes or other written/electronic records, though questionnaires and interviews (e.g. of patients, colleagues, other medical professionals) are also possibilities. The methods should be appropriate for the data required, and also easy and practical. They should be applied as rigorously as possible, but the consequences of imperfections are not as serious as for research studies. Details can be found in any research methods textbook.
You will collect information about the criteria identified in Stage 2. There can be many criteria or as few as one: it is important to do only as much as is feasible for you. Mostly you can design your own data collection form for the purpose. If the total number of relevant cases/episodes/events etc is small then you may use all of them, but probably you will want to take a sample, to reduce the workload. For retrospective analysis the sample should be reasonably representative of the whole, so you could take it randomly (e.g. using random numbers) or systematically (e.g. every tenth case). If it’s going to be prospective then you can simply take the first n cases. How large should n be? You are aiming for good enough estimates, not precise ones, so not very large, though bigger and over a longer time period if you’re measuring something that varies a lot.
Questionnaires are more difficult to do well, so make them simple or use an off-the-peg one or get some help with it.
Pilot data collection processes, forms and questionnaires to improve their reliability. Although you will not need ethical approval you may need to get patient consent for questionnaires, and you must maintain confidentiality.

Step 4. Compare your data against the set standards and consider how to improve what you do

If you were able to set standards in Step 2 then now you can compare your own data against these. Does your practice fall short in any of the recorded criteria and can you explain or justify these? Nobody else need see this information, so you can be honest with yourself. On the other hand you may wish to discuss your findings with colleagues before deciding how to proceed. If significant shortcomings are identified then you need to consider what can be changed that might improve the situation. If you were not able in Step 2 to set explicit standards then try to do so now, with the information you’ve just gathered.
Name the specific practice improvements that are needed.

Step 5. Implement changes

Devise an action plan.
Carry out the plan.
This is easier said than done but there’s no point getting this far if you don’t try to make improvements

Step 6. Monitor the effects of changes

If you stop after Step 5 then you won’t know whether you have met your objectives. Give the changes time to filter through and then repeat the data collection, as in Step 3. If you now meet the standards then congratulations, and the audit may stop here (though you may wish to repeat monitor from time to time). If not, then it is time for further thought:

  • Are the standards too demanding?
  • Are the criteria not the most appropriate?
  • Are different/extra actions needed?
  • Is more effort/persistence/finesse required in implementing the changes?

Whatever the situation, the solution is often to repeat the cycle after altering some parameter or other.

Step 7. Repeat the audit cycle

Whether or not you are happy with the outcome after the first cycle you may wish to continue in order to further develop your ideas about good practice.