The process of self-evaluation employs three main types of measures and indicators:
The following commentary outlines the purpose and function of each within the self-evaluation process.
Quality Indicators
All education authorities are now well experienced in the use of QIs. They are important management tools that assist an organisation in checking whether its aims and expectations are being met. They can be used to evaluate the organisation’s overall performance and effectiveness, or to examine specific aspects of performance in more detail. This assists the evaluators to develop a clear picture of the effectiveness of such complex organisations as educational establishments or local authorities. When used most effectively, therefore, the QIs become objective reference points against which judgements can be made by experienced practitioners. Consequently, areas of strength and those in which improvement is required can be identified and action taken. This is the essence of effective self-evaluation. The aim for all authorities should be to develop and embed a culture at all levels, where systematic, robust and ongoing self-evaluation permeates every establishment and every service.
The prudent evaluator will always recognise the complexity of the system that is being reviewed, and the interplay of organisational factors, competing priorities and local circumstances. It is important that QIs should not be used as simple checklists or in isolation from other evaluation techniques or processes. Self-evaluation should be able to focus on the organisation’s mission or central purpose, which must always be the key reference point for contextualising the QIs and suggesting features to examine.
Local authorities are complex organisations and are subject to change. The QIs in the new QMIE2 framework provide a flexible structure for evaluating key inputs such as:
The QIs in the framework also address the delivery of key educational process and the impact on stakeholders such as:
Ethos indicators and perception measures
Ethos indicators and perception measures provide a structure for evaluating the views or perceptions of key stakeholders, who include:
Ethos indicators and perception measures are used to evaluate the impact on stakeholders of education services provided by the authority. They draw on evidence from a range of sources such as stakeholder surveys, satisfaction questionnaires, focus group discussions and a range of quantitative indicators such as learners’ attendance rates, staff absence rates and attendance by parents at meetings.
Performance indicators and outcome measures
Performance indicators measure the key outcomes of the service. They measure the extent to which the authority has achieved its vision, values, aims and targets, as expressed within its local service and/or improvement plans. They also measure the extent to which the service is delivering those outcomes established through the national priorities and national improvement agenda. When judging performance in these respects, authorities will rely on increasingly sophisticated analysis of a wide range of quantitative management and performance data such as examination results, Statutory Performance Indicators (SPIs) and benchmarking with comparator schools, services and authorities.
It is important not to concentrate on one type of indicator at the expense of others. While good processes do, in the main, tend to lead to good outcomes, the direct relationship may not be quite as simple to observe. For example, there may well be a time lag between improving a process and being able to identify evidence which demonstrates that this has had an impact or has led to improved outcomes. Equally, where outcomes are weaker, it is important to evaluate which processes require to be improved to impact positively on these outcomes. A further and very important step is to benchmark those inputs which are being used in the process, for example staffing levels, costs and time allocations, so that performance can be compared with that of similar organisations and Best Value judgements made.
Finally, it may be misleading to look at any one key area of the framework or at isolated QIs without cross-referencing these with others in order to corroborate evaluations. A much more considered and holistic approach is essential to develop a clear picture of the authority’s effectiveness.
The self-evaluation cycle
In developing a systematic and manageable approach to self-evaluation, it is likely that each authority will plan its work across a rolling period or cycle. In the course of this cycle each authority might aim to evaluate the evidence available to it in order to judge the effectiveness of its performance against the six high-level questions:
Through systematically keeping these high-level questions and their component parts under review over the lifetime of the self-evaluation cycle, the authority should be able to identify, celebrate and build on areas of strength and address areas of underperformance, thereby identifying key priorities for improvement. It is expected that, in addition to using national frameworks of performance and quality indicators, authorities will wish to use a range of other tools and measures, for example, through their involvement in quality schemes such as Charter Mark, IiP or EFQM, through benchmarking activities in Best Value, or by using their own local measures and targets.
Scoping the self-evaluation programme
The self-evaluation programme needs to be planned and resourced systematically. One way to decide on the most relevant and appropriate activities across the cycle is to carry out a scoping exercise, which takes account of the following:
The scoping exercise should establish how the six high-level questions and their component parts are to be addressed across the self-evaluation cycle and the indicators and measures to be used in each case. However, the impact of meeting the needs of children and families and adult learners and the achievement of key outcomes need to be kept under close review on a regular basis.
The rolling programme should be subject to review every year and should, if appropriate, be adjusted to take account of such factors as new inspection evidence, Best Value audits, the outcomes of improvement plans and activities, changes to national or local policy and rising or falling trends across a range of performance measures, particularly those most closely associated with learners’ attainment and achievement. It is important to view the self-evaluation programme as an ongoing, organic and dynamic process with a clear focus on impact, outcomes, continuous improvement and the pursuit of excellence.
The external inspection process
An inspection of the education functions of a council is a major exercise which can be carried out through:
Taking a high-level view and taking a closer look
To answer the six high-level questions, authorities will wish to take a proportionate approach to taking a high-level view and taking a closer look in terms of using relevant indicators and/or measures. In reaching an evaluation of performance across a service, some indicators/measures may only be used to gain a high-level view, while others will be used to take a closer, more focused look at specific areas of service delivery. A balance of high-level views and taking-a-closer-look activities, using appropriate indicators and measures, should allow an authority to:
On a regular basis, authority personnel can take a high-level view by scanning across selected indicators and outcome measures and assigning one of the six levels to each, on the basis of considered professional judgement, using evidence that has been gathered in the normal course of work. This will provide an ongoing impression of areas of perceived major strength and of those where more attention is required. The focus and emphasis should always be on impact and outcomes, which can then direct appropriate audit trails into selected inputs and processes. It is neither necessary nor practicable to audit all inputs and processes on an annual basis.
It is important to bear in mind that:
In addition to taking this broad, high-level view, authorities will also wish to take a closer look at specific areas, which might include:
It is properly a matter for each authority in its scrutiny of performance through the close examination of impact and outcome measures to identify those areas in which a closer look is required.
Identifying the evidence
The process of gathering evidence at establishment or service level is essentially the same as that at authority level. However, there will be considerable differences in the range and sources of evidence that might be used. The following are examples of the sources of evidence which might inform an authority’s self-evaluation, but it is stressed that this list is neither prescriptive nor exhaustive:
Authority self-evaluation:
Performance Data:
Documentation:
Stakeholders’ Views:
Stakeholders might include:
A range of forums, procedures and techniques can be used or accessed to gather stakeholders’ views. These often include:
Direct Observation:
The selection of evidence should be kept to manageable proportions by, for example, a process of sampling. The extent and nature of sampling should be thought through carefully and systematically, taking into account considerations of depth and breadth. For example:
The principle of triangulation has been tried and tested over many years by HMIE, by other independent external evaluators and by authorities themselves. In essence, triangulation involves the scrutiny of one source of evidence, backed up by another and corroborated by a third line of enquiry. A good example would be an evaluation based on the examination of a key policy document, the implementation of which was then discussed with senior authority staff and corroborated in further discussion with relevant stakeholders such as school staff, parents or young people. Through this process of triangulation, it should then be possible to evaluate the impact of policy in meeting the needs of key stakeholders.
The self-evaluation questions
Part 7 contains a few selected practical examples of questions which local authority personnel might ask themselves and their staff to help them arrive at evaluations for certain indicators. These relate to features of an organisation that may be present in whole or in part if that particular theme is being delivered effectively. The suggested questions are designed to support the evaluator in the search for evidence.
The questions provided are intended to be neither comprehensive nor prescriptive. They are not designed to be used as a checklist. It is properly a matter for each authority to determine for each indicator the relevant questions to ask and the appropriate features to look for.
In asking these or other relevant questions and in considering the Level 2 and Level 5 illustrations of evaluations against the QIs, it is also important to recognise that it is not necessary for an aspect of provision to exhibit all of the features in a particular illustration before it is deemed to be "good practice". It is expected that evaluators will bring their own knowledge and experience to make a significant and important contribution to the evaluation process.
Recording and reporting
The authority will wish to develop a clear and reliable system to record the outcomes of all activities within the self-evaluation programme. The range of such activities will be broad and the management and administration of the programme itself and of its component parts will be important. The evidence from self-evaluation will have a very important place in external inspections, such as INEA2 inspections and Best Value audits. It is important that the whole self-evaluation process is viewed as complementary to the external inspection process, with the authority’s self-evaluation report potentially having a major influence on the scope of the external inspection. The ongoing process of self-evaluation and reporting on outcomes will assist the authority in providing HMIE with an evaluative statement on its performance in addressing the high-level questions at any given time prior to the scoping of an external inspection. Given the ongoing cyclical nature of self-evaluation and the potential for generating statistical and written evidence, maximum use should be made of ICT systems.