Structure
The framework for self-evaluation described in this section (and summarised in Figure 2 on page 13) provides a systematic approach for councils to use for self-evaluation. The framework does not assume a particular organisational structure within the council. It can be applied whether responsibility for all education functions is located within a single department or is distributed across a number of different council departments.
The framework has been developed in accordance with the principles of the Excellence Model of the EFQM and can be used in conjunction with other quality models such as IiP, Charter Mark and ISO 9000. See Appendix I. Evidence produced through the use of other models can contribute to overall evaluations.
The framework is based on six high-level questions which can be answered by evaluating the quality of education systematically across ten Key Areas.
The inputs and processes outlined in Key Areas 5-9 contribute to the outcomes and impact identified in Key Areas 1-4.
Key Areas 1-9 contain indicators and measures, each with themes which focus on specific aspects of the service being evaluated.
Key Area 10 provides guidance which can be used to evaluate the degree of confidence reached by those carrying out the evaluation that the council has the capacity to continue to improve. This evaluation will take into account the evaluations of other Key Areas, its track record in improvements to date and significant aspects of its internal and external contexts.
Using the framework in self-evaluation
Councils can use the framework to provide a systematic structure for self-evaluation.
They can also use the framework as a diagnostic tool. By looking first at the outcomes and impact of the education they provide (Key Areas 1-4), they can identify key issues for further exploration, observation and analysis using the tools provided within Key Areas 5-9.
Finally, councils are encouraged to arrive at an evaluation of their overall capacity for improvement, using the guidance in Key Area 10.
Using the framework in inspection
When engaging in inspection and reporting activities, HMIE will focus on specific Key Areas and indicators selected from the overall framework. Some of these Key Areas and indicators may be used across all or most councils, while others may be used only in particular contexts and will depend on the scope of the inspection. The scope of each inspection will be set using existing evaluations and evidence to determine what is already known about the council. It will build on the self-evaluation already carried out by authorities. The inspection team will always seek to answer the questions:
Indicators drawn from the Key Areas relating to these questions will normally form the core of the set of indicators used during inspection. Beyond this, inspection activities might focus on specific Key Areas and indicators selected from the rest of the framework. The selection will depend on decisions taken during scoping activities and may be determined following evaluation using the core set of indicators.
HMIE will use the guidance in Key Area 10 to arrive at an evaluation of an individual council’s overall capacity for improvement.
Answering the high-level questions
What key outcomes have we achieved?
Key Area 1 focuses on the overall performance of a council in relation to its education functions. It provides a structure for councils to use when evaluating their success as organisations in delivering demonstrable outcomes as outlined in legislation and programmes for development. For example, these could include outcomes in delivering national priorities for education, improvements in attainment in 5-14 levels, Scottish Qualifications Authority (SQA) results or in achievement through The Duke of Edinburgh’s Award, programmes accredited through the Award Scheme Development and Accreditation Network (ASDAN) or Young Quality Scot Awards.
Also important are the improvements made in relation to a council’s strategic priorities and distinctive vision, values, aims and targets. These targets will include locally determined targets for social, education, health and care objectives in the context of children’s services and other plans. Evidence would appear in reports on strategic and operational plans and as observable improvements. In education, examples could include achievement of targets for participation in adult learning, cultural experiences, residential experiences and outdoor education.
Evidence of demonstrable outcomes includes trends over time and other aggregated data which provide indications of the success of a council in improving the quality of the service it provides, both overall and in comparison with other councils.
Key Area 1 also focuses on the extent to which councils fulfil their statutory duties, meet legislative requirements, follow appropriate codes of practice and are financially secure. Examples include legislation such as the Race Relations Amendment Act, the Disability Discrimination Act and the Local Government Scotland Act. Fulfilling these duties and following relevant codes of practice are key aspects of overall performance.
How well do we meet the needs of our stakeholders?
Key Areas 2, 3 and 4 focus on the impact on key groups of stakeholders of the council’s delivery of education processes. In other words, these areas look at the benefits which stakeholders derive from the council’s education services. Stakeholders include learners who are in direct receipt of the service together with those who support them and who also have a significant interest in the delivery of high-quality services, for example, parents and families.
Stakeholders also include the staff within the council who receive care, support, training and opportunities for involvement in decision-making and career development. Their motivation and satisfaction is of considerable importance if the council is to operate effectively. Evaluations would draw upon the views of staff together with other information, such as rates of absence or retention. The quality of the care and support given to staff can be evaluated by considering evidence of teamworking, rates of involvement in professional development and achievement of qualifications and accreditation of the service as a whole by national schemes like IiP and Scotland’s Health at Work (SHAW).
Finally, stakeholders include members of the community. These stakeholders include those from the immediate local community on whose lives and experiences the service has a demonstrable impact. They also include the wider national and international community. The service may make a significant contribution to thinking and practice within a particular area of work. For example, staff may be actively involved in working groups, lead national initiatives and share innovative practice with others. Evidence of the council’s impact beyond its immediate context may come from colleagues and peers, from published reports or from other media sources.
When evaluating the impact of the council on the range of stakeholders, consideration should be given to the overall balance of strengths and weaknesses. Those carrying out the evaluation need to be alert to the possibility that in order to bring about improvements in one area of impact, the quality of service delivery in another area could diminish, with an overall detrimental effect on service to stakeholders.
Evaluations of the quality of impact in these Key Areas will take into account direct observation and quantitative data together with evidence of stakeholders’ views, in order to arrive at overall judgements of the impact of the service on its key stakeholders. Where evidence from these sources is conflicting or indicates significant weaknesses, evaluators should follow audit trails to identify and address the possible causes, using indicators from other Key Areas in the quality framework. The focus of Key Areas 2, 3 and 4 will be on evaluating the impact on specific groups of learners and other stakeholders, and arriving at holistic evaluations of the overall impact on their experiences.
There may be apparent discrepancies between the evaluations given within Key Area 1 and those given within Key Areas 2, 3 or 4. An improving authority may provide considerable evidence of impact on its stakeholders. Significant changes in measurable outcomes such as those included in Key Area 1, however, may take longer to become apparent. For example, young people may indicate through their views and behaviour, that they are learning more effectively. They may show increased motivation and less evidence of disaffection. It may take time, however, for improvements such as these to translate into increases in attainment levels or reductions in offending. In other words, there may be a time lag between improvements in terms of benefits for stakeholders and the overall measurable outcomes of the council. Evaluations of impact may therefore be noticeably more positive than evaluations of outcomes. The opposite may also be true. An organisation such as a high performing school might deliver and maintain very good examination results. However, the experiences of young people may be evaluated less highly. They may not receive sufficient stimulation or challenge or have poor relationships with their peers or with staff. They may have few opportunities for, or little commitment to independent learning. In such circumstances, overall outcomes may be very positive but evaluations of impact on learners less so.
How good is our delivery of education processes?
Key Area 5 focuses on the work of the council in relation to its education functions. This relates to the quality of the curriculum, learning and teaching and relationships with stakeholders. For example, education and health might use the indicators to help them evaluate their processes for working together to reduce rates of teenage pregnancy, and for supporting young women who do become pregnant but wish to continue their education. Key Area 5 provides indicators for evaluating the quality of the authority’s services and the extent to which these take account of the need to promote and ensure inclusion and equality.
Key Area 5 also looks at the quality of the council’s processes for improving its work and how well it works with its key stakeholders and partners to achieve this. Quality improvement should be built into the core work of the organisation, not be a bolt on. Evidence of the effectiveness of the authority’s key processes will be seen in the impact they have on stakeholders and in its overall performance.
How good is our management?
Key Areas 6, 7 and 8 focus on the operational management activities necessary to ensure effective service delivery and to deliver Best Value. These activities include the council’s arrangements for developing and updating policies, for involving its stakeholders, for operational planning, for managing staff, finance, information and resources and for developing productive partnerships. Strengths and weaknesses in these areas will normally affect the quality of the key processes delivered (Key Area 5), their impact on stakeholders (Key Areas 2, 3 and 4) and the performance of the council as a whole in relation to its education functions (Key Area 1).
How good is our leadership?
Key Area 9 focuses on the strategic direction of the council in relation to its education functions. It looks at its corporate purpose and the expression and delivery of its aspirations by means of strategic planning with partner agencies and the community.
This Key Area considers the quality of leadership and direction at strategic level, and within teams and organisational units. It looks at the quality of interactions with people within the organisation and with partners in other agencies. It also focuses on the role of leaders in bringing about change and improvement, including innovation and, where necessary, step-change.
Strengths and weaknesses in leadership will reflect the extent to which leaders make a difference to the quality of outcomes achieved by the authority as a whole and to the benefits derived by stakeholders. Evaluation will take account of the impact of leadership on the experiences of learners and other key stakeholders and the extent and quality of the outcomes demonstrated.
The indicators in Key Area 9 should be capable of being applied at more than one level within the organisation. They could, for example, be applied to the work of the chief executive and elected members but also to the work of those who lead specific education departments, services or teams, including project teams and working groups.
What is our capacity for improvement?
Judgement of an organisation’s capacity for improvement takes into account the evaluations arrived at in Key Areas 1-9, with particular reference to the quality of the leadership and management of the council and overall impact and outcomes. The council’s focus on improvement and its track record in bringing about improvement are particularly important, as is the accuracy of its self-evaluation, which is used as the basis for planned improvements. The judgement also takes into account any significant aspects of the authority's internal or external context, for example, impending retirements of senior staff, plans to restructure or significant changes in funding. The judgement reviews the past, and looks forward to the future.
Judgements of a council’s capacity for improvement could be expressed in terms of a degree of confidence that it has the capacity to continue to improve. The judgement may be that the evaluators are confident that the council has the capacity to continue to improve. This judgement would be made when highly effective leadership and management have sustained high levels of quality and brought about major improvements to outcomes and impact on learners and other stakeholders. Evidence at the time would indicate that these improvements were sustainable and that improvement would continue. No significant changes in the internal or external context of the organisation would be apparent or predicted at the time the judgement was made.
When there are reservations about one or more of these aspects, the use of other terminology would be more appropriate. For example, those carrying out the evaluation might have only ‘limited confidence’, or indeed, ‘no confidence’, that the council has the capacity to continue to improve. It would be important for evaluators to note the nature of their reservations, for example, by pointing to specific aspects of the council’s work or its current or future context.
Figure 2 - The QMIE2 framework