The framework described in this section provides a systematic approach for educational psychology services to use for self-evaluation. The framework does not assume a particular organisational structure.
The framework has been developed in accordance with the principles of the Excellence Model of the EFQM (see Appendix 1) and evidence produced through the use of other models can contribute to overall evaluations.
The framework is based on six high-level questions which can be answered by evaluating the quality of educational systematically across ten Key Areas.
What key outcomes have we achieved?
Key Area 1. Key performance outcomes
How well do we meet the needs of our stakeholders?
Key Area 2. Impact on service users
Key Area 3. Impact on staff
Key Area 4. Impact on the community
How good is our delivery of key processes?
Key Area 5. Delivery of key processes
How good is our management?
Key Area 6. Policy development and planning
Key Area 7. Management and support of staff
Key Area 8. Partnerships and resources
How good is our leadership?
Key Area 9. Leadership
What is our capacity for improvement?
Key Area 10. Capacity for improvement
The inputs and processes outlined in Key Areas 5-9 contribute to the outcomes and impact identification in Key Areas 1-4.
Key Areas 1-9 contain indicators and measures, each with themes which focus on specific aspects of the service being evaluated.
Key Area 10 provides guidance which can be used to evaluated the degree of confidence reached by those carrying out the evaluation that the service has the capacity to continue to improve. This evaluation will take into account the evaluations of other Key Areas, its track record in improvements to date and significant aspects of its internal and external contexts.
Educational psychology services can use the framework to provide a systematic structure for self-evaluation.
They can also use the framework as a diagnostic tool. By looking first at the outcomes and impact of the service they provide (Key Areas 1-4), they can identify key issues for further exploration, observation and analysis using the tools provided within Key Areas 5-9.
Finally, educational psychology services are encouraged to arrive at an evaluation of their overall capacity for improvement, using the guidance in Key Area 10.
Using the framework in inspection
When engaging in inspection and reporting activities, HMIE will focus on specific Key Areas and indicators selected from the overall framework. Some of these Key Areas and indicators may be used across all or most services, while others may be used only in particular contexts and will depend on the scope of the inspection. The inspection team will always seek to answer the questions:
Indicators drawn from the Key Areas relating to these questions will normally form the core of the set of indicators used during inspection. Beyond this, inspection activities might focus on specific Key Areas and indicators selected from the rest of the framework. The selection will depend on decisions taken during scoping activities and may be determined following evaluation using the core set of indicators.
HMIE will use the guidance in Key Area 10 to arrive at an evaluation of an individual service’s overall capacity for improvement.
What key outcomes have we achieved?
Key Area 1 focuses on the overall performance of a service in relation to its key functions. It provides a structure for services to use when evaluating their success as organisations in delivering demonstrable outcomes as outlined in legislation and programmes for development. For example, these could include outcomes in delivering national priorities for education, improvements in achievement for children and young people with additional support needs, reducing exclusion and promoting more effective transitions for children and young people entering school education or moving into the post-school sector. Also important are the improvements made in relation to the service’s strategic priorities and distinctive vision, values, aims and targets. These targets may include locally determined targets for social, education, wellbeing and care objectives in the context of Children’s Services and other plans.
Key Area 1 also focuses on the extent to which services fulfil their statutory duties, meet legislative requirements, follow appropriate codes of practice and are financially secure. The educational psychological service is a statutory service with a wide range of functions which are set down in Section 4 of the Education (Scotland) Act 1980 and in subsequent amendments including the Education (Additional support for Learning) (Scotland) Act 2004. The Children Scotland Act (1995) extended children’s rights in a number of areas and the Standards in Scotland’s School etc Act (2000) places additional duties on local authorities which have implications for the educational psychological service. The Scottish Executive report Review of Provision of Educational Psychology Services in Scotland published in 2002, makes specific recommendations about the way that educational psychological services are expected to perform in order to fulfil their statutory duties. Fulfiling these duties and following relevant codes of practice are key aspects of overall performance.
How well do we meet the needs of our stakeholders?
Key Areas 2, 3 and 4 focus on the impact on key groups of stakeholders of the service’s delivery of key processes. In other words, these areas look at the benefits which stakeholders derive from the educational psychology service. Stakeholders include children, young people, families, schools, social workers and health professionals who are in direct receipt of the service together with those who support them and who also have a significant interest in the delivery of high-quality services, for example, parents and families.
Stakeholders also include the staff within the service who receive care, support, training and opportunities for involvement in decision-making and career development. Their motivation and satisfaction is of considerable importance if the service is to operate effectively. Evaluations would draw upon the views of staff together with other information, such as rates of absence or retention. The quality of the care and support given to staff can be evaluated by considering evidence of teamworking, rates of involvement in professional development and achievement of qualification and accreditation of the service as a whole by national schemes like liP and Scotland’s Health at Work (SHAW).
Finally, stakeholders include members of the community. These stakeholders include those from the immediate local community on whose lives and experiences the service has a demonstrable impact. They also include the wider national and international community. The service may make a significant contribution to thinking and practice within a particular area of work. For example, staff may be actively involved in working groups, lead national initiatives and share innovative practice with others. Evidence of the service’s impact beyond its immediate context may come from colleagues and peers, from published reports or from other media sources.
When evaluating the impact of the service on the range of stakeholders, consideration should be given to the overall balance of strengths and weaknesses. Those carrying out the evaluation need to be alert to the possibility that in order to bring about improvements in one area of impact, the quality of service delivery in another area could diminish, with an overall detrimental effect on service to stakeholders.
Evaluations of the quality of impact in these Key Areas will take into account direct observation and quantitative data together with evidence of stakeholders’ views, in order to arrive at overall judgements of the impact of the service on its key stakeholders. Where evidence from these sources is conflicting or indicates significant weaknesses, evaluators should follow audit trails to identify and address the possible causes, using indicators from other Key Areas in the quality framework. The focus of Key Areas 2, 3 and 4 will be on evaluating the impact on specific groups of stakeholders including children, young people and families, and arriving at holistic evaluations of the overall impact on their experiences.
There may be apparent discrepancies between the evaluations given within Key Area 1 and those given within Key Areas 2, 3 and 4. An improving service may provide considerable evidence of impact on its stakeholders. Significant changes in measurable outcomes such as those included in Key Area 1, however, may take longer to become apparent. For example, young people may indicate through their views and behaviour, that they are learning more effectively. They may show increased motivation and less evidence of disaffection. It may take time, however, for improvements such as these to translate into increases in attainment levels or reductions in offending. In other words, there may be a time lag between improvements in terms of benefits for stakeholders and the overall measurable outcomes of the service. Evaluations of impact may therefore be noticeable more positive than evaluations of outcomes. The opposite may also be true. An organisation such as a high-performing school might deliver and maintain very good examination results. However, the experiences of young people may be evaluated less highly. They may not receive sufficient stimulation or challenge or have poor relationships with their peers or with staff. They may have few opportunities for, or little commitment to independent learning. In such circumstances, overall outcomes may be very positive but evaluations of impact on learners less so.
How good is our delivery of education processes?
Key Area 5 focuses on the work of the service in relation to its key functions. This relates to the quality of the delivery of the key educational psychological services. The Scottish Executive report Review of Provision of Educational Psychology Services in Scotland published in 2002, makes specific recommendations about the way that educational psychological services are expected to perform in order to fulfil their statutory duties.
The Review of Provision of Educational Psychology Services in Scotland identifies a number of key services which underpin the delivery of the statutory duties of an educational psychology service. These are looked at in turn in Key Area 5. The way in which these services are delivered will impact on performance (Key Area 1) and stakeholders (Key Area 2)
Key Area 5 also looks at the quality of the service’s processes for improving its work and how well it works with its key stakeholders and partners to achieve this. Quality improvement should be built into the core work of the organisation, it should not be a bolt on. Evidence of the effectiveness of the service’s key processes will be seen in the impact they have on stakeholders and in its overall performance.
How good is our management?
Key Areas 6, 7 and 8 focus on the operational management activities necessary to ensure effective service delivery and to deliver Best Value. These activities include the service’s arrangements for developing and updating policies, for involving its stakeholders, for operational planning, for managing staff, finance, information and resources and for developing productive partnerships. Strengths and weaknesses in these areas will normally affect the quality of the key processes delivered (Key Area 5), their impact on stakeholders (Key Areas 2, 3 and 4) and the performances of the service as a whole in relation to its key functions (Key Area 1).
How good is our leadership?
Key Area 9 focuses on the strategic direction of the service in relation to its key functions. It looks at its corporate purpose and the expression and delivery of its aspirations by means of strategic planning with partner agencies and the community.
This Key Area 9 considers the quality of leadership and direction at strategic level, and within teams and across the service as a whole. It looks at the quality of interactions with people within the organisation and with partners in other agencies. It also focuses on the role of leaders in bringing about change and improvement, including innovation and, where necessary, step-change.
Strengths and weaknesses in leadership will reflect the extent to which leaders make a difference to the quality of outcomes achieved by the service, and the authority as a whole and to the benefits derived by stakeholders. Evaluation will take account of the impact of leadership on the experiences of children, young people and families, and other key stakeholders and the extent and quality of the outcomes demonstrated.
The indicators in Key Area 9 should be capable of being applied at more than one level within the organisation. They could, for example, be applied to the work of the principal educational psychologist and senior officers with responsibility for the service but also to the work of those who lead specific, teams, including project teams and working groups.
What is our capacity for improvement?
Judgement of an organisation's capacity for improvement takes into account the evaluations arrived at in Key Areas 1-9, with particular reference to the quality of the leadership and management of the service and overall impact and outcomes. The service’s focus on improvement and its track record in bringing about improvement are particularly important, as is the accuracy of its self-evaluation, which is used as the basis for planned improvements. The judgement also takes into account any significant aspects of the service’s internal or external context, for example, impending retirements of senior staff, plans to restructure or significant changes in funding. The judgement reviews the past, and looks forward to the future.
Judgements of a service’s capacity for improvement could be expressed in terms of a degree of confidence that is has the capacity to continue to improve. The judgement may be that the evaluators are confident that the service has the capacity to continue to improve. This judgement would be made when highly effective leadership and management have sustained high levels of quality and brought about major improvements to outcomes and impact on children, young people, families and other stakeholders. Evidence at the time would indicate that these improvements were sustainable and that improvement would continue. No significant changes in the internal or external context of the organisation would be apparent or predicted at the time the judgement was made.
When there are reservations about one or more of these aspects, the use of other terminology would be more appropriate. For example, those carrying out the evaluation might have only ‘limited confidence’, or indeed, ‘no confidence’ that the service has the capacity to improve. It would be important for evaluators to note the nature of their reservations, for example, by pointing to specific aspects of the service’s work or its current or future context.