[Previous] [Contents] [Next]

Quality Management in Local Authority Educational Psychology Services 2: Self-evaluation Toolkit

Introduction

1.1 The toolkit has been developed by the profession and the two training universities of Dundee and Strathclyde to support self-evaluation. Educational psychologists from across Scotland, representing all professional levels, have been directly involved in the consultation and development of this document.

The toolkit has been designed to provide a more detailed look at self-evaluation. It outlines a systematic approach for educational psychology services to use when evaluating the effectiveness with which they deliver their services.

Evidence produced through the use of this or other evaluation models can contribute to overall evaluations. For example, evidence produced through the use of models such as Charter Mark can contribute to evaluations made using measures and indicators within this framework and vice versa. Many services in the past have used the Psychological Service Quality Assurance Performance Indicators. The new framework has built on previous models and taken account of recent developments relating to quality assurance.

The process of self-evaluation

1.2 The process of self-evaluation helps services to:

• recognise the positive impact their work has on stakeholders for example, children and young people, parents, families, educational staff and partner agencies;
• highlight levels of service which need to be maintained or where improvement is needed and where they should be working to achieve excellence;
• identify what they can do to make things better for stakeholders; and
• inform stakeholders about the quality of services in the area.

The process of using self-evaluation for improvement is based around three key questions:

How are we doing?
How do we know?
What are we going to do now?

The following provides more detailed information about how a service might go about answering these three questions.

How are we doing?

1.3 Practitioners have always reflected on the quality of the services they provide. By working together to gather information to evaluate the impact of services, educational psychologists can come to a shared view of how well they are doing and how they can make things better. To ensure the quality of provision is maintained and improved, service staff also need to evaluate how effectively services are delivered and managed, and how well they plan for improvement.

Making such evaluations is dependent on a shared understanding between service providers and stakeholders of what constitutes high-quality outcomes and processes. The quality and performance indicators in this document set out to support the development of such an understanding by all those concerned with delivering or evaluating educational psychology services. Answering the question How are we doing? requires services to summarise their impact on external and internal stakeholders, and indeed the community as a whole.

How do we know?

1.4 Self-evaluation involves:

• a broad view of performance across the six high-level questions
• a closer look at particular aspects of work.

Forming a broad view

1.5 Practitioners within a particular service can use the quality indicators to form a broad view of quality across their service. Using evidence that has been gathered in the normal course of their work, they can identify broad strengths and weaknesses. This will enable them to make an immediate evaluation of areas of major strength, or areas where more attention is required.

Taking a closer look

1.6 Often it may not be manageable or even helpful to try to evaluate every aspect of the service in an in-depth way at the same time. In order to have a greater understanding of the effectiveness of particular aspects of practice, managers and practitioners can take a closer look at them. The stimulus to take a closer look could derive from a range of issues that includes:

• the decision of a small group of practitioners to follow up a particular issue or area on which they have been working in order to find ways to evaluate and improve what they are doing;
• an area of priority identified during the broad view of self-evaluation;
• a national priority or a local improvement objective;
• a survey of the views of stakeholders in an area;
• the outcomes of an inspection or inquiry; and
• research findings which have implications for the quality of the service.

This means that specific groups of practitioners can also carry out self-evaluation by asking themselves focused questions such as:

• How are educational psychologists contributing to improving outcomes for children and young people looked after and accommodated by the local authority?
• How well is the service complying with legislation and responsiveness to guidance and codes of practice?

It could also focus on specific themes by asking such questions as:

• How well does the service consult with children and young people, and take account of their views in the delivery of the service?
• How effectively do educational psychologists work in teams?

By encouraging practitioners to structure their discussion of, and reflection on, their work, the quality and performance indicator framework can support improvement by individuals and teams as well as at operational and managerial levels.

How do we gather evidence?

1.7 There are a number of sources of evidence which can inform a service about how well the needs of stakeholders are being met and what differences are being made. By using a range of indicators and sources of evidence, an holistic view of quality can be determined. There are four key sources of evidence from which evaluations can ultimately be made. These are:

• performance data
• relevant documentation
• stakeholders views and feedback
• direct observations of practice.

1.8 These sources of evidence are complementary. A single source may not provide enough evidence to enable a reliable or robust evaluation to be made. The principle has been tried and tested over many years by independent external evaluators and entails the scrutiny of one source of evidence, backed up by another and corroborated by other lines of enquiry. An example would be an evaluation based on an analysis of the educational psychology service development plan, the implementation of which was then fleshed out in discussion with senior educational psychology staff and further corroborated in discussion with stakeholders such as parents, carers and families. Through such a process of robust self-evaluation it would be possible to monitor the outcomes of putting policy into practice, and, by doing so, evaluate the benefits of policy in meeting the needs of stakeholders.

Performance data

1.9 Examples of performance data would be statistical information relating to local and national objectives.

Relevant documentation

1.10 Examples of relevant documentation could be a statement of the service’s vision, values and aims, service development plans, standards and quality reports.

Stakeholders’ views

1.11 Information can be collected systematically when stakeholders are accessing and using a service, or at the end their involvement with a service. Services should also have procedures for surveying stakeholders’ views using questionnaires or focus groups.

1.12 Whatever approach is used, gathering information from stakeholders is an essential part of the self-evaluation process. Without it, services will find it very difficult to understand the impact of their work on stakeholders. It is almost impossible to answer the high-level questions with any degree of confidence without including the views of stakeholders.

Direct observation

1.13 The educational psychology service is embedded and managed within local authority systems and delivers its statutory functions within that local context. The service works collaboratively to support and empower others to improve outcomes for stakeholders. This approach is supported by evidence-based practice that recognises the effectiveness of a process of collaborative, contextual assessment and intervention.

1.14 To measure the impacts and outcomes of the involvement of the service, it is necessary to consider the relationships the service has and the context in which its services are delivered. The impact of the service is concerned with the immediate and current experiences of stakeholders: it has a focus on customer service. Outcomes are longer-term and largely statistical measures relating to improvement programmes and achievement data. Outcomes are concerned with the service’s success in achieving its objectives and also its contributions to the education authority’s objectives.

1.15 The service operates at the different levels of delivery – on behalf of the individual, school/provision, local authority or within a national context – at different times. For example, with regards to individual case work, sometimes the service will be in direct contact with a child or young person and, at other times, the service will work through a parent or teacher to improve outcomes for children and young people. The impact of the psychological service can be direct or indirect. The work of the service can be directly observed in a range of contexts as illustrated opposite:

What are we going to do now?

1.16 It is important, when undertaking any form of self-evaluation, to keep focused on the end purpose, which is improving outcomes for stakeholders. The answer to, What are we going to do now? must therefore always be a plan for action, which will make a positive difference to the stakeholders who use the service.

Planning for improvement

1.17 Effective and robust self-evaluation provides a strong basis for good planning. Such planning for the service takes place within the framework of the authority’s strategic and operational plans. The service will have its own improvement plan or equivalent.Self-evaluation can support these planning processes and ensure that plans are built on robust, evidence-based knowledge of the quality of impact and outcomes. The results of the planning will then be made publicly available in the standards and quality report.

Action for improvement

1.18 Whether, and how, the outcomes of self-evaluation are reported depends on the purpose and the level of the work done. The purpose of reporting should help decision-making about how to make future improvements, and also inform stakeholders.

1.19 The following examples outline a range of reports which result from self-evaluation activity.

• A service decides to find out the views of young people who have been directly involved with its work. A group of young people are selected to be part of this project. Parental permission is sought before the young people are approached by the service. Permission and agreement are also sought directly from the young people. A series of focus groups with young people in various establishments takes place. A list of ten key questions are used in the focus groups. The results of the questions are analysed. The findings are used to improve the guidelines for direct involvement with young people. The service also feeds back the results of the evaluation to the stakeholders involved and other interested parties.
• A service decides to evaluate the contribution it makes to multi-agency meetings and to explore the impact a pilot collaborative approach has on children and their families. This is done using focus groups including education staff, parents and children. Questionnaires are also sent out to all schools. As a result of a positive self-evaluation, the service uses the approach across the whole service. The service also feeds back the results of the evaluation to the stakeholders involved and other interested parties.
• A service evaluates its contribution to national and local authority priorities by building these priorities into its development plan. It takes feedback from all stakeholders, including education service management, on the tasks achieved and quality of the input provided by the service to a range of local authority working groups and initiatives. The service publishes impacts and outcomes in its yearly standards and quality report.

1.20 As a service works across a wide range of stakeholders, meeting the needs of stakeholders can involve a number of professional and user groups. It is important, therefore, that the outcomes of self-evaluation and plans for improvement feed into the appropriate planning structures. Depending on the work undertaken, the outcomes may result in priorities being included in future plans. The following list gives some examples:

• individualised educational programme for a child or young person;
• support service or school development plan;
• plan at cluster or neighbourhood level;
• local health improvement plan;
• education authority improvement plan; and
• children’s services plan.

1.21 Whatever the planning structure(s) for taking forward improvement, it will help if a manageable number of priorities is selected for which a service can identify specific, achievable, measurable and time-bound targets. Even where a service can see how improvements can be made across a number of aspects, the service may wish to focus on those of greatest concern. It will make more impact on stakeholders if a manageable number of priorities are taken forward effectively. In some cases, small changes in practice identified by practitioners can have a significant positive impact for many different stakeholders. It is not always necessary for self-evaluation to result in major changes or reviews of practice. Often these changes are part of day-to-day professional working which focuses on continuous improvement.

1.22 The process of self-evaluation as described above is summarised by the following diagram.

The toolkit has been designed to support educational psychology services and practioners in addressing these questions. In the first column headed How are we doing? questions have been developed by the profession to explore this theme. In the second column headed How do we know? examples of four main sources of evidence, have been outlined:

• performance data;
• relevant documentation;
• stakeholders’ views and feedback; and
• direct observation.

In Appendix II, examples of relevant documentation for quality indicators have been listed for consideration.

In the third column headed What are we going to do now? a series of statements are provided for a service to review. By examing these statements, the service should be able to assess how it can improve the overall quality of the educational psychology service.

It is essential that self-evaluation should not be viewed as a discrete summative task but one that is formative and will allow the service to improve outcomes for stakeholders. To achieve this end, there is a final sub-section in the third column headed We will? By completing this sub-section in an open and robust fashion, the service should be assisted in formulating an action plan for improvement (see figure 1).

[Previous] [Contents] [Next]