The CLD performance and quality indicators are a core component of an overall quality-improvement framework. The framework has been developed to focus on six high-level questions which encompass the key dimensions of CLD:
1. What key outcomes have we achieved?
2. How well do we meet the needs of our stakeholders?
3. How good is our delivery of key processes?
4. How good is our management?
5. How good is our leadership?
6. What is our capacity for improvement?
Self-evaluation, external inspection and review of CLD provision will focus principally on the experiences of young people, adults and the community and the impact CLD provision has on peoples’ lives.
Each indicator is organised to help us make a judgement about the strengths and weaknesses in that key area. The themes structure the process of reflecting on and discussing participants’ outcomes and our practice with fellow practitioners. Relating practice to the illustrations helps to identify strengths and weaknesses. By identifying and describing the strengths and weaknesses in our area we can focus on what we need to do to make improvements.
To answer this overarching question in the context of CLD provision and have a full understanding of the impact of the delivery of a service, we need to gather information. Practitioners have always reflected on the quality of the services they provide. By working together to evaluate the impact of our services, we can come to a shared view of how well we are doing and how we can make things better. To ensure the quality of provision is maintained and improved, we also need to evaluate how effectively services are delivered and managed, and how well we plan for improvement.
Making such evaluations is dependent on a shared understanding within and across service providers and stakeholders of what constitutes high-quality outcomes and processes. The quality and performance indicators in this document set out to support the development of a shared understanding for all those concerned with delivering or evaluating CLD provision. The indicators are organised round the high-level questions above and, taken together, answer the overarching question, How good is our community learning and development?. Answering this question requires us to summarise the impact of CLD provision on young people, adults and the community and the quality of outcomes for them. The overarching framework of indicators, measures and themes is given below.
Forming a broad view
Practitioners within a particular CLD service or in multi-disciplinary groups such as the local CLD partnership or ALN partnership can form a broad view by scanning across all the quality indicators. Using evidence that has been gathered in the normal course of work, they can identify broad strengths and weaknesses. This will give an immediate impression of areas of major strength, or areas where more attention is required.
Taking a closer look
It may not always be helpful or manageable to try to evaluate every aspect of provision in depth at the same time. In order to have a greater understanding of the effectiveness of particular aspects of practice, managers and practitioners can also take a closer look at them. The stimulus to take a closer look could derive from a range of issues that include:
This means specific groups of practitioners can also carry out self-evaluation by asking themselves such questions as:
It could also focus on specific themes by asking such questions as:
By encouraging practitioners to structure their discussion and reflection on the work they are doing, the quality and performance indicator framework can support improvement by individuals and small teams as well as at operational and managerial levels.
In order to plan effectively we need to know how well we are doing. Effective self-evaluation provides a strong basis for good planning.
Planning for CLD strategies and ALN within a framework of Community Planning has encouraged practitioners to work together to identify local objectives to improve CLD provision. Self-evaluation can support these planning processes and ensure that plans are built on robust, evidence-based knowledge of the quality of outcomes, provision and the needs of the community.
We have a number of sources of evidence which can tell us how well we are meeting the needs of stakeholders and what difference we are making.
There are, essentially, four main sources of evidence from which evaluations can ultimately be made. These are:
These sources of evidence are complementary. No single source can meaningfully provide enough evidence on its own to enable a reliable or robust evaluation to be made. The principle of triangulation has been tried and tested over many years by Her Majesty’s Inspectorate of Education (HMIE), by other independent external evaluators and by CLD services.
In essence, triangulation involves the scrutiny of one source of evidence, backed up by another and corroborated by a third line of enquiry. An example would be an evaluation based on the examination of a key youth policy document, the implementation of which was then fleshed out in discussion with senior CLD staff and further corroborated in discussion with relevant stakeholders such as young people, youth workers and school staff. Through this process of triangulation, it should then be possible to observe the outcome of putting policy into practice, and, through this, to evaluate the impact of policy in meeting the needs of key stakeholders.
Performance outcomes and relevant documentation
Since 2000, partnerships in all local authority areas in Scotland have developed community learning strategies with a range of performance objectives. This initiative was supported by the publication, Learning Evaluation and Planning (LEAP) that encouraged new approaches to planning that were needs-led and outcome-focused. Since 2003, the Local Government Act has required local authorities and other key stakeholders to develop Community Plans. Government guidance in February 2004 on the future development of CLD was issued to Community Planning Partnerships. This guidance requires local authorities to lead the development of CLD strategies and articulates three National Priorities for CLD:
In addition, funding for area regeneration through Social Inclusion Partnerships and the Better Neighbourhood Services Fund was replaced by the Community Regeneration Fund in 2004. Community Planning Partnerships now deliver regeneration programmes on the basis of Regeneration Outcome Agreements. As a result, CLD providers are working to achieve a range of outcomes and targets set out in key planning documents.
Information on the extent to which these outcomes are being achieved can be collated as part of the self-evaluation process and will give a clear indication of the progress being made. Any difficulties in achieving targets set can be followed up by a taking a closer look to find out what the barriers to achieving the objectives are.
Other objectives will refer to improving the quality of the provision and will not always lend themselves easily to the collation of performance data. The quality indicators will help us evaluate progress in these areas.
This information is gathered primarily from young people, adults and the community along with any advocates on behalf of vulnerable or excluded members of the community. It may also include information from surveys of the general public or voluntary or community organisations working in the area.
Information can be collected systematically when young people, adults and community organisations are using a service, or when they end their involvement with a service. We may also, from time to time, specifically survey participants or meet groups of participants to find out their views.
Whatever approach we decide to use, gathering information from participants is an essential part of the self-evaluation process. Without it, it is very difficult to understand the impact of our work on young people, adults and the community and to know if we are doing the right things. It is almost impossible to answer the high-level questions with any degree of confidence without including the views of participants.
Stakeholders also include paid and voluntary staff and partners with whom we work. Their views should also be sought.
The indicators can also be used to help us to form a view on the level of effectiveness in a particular aspect of practice. Inspectors from HMIE evaluate using six levels of effectiveness. Practitioners engaged in self-evaluation can use the same six-point scale. Awarding levels is a professional skill rather than a technical process and there are many ways in which provision can merit a particular evaluation.
The following describe key characteristics at each of six levels.
An evaluation of excellent will apply to performance which is a model of its type. The outcomes for young people, adults and the wider community along with their experience of provision will be of a very high quality. An evaluation of excellent will represent an outstanding standard of performance, which will exemplify very best practice and is worth disseminating beyond the service or area. It will imply these very high levels of performance are sustainable and will be maintained.
An evaluation of very good will apply to performance characterised by major strengths. There will be very few areas for improvement and any that do exist will not significantly diminish the experience of young people, adults and the wider community. While an evaluation of very good will represent a high standard of performance, it is a standard that should be achievable by all. It will imply that it is fully appropriate to continue the CLD provision without significant adjustment. However, there will be an expectation that CLD practitioners will take opportunities to improve and strive to raise performance to excellent.
An evaluation of good will apply to performance characterised by important strengths which, taken together, clearly outweigh any areas for improvement. An evaluation of good will represent a standard of performance in which the strengths have a significant positive impact. However, the quality of outcomes and experiences of young people, adults and the community will be diminished in some way by aspects where improvement is required. It will imply that the CLD services should seek to improve further the areas of important strength, but take action to address the areas for improvement.
An evaluation of adequate will apply to performance characterised by strengths, which just outweigh weaknesses. An evaluation of adequate will indicate that young people, adults and the community have access to a basic level of service. It represents a standard where the strengths have a positive impact on the experiences of young people, adults and the wider community. However, while the weaknesses will not be important enough to have a substantially adverse impact, they will constrain the overall quality of outcomes and experiences. It will imply that CLD practitioners should take action to address areas of weakness while building on strengths.
An evaluation of weak will apply to performance which has some strengths but where there will be important weaknesses. In general, an evaluation of weak may be arrived at in a number of circumstances. While there may be some strengths, the important weaknesses, either individually or collectively, are sufficient to diminish the experiences of young people, adults and the community in substantial ways. It may imply that some young people or adults may not have their needs met or be left at risk unless action is taken. It will imply the need for structured and planned action on the part of the CLD service and partners.
An evaluation of unsatisfactory will apply when there are major weaknesses in performance in critical aspects requiring immediate remedial action. The outcomes and experiences of young people, adults and the community will be unsatisfactory in significant respects. In almost all cases, practitioners responsible for provision evaluated as unsatisfactory will require support from senior managers in planning and carrying out the necessary actions to effect improvement. This may involve working alongside other staff or agencies. Urgent action will be required to ensure the young people, adults and the community have their needs met.
Each quality indicator in this document contains a set of illustrations at the levels of very good and weak. Illustrations are intended to provide examples, not to be fully comprehensive. Outcomes and practice not described in the illustrations will still contribute to the evaluation.
At whatever level within the service we are using the quality indicators to look at our practice and answer the question, How good is our community learning and development? we should now be in a position to plan for improvement and report on how well we are doing. It is important when undertaking any form of self-evaluation to keep focused on the end purpose, which is making things better for young people, adults and the community. The answer to, What are we going to do now? must therefore always be a plan for action, which will make a positive difference in the lives of people who use our services.
Whether and how the outcomes of self-evaluation are reported depends on the purpose and the level of the work done. The purpose of reporting is primarily to help decision-making about how to make improvements, and to inform stakeholders, such as young people, adults, the wider community, the CLD partnership, the ALN partnership, the public, elected members of councils, voluntary and community organisations, and police and health boards.
The following examples of different types of reports may both be the outcomes of self-evaluation activity.
The report should:
Meeting the needs of our stakeholders involves a number of professional groups. Developments to make things better for young people, adults and the community can involve a wide range of practitioners in a number of organisations, including voluntary or community organisations. It is important therefore that the outcomes of self-evaluation and plans for improvement feed into the appropriate planning structures and are shared across services and providers. Depending on the work undertaken, the outcomes may result in priorities being included in a number of planning structures. The following list gives some examples.
Whatever the planning structure(s) for taking forward improvement, it will help if we select a manageable number of priorities for which we can identify specific, achievable, measurable and time-bound targets. Even where we can see how improvements can be made across a number of aspects, we may wish to focus on those of greatest concern. It will make more impact on services if a manageable number of priorities are taken forward, with a notable impact on the outcomes and experiences of young people, adults and the community. In some cases small changes in practice identified by a group of practitioners in an area can have a significant positive impact. It is not always necessary for self-evaluation to result in major changes or reviews of practice.
It is always useful to have an external measure of how we are doing. HMIE carry out inspections and reviews to evaluate the quality of CLD provision. Information from inspections can form part of the evidence for local self-evaluation. HMIE has a responsibility to evaluate how well CLD provision meets the needs of young people, adults and the community across all areas of Scotland. Inspection by HMIE covers much of the same ground as local self-evaluation. Inspectors will gather evidence, make professional evaluations using the quality and performance indicators and answer at least some of the six high-level questions outlined above. They will write a report which will address the questions outlined, based on evaluations of the evidence gathered and summarise the key strengths and areas requiring further development in an area. Any self-evaluation which has recently been carried out locally will contribute to the inspection process. HMIE is also commissioned by Scottish Executive departments to undertake reviews of national voluntary sector organisations engaged in CLD delivery. These reviews operate in a similar way to CLD inspections.