How Good is Our Community Learning and Development? 2 Self-evaluation for Quality Improvement: page 4

How Good is Our Community Learning and Development? 2 Self-evaluation for Quality Improvement

Part 2: Performance and Quality Indicators

The framework for self-evaluation explained

The framework for self-evaluation described in the following pages provides a systematic approach for CLD partnerships and providers to use when evaluating the effectiveness with which they deliver their services. The framework does not assume a particular organisational structure for CLD within a local authority. It can be applied whether responsibility for all CLD functions is located within a single department or is distributed across a number of different council services or partner agencies.

The framework has been developed in accordance with the principles of the Excellence Model of the European Foundation of Quality Management (EFQM) and can be used in conjunction with other quality models, for example, Investors in People (IiP), Charter Mark and ISO 9000. Evidence produced through the use of other models can contribute to overall evaluations. The diagram in Appendix 1 shows how these models relate to the framework for self-evaluation.

The framework is based on six high-level questions which can be answered by evaluating the quality of CLD provision systematically across ten Key Areas.

What key outcomes have we achieved?
Key Area 1. Key performance outcomes
How well do we meet the needs of our stakeholders?
Key Area 2. Impact on service users
Key Area 3. Impact on staff and volunteers
Key Area 4. Impact on the local and wider community
How good is our delivery of key processes?
Key Area 5. Processes and delivery
How good is our management?
Key Area 6. Policy development and planning
Key Area 7. Management and support of staff
Key Area 8. Partnerships and resources
How good is our leadership?
Key Area 9. Leadership
What is our capacity for improvement?
Key Area 10. Capacity for improvement

The inputs and processes outlined in Key Areas 5-9 contribute to the outcomes and impacts identified in Key Areas 1-4.

Key Areas 1-9 contain indicators and measures, each with themes which focus on specific aspects of the area being evaluated.

Key Area 10 provides guidance which can be used to evaluate the degree of confidence reached by those carrying out the evaluation that the council and its partners have the capacity to continue to improve the quality of CLD provision. The evaluation will take into account their track record in making improvements to date and significant aspects of their internal and external contexts.

Using the framework in self-evaluation

CLD services and partner agencies can use the framework to provide a systematic structure for evaluation. They may choose to address each Key Area in turn or to group Key Areas for evaluation according to their established cycle, whether that is organised at the level of individual services/ departments or corporately across the council.

CLD providers can also use the framework as a diagnostic tool. By looking first at the outcomes and impact of the learning experiences they provide (Key Areas 1-4), they can identify key issues for further exploration, observation and analysis using the tools provided within Key Areas 5-9.

Finally, CLD providers are encouraged to arrive at an evaluation of their overall capacity for improvement, using the guidance in Key Area 10.

Individual organisations can also use the framework for self-evaluation. It is possible to interpret the framework to suit your organisation’s context. For example, you could adapt the indicative theme in 6.3 which reads ‘articulation of operational plans with the community plan and CLD strategy’ to best suit the planning arrangements of your organisation.

Using the framework in inspection

When engaging in inspection and reporting activities, HMIE will focus on specific Key Areas and indicators selected from the overall framework. Some of these Key Areas and indicators may be used across all or most CLD providers, while others may be used only in particular contexts, depending on decisions taken during scoping activities. In some cases, HMIE may also choose to focus on those specific themes from individual indicators which provide the most appropriate tools for use in the context of a particular CLD service. It will be the responsibility of the CLD providers being inspected to provide evidence to support the level of performance indicated within their self-evaluations.

HMIE will use the advice in Key Area 10 to arrive at an evaluation of CLD providers’ overall capacity for improvement.

Answering the high-level questions

What key outcomes have we achieved?

Key Area 1 focuses on the overall performance of CLD provision. It provides a structure for councils to use when considering their success as organisations in delivering national priorities for CLD as set out in guidance and programmes for development, as well as strategic priorities relating to their distinctive vision, values and aims. Key Area 1 also focuses on the extent to which councils fulfil their statutory duties and are financially secure, both of which are key aspects of overall performance. This Key Area takes a broad, long-term perspective which focuses on a council’s successes in improving the quality of CLD, both overall and in comparison with other providers. Individual organisations can also use this indicator to look at their overall performance.

How well do we meet the needs of our stakeholders?

Key Areas 2, 3 and 4 focus on the impact of CLD Provision on key groups of stakeholders. In other words, these areas look at the benefits which stakeholders derive from the services delivered by CLD partners. Evaluation in these Key Areas will take into account stakeholders’ views, together with evidence from direct observation and quantitative data, in order to arrive at overall judgements of the CLD services’ impact on its key stakeholders. Where evidence from these sources is conflicting or indicates significant weaknesses, CLD providers should follow audit trails to identify and address the possible causes, using indicators from other Key Areas in the quality framework. This investigation and analysis could focus on a number of issues, for example, the quality of the council’s CLD provision, its processes for communicating with, and involving stakeholders, and/or its management of information. The focus will be on evaluating the impact on specific groups of participants, and arriving at holistic evaluations of the overall impact on their experiences.

How good is our delivery of key processes?

Key Area 5 focuses on the work of the provider in relation to its CLD functions. It articulates clearly with the competences required of professionally qualified staff in Scotland and is consistent with the National Occupational Standards that have been developed for specific aspects of CLD. In other words, it builds upon the processes that effective practitioners employ to achieve maximum impact in their work.

How good is our management?

Key Areas 6, 7 and 8 focus on the operational activities necessary to ensure effective and efficient delivery and Best Value. These activities include the provider’s processes for developing and updating policies, for involving its stakeholders, for operational planning, for managing staff, finance and resources and for developing productive partnerships. Strengths and weaknesses in these areas will affect the quality of the processes delivered within Key Area 5, their impact on stakeholders as described in Key Areas 2, 3 and 4 and the performance of the providers as a whole in relation to CLD provision in Key Area 1.

How good is our leadership?

Key Area 9 focuses on the strategic direction of the council and its partners in relation to CLD. It looks at their corporate purpose and the expression and delivery of their aspirations by means of strategic planning with partner agencies and the community. It considers the quality of leadership at strategic level, and within teams and organisational units. Individual organisations can also use this indicator to consider the effectiveness of their leadership.

What is our capacity for improvement?

Judgement of an organisation’s ability for improvement takes into account the evaluations arrived at in Key Areas 1-9, with particular reference to the quality of the leadership and management of the CLD provider and overall impact and outcomes. The CLD provider’s focus on improvement and its track record in bringing about improvement are particularly important, as is the accuracy of its self-evaluation, which is used as the basis for planned improvements. The judgement also takes into account any significant aspects of the CLD provider or partnership’s internal or external context, for example, impending retirements of senior staff, plans to restructure or significant changes in funding. The judgement reviews the past, and looks forward to the future.

Judgements of a CLD provider’s capacity for improvement could be expressed in terms of a degree of confidence that it has the capacity to continue to improve. The judgement may be that the evaluators are confident that the provider or partnership has the capacity to continue to improve. This judgement would be made when highly effective leadership and management have brought about major improvements to outcomes and impact on participants and other stakeholders. Evidence at the time would indicate that these improvements were sustainable and that improvement would continue. No significant changes in the internal or external context of the organisation would be apparent or predicted at the time the judgement was made.

When there are reservations about one or more of these aspects, the use of other terminology would be more appropriate. For example, those carrying out the evaluation might have only ‘limited confidence’, or indeed, ‘no confidence’, that the provider or partnership has the capacity to continue to improve. It would be important for evaluators to note the nature of their reservations, for example, by pointing to specific aspects of the work or its current or future context.

Key Area 1. Key performance outcomes

No.	Performance Indicator	Measures
What key outcomes have we achieved?
1.1	Improvements in performance	Performance data and measures showing trends over time Quality of learning/development activities provided by individual centres, projects and programmes Performance against aims, objectives and targets
1.2	Fulfilment of statutory duties	Financial performance Compliance with legislation and responsiveness to guidance and codes of practice

Key Area 2. Impact on service users

No.	Quality Indicator	Indicative Themes
How well do we meet the needs of our stakeholders?
2.1	Impact on participants	Qualitative and quantitative data that demonstrate the extent to which learners are: included and participating achieving and attaining progressing. Extent to which participants report that their learning experiences enable them to become: successful learners, confident individuals, responsible citizens and effective contributors safe, nurtured, healthy, achieving, active, respected and responsible and included.

Key Area 3. Impact on staff and volunteers

No.	Quality Indicator	Indicative Themes
How well do we meet the needs of our stakeholders?
3.1	Impact on paid and voluntary staff	Quantitative and qualitative data, together with reports from voluntary and paid staff that demonstrate the extent to which they: feel motivated, confident and valued improve their practice through training and development activities have positive experiences of the quality of central services and the work of partner agencies work in teams effectively.

Key Area 4. Impact on the community

No.	Quality Indicator	Indicative Themes
How well do we meet the needs of our stakeholders?
4.1	Impact on the local community	Quantitative and qualitative data, together with reports from community groups, that demonstrates the extent to which community groups have: confident, skilled and active community members active and influential roles in local and wider decision making developed local services, where appropriate, in response to priority needs effective planning, management and evaluation arrangements ensured that they are inclusive and value social and cultural diversity productive networks and relationships with other agencies and organisations.
4.2	Impact on the wider community	Evaluations of quantitative and qualitative data that demonstrate the extent to which CLD provide: encourage and support creativity and innovation learn from and adopt leading-edge practice influence wider policy or practice anticipate and respond rapidly and flexibly to change.

Key Area 5. Processes and delivery

No.	Quality Indicator	Indicative Themes
How good is our delivery of key processes?
5.1	Opportunities for people in the community	Range and coherence of opportunities and programmes Responsiveness to participants’ needs and aspirations Promotion of learning and development opportunities for all participants Arrangements for participant progression
5.2	Context for learning/ development	Guiding prospective participants Relationships with participants that support learning/development Environment for learning/development Celebrating success
5.3	Planning for learning/ development	Recognising prior learning Identifying individual and group learning/development needs and aspirations Planning sessions and learning activities Recording achievement and progression
5.4	Facilitating learning/ development	Range and appropriateness of methods used by staff Range, appropriateness and use of resources by staff Level of challenge, pace and balance of activities Assessment as part of learning/development Promoting independence in learning/development
5.5	Participant learning/ development	Motivation and engagement Use of resources by participants Learner contribution to learning and assessment Participant reflection on their own learning/development
5.6	Engaging with communities and other stakeholders to identify and plan to meet their own needs	Arrangements for identifying community needs and aspirations, including literacies needs Knowledge and understanding of community needs Plans informed by community needs and aspirations Assessment and recording of progress and achievement
5.7	Developing skills and confidence for community engagement	Support for community members and volunteers Building effective relationships with participants Training and development for community members and volunteers Feedback on progress Progression of community members and volunteers
5.8	Assisting communities to exercise power and	Supporting community organisations in managing change Networking within the community influence to achieve Community influence and representation outcomes Community engagement in community planning in line with community engagement standards Assisting communities to provide and manage services Celebrating success
5.9	Inclusion, equality and fairness	Inclusion of excluded communities, groups and individuals Addressing barriers to participation Access to specialist services to meet specific needs Promoting inclusion, equality, fairness and positive attitudes to social and cultural diversity Compliance with equalities legislation
5.10	Improving services	Evaluating information from participants and other stakeholders Evaluating outcomes and impact Arrangements for reflective practice and self-evaluation Planning for improvement and monitoring progress Reporting progress to stakeholders

Key Area 6. Policy development and planning

No.	Quality Indicator	Indicative Themes
How good is our management?
6.1	Policy review and development	Range and appropriateness of policies Coherence with corporate policy Links to strategic vision, values and aims Managing, evaluating and updating policies
6.2	Participation of service users and other stakeholders	Involvement in policy development Communication and consultation about aims, provision and performance Active participation in the work of CLD providers
6.3	Operational planning	Developing, implementing and evaluating plans Articulation of operational plans with the community plan and CLD strategy Use of performance information Staff and partner engagement in planning and evaluation Planning for sustainability

Key Area 7. Management and support of paid and voluntary staff

No.	Quality Indicator	Indicative Themes
How good is our management?
7.1	Sufficiency, recruitment and retention	Identifying and meeting human resource needs Recruitment, appointment and induction procedures Care and welfare Equality and fairness in recruitment and promotion Recognition
7.2	Deployment and teamwork	Appropriateness and clarity of remits Deployment to achieve planned priorities Teamworking Communication and involvement in decision-making
7.3	Development and support	Processes for staff review, support and supervision Training and development Joint training with staff from partner agencies

Key Area 8. Partnerships and resources

No.	Quality Indicator	Indicative Themes
How good is our management?
8.1	Partnership working	Clarity of purposes and aims Service level agreements, roles and remits Working across agencies and disciplines Staff roles in partnerships
8.2	Financial management	Setting budgets and enterprise in securing funding Range and implementation of financial procedures and controls Processes for collecting, evaluating and communicating financial information Providing Best Value
8.3	Resource and risk management	Accommodation Resources and equipment Health and safety Arrangements for ensuring the protection and welfare of children, young people and vulnerable adults
8.4	Information systems	Data collection, storage and retrieval Linkages between, and sharing of, information Processes for analysing, evaluating and using information

Key Area 9. Leadership

No.	Quality Indicator	Indicative Themes
How good is our leadership?
9.1	Vision, values and aims	Appropriateness and coherence with corporate and community vision, values and aims Sharing and sustaining the vision Promotion of positive attitudes to social and cultural diversity
9.2	Leadership and direction	Strategic planning and communication Strategic deployment of resources Evaluation of risk
9.3	Developing people and partnerships	Developing leadership capacity Building and sustaining relationships Teamwork and partnerships
9.4	Leadership of change and improvement	Support and challenge Creativity, innovation and step change Continuous improvement

Key Area 10. Capacity for improvement

What is our capacity for improvement?

This last of the high-level questions requires a global judgement based on evidence and evaluations of all Key Areas. In answering this question CLD providers should also take into account contextual issues such as, impending retirements of senior staff, plans to restructure and significant changes in funding. They should also consider their ability to respond quickly to change and to be creative and innovative in the pursuit of excellence.

The council CLD service and partners should be able to make a statement with the following components:

"The Council is confident/not confident that the evidence and evaluation to date indicates that:

overall improvements have been made to key outcomes and impacts on stakeholders;
leadership and management are effective; and
quality improvement arrangements are effective and the council CLD service has the capacity to continue improving."

The levels of confidence expressed for each component may be different and may include some reservations or caveats, but should lead to an overall statement of confidence in the council’s capacity to improve in relation to CLD. For example, the statement could say,
"The council is confident that the evidence and evaluation to date indicates that:

overall improvements have been made to achieving key outcomes and to meeting the needs of service users but the achievements of young people through youth work requires improvement;
leadership and management are currently effective but key posts will become vacant in the near future; and
quality improvement arrangements are effective in all areas except adult literacies and the council has demonstrated the capacity to continue improving."

Individual CLD providers can also use this indicator to form a global judgement on their capacity for improvement.

[Previous] [Contents] [Next]