Analysis of HMIE Reviews of Quality and Standards in Scotland's Colleges Academic Years 2004/05 to 2007/08

A Report by HM Inspectorate of Education for the Scottish Funding Council

JANUARY 2009

Contents

1. Executive summary

2. Introduction

3. Confidence statements

4. Analysis of subject reviews
4.1 Learning and teaching process
4.1.1 Distribution and analysis of grades awarded by subject area
4.1.2 Strengths and weaknesses
4.2 Learner progress and outcomes
4.2.1 Distribution and analysis of grades awarded by subject area
4.2.2 Strengths and weaknesses
4.3 Other significant factors

5. Analysis of cross-college reviews
5.1 Distribution and analysis of grades by quality element
5.2 Strengths
5.2.1 Analysis of strengths identified in cross-college reviews: Educational provision: design, planning and management
5.2.2 Analysis of strengths identified in cross-college reviews: Learner services, resources and staff
5.2.3 Analysis of strengths identified in cross-college reviews: Quality assurance, improvement and enhancement
5.3 Main points for action
5.3.1 Analysis of main points for action identified in cross-college reviews: Educational provision: design, planning and management
5.3.2 Analysis of main points for action identified in cross-college reviews: Learner services, resources and staff
5.3.3 Analysis of main points for action identified in cross-college reviews: Quality assurance, improvement and enhancement

6. Sector-leading and innovative practice

7. Some key messages from 2004-08 reviews: summary

8. College staff evaluations of the review process
8.1 Evaluations from review phase one
8.2 Evaluations from the follow-through phase

9. Glossary

1. Executive summary

Her Majesty’s Inspectorate of Education (HMIE) carried out external reviews in all 43 of Scotland’s colleges between 2004 and 2008 on behalf of the Scottish Further and Higher Education Funding Council (SFC).

HMIE expressed confidence in the learning and teaching processes in all 43 colleges. In 41 colleges HMIE expressed confidence that learners were progressing well and achieving appropriate outcomes. In three colleges, HMIE stated that it was not confident that the college was managing well and improving the quality of its services for learners. Where a lack of confidence was expressed, SFC required the college to undergo a follow-up review.

Almost all subjects reviewed over the four year review cycle demonstrated either good or very good learning and teaching processes. Learners were enthusiastic about their learning experiences and teaching staff applied their specialist knowledge effectively. The majority of programmes had high retention and attainment rates but on specific programmes they were low. Most learners made good progress in achieving their learning goals, gained appropriate vocational skills and progressed into further study or employment. Weaknesses identified during the review process included ICT not being used sufficiently by teaching staff. A few staff also used too narrow a range of learning and teaching approaches and did not systematically check learners’ understanding or provide them with effective feedback. A few colleges needed to ensure effective systematic profiling of learners’ core skills.

Almost all cross-college grades were either good or very good. Only a few areas reviewed were graded either fair or unsatisfactory. College aims and objectives took account of both local and national government priorities. There was effective leadership and partnership links with stakeholders which helped to widen the range of learners taking part in college programmes. More than a few colleges needed to make improvements in developing their approaches to target setting with outcome-based and measurable plans.

Managers and their staff were approachable and they provided helpful guidance and support for learners. This contributed to high levels of learner success. Almost all colleges had high quality buildings and facilities which provided well for their learners’ needs and for the programmes on which they were studying. Staff and learners had good access to VLE/ICT facilities. There was good provision of assistive technology and accommodation for those learners with impaired mobility and with extended learning support needs. Staff were well qualified and demonstrated a strong commitment to team work. Teaching staff benefitted from effective continuing professional development (CPD) but a few colleges needed to evaluate the effectiveness of their CPD, especially its impact on learning and teaching.

Colleges had developed comprehensive quality improvement systems which helped to improve their learners’ experience. Quality procedures were well understood by staff who were committed to improving the quality of the learners’ experience. More than a few colleges needed to ensure that course team action plans included specific measurable targets to address identified weaknesses.

Through the external review process, 182 examples of sector-leading and innovative practice (SLIP) were identified. HMIE published them on its website and elsewhere to enable them to be replicated, adopted and customised by other colleges.

2. Introduction

The service level agreement between the SFC and HMIE details the number of college external reviews to be delivered each year. Over academic years 2004-05, 2005-06, 2006-07 and 2007-08 (referred to in this report as 2004-08) HMIE carried out reviews in all 43 of Scotland’s colleges.

The review model which was introduced in academic year 2004-05 had two phases. Phase one was similar in all colleges. During it, college reviewers focused on quality elements related to leadership and quality improvement. Subject reviewers focused on the learning and teaching process and learner progress and outcomes. Evaluative activity only focused on the other quality elements during the subject review insofar as they highlighted other significant factors that had an impact (positive or negative) on the quality of learning and teaching or on learner progress and outcomes.

The follow-through phase of the review was a differentiated one where activity was proportionate to the strengths and weaknesses identified during the first phase. These follow-through activities focused on exploring the extent and causes of weaknesses and assisted colleges in considering action plans to address weaknesses. Examples of SLIP were also further explored and described in writing during the follow-though phase.

Summaries of the SLIP examples identified during external reviews, are available in Colleges 2004-08: Short summaries of all sector-leading and innovative practice on the HMIE website at this link: http://www.hmie.gov.uk/GoodPractice/Default.aspx.

Percentages in this report have been rounded up or down to the nearest whole number. This has resulted in totals not adding up to 100% in some calculations and charts.

The four-point scale (very good, good, fair and unsatisfactory) was used in external college reviews over the period 2004-08. The definitions of the grades used in these HMIE college reports are:

Very good major strengths
Good strengths outweigh weaknesses
Fair some important weaknesses
Unsatisfactory major weaknesses

3. Confidence statements

An important aspect of the external review model used in 2004-08 was the inclusion in reports of confidence statements. These statements were additional to the grades for each cross-college quality element and each subject area. The confidence statements applied at the whole-college level to learning and teaching process; learner progress and outcomes; and leadership and quality management.

HMIE issued confidence statements for each college reviewed, covering the three main focal points of review: learning and teaching process; learner progress and outcomes and leadership and quality management. HMIE indicated whether it was confident or not in these three key areas. Confidence could be expressed in the following ways:

• HMIE is confident, unqualified;
• HMIE is confident, qualified by the term "overall" drawing the college’s attention to some inconsistency in practice or outcomes;
• HMIE is confident, qualified by reference to specific points; or
• HMIE is not confident.

HMIE expressed confidence in all 43 colleges reviewed in 2004-08 in relation to learning and teaching process. In three colleges, the statement of confidence was qualified by identification of the college’s need to address weaknesses in specific subject areas.

HMIE expressed confidence in 41 colleges reviewed during 2004-08 in respect of learner progress and outcomes. In two colleges HMIE was not confident that learners were progressing well and achieving appropriate outcomes. In 17 colleges, a total of 27 statements of confidence were qualified. For six colleges these qualifications identified the need to address weaknesses in retention and attainment, in five colleges, they related to low attainment and in one college, to low retention.

HMIE also expressed confidence in 40 colleges reviewed in 2004-08 in respect of leadership and quality management. In 24 colleges, HMIE concluded without qualification that the college being reviewed was managing well and improving the quality of experience for learners. In the other colleges, qualifications related variously to the need to:

• improve the effectiveness of self-evaluation processes;
• develop further some aspects of monitoring, evaluation and improvement; and
• address some important weaknesses in relation to one or more of:
  • guidance;
  • access and inclusion;
  • health and safety;
  • staff; and
  • resources and services to support the learner.

In three colleges, HMIE reported that it was not confident that the college was managing well and improving the quality of its services for learners.

Follow-through phase of review

The aim of the follow-through phase of review was to assist the colleges in preparing agendas for action to improve the quality of the learner experience. It:

• was designed to be proportionate to the strengths and weaknesses identified during the phase one external review;
• enabled reviewers to investigate and write up selected practice associated with some of the key strengths identified during the phase one review, and other examples of innovative or sector-leading practice;
• enabled reviewers and college staff to investigate the causes and extent of any significant weaknesses identified in learning and teaching process, and learner progress and outcomes during the phase one review, drilling down using selected (and occasionally all) A quality elements; and
• enabled external reviewers and college managers to investigate the impact and causes of any significant weaknesses identified in the Leadership and quality management elements.

During the four years, 2004-2008, the follow-through phase of review had become well established. The commentary below for 2007-08 is included in this analysis of reviews to provide a fuller picture of the complete review process.

In all of the 12 colleges reviewed in 2007-08, reviewers in the follow-through phase investigated and wrote up selected good practice associated with some of the key strengths identified during the phase one review, and other sector-leading and innovative examples of good practice. In three of these colleges, follow-through related entirely to SLIP. In six colleges, five to nine examples of SLIP per college were confirmed during the follow-through phase, with three or fewer SLIP examples being confirmed during follow-through in each of the remaining six colleges. A total of 50 SLIP examples were published through the review reports for 2007-08.

During the follow-through phase, reviewers also investigated the causes and extent of weaknesses identified in both subject and cross-college quality elements. In 2007-08 only four out of the 12 colleges required follow-through discussions relating to issues identified during the subject areas reviewed. In most colleges, further discussions were held over issues identified in cross-college areas.

In 2007-08, A7: Learner progress and outcomes accounted for most follow-through activity (20%) related to weaknesses in the subject elements. The quality elements B6: Quality assurance and B7: Quality improvement and enhancement together accounted for 45% of follow-through phase activities related to weaknesses in the cross-college elements. The single quality element that accounted for most activity relating to weaknesses during the follow-through phase of cross-college elements was B7 Quality improvement and enhancement (35%).

Follow-up arrangements and follow-up reviews

Where a review resulted in HMIE expressing confidence in a college, SFC required the college to address weaknesses identified in subject areas and the main points for action associated with the findings on leadership and quality management in the report. Routinely, college progress was monitored by the designated college HMI, who reported any issues that arose to SFC, by exception.

Where a review resulted in HMIE not being able to express confidence in a college, SFC required a follow-up review - usually 18 to 24 months after the publication of the report. Follow-up reviews were bespoke and designed to gather sufficient evidence to allow HMIE to determine whether it could subsequently express confidence in the college.

Over 2004-08, SFC specified a follow-up review in four colleges. There were three such follow-up reviews during 2007-08, and another is scheduled for 2008-09. HMIE expressed confidence in two of the colleges that had follow-up reviews in 2007-08 but is still not confident that one of the colleges is managing well and improving the quality of its services for learners. A further review of this college is scheduled to take place in 2008-09, using the new External quality arrangements for Scotland’s colleges, September 2008.

4. Analysis of subject reviews

Subject reviews under the SFC/HMIE quality framework used during this period concentrated on the learning and teaching process, and learner progress and outcomes.

4.1 Learning and teaching process

This section of the report focuses on the grades awarded for quality element

A5: Learning and teaching process.

4.1.1 Distribution and analysis of grades awarded by subject area

The 18 subject areas reviewed over 2004-2008 were:

Art and design
Business, management and administration (BMA)
Care
Computing and ICT (Information and communications technology)
Construction
Education and training
Engineering
Hairdressing, beauty and complementary therapies (HBCT)
Hospitality and tourism
Land-based industries
Languages and English for Speakers of Other Languages (ESOL)
Media
Nautical studies
Performing arts
Science
Social subjects
Special programmes
Sport and leisure.

Figure 1 shows the range of subject areas covered by the external reviews, and the grades awarded for A5: Learning and teaching process.

Figure 1: The grades awarded for A5: Learning and teaching process by subject area over 2004-08

In 2004-08 HMIE conducted a total of 293 subject reviews in these 18 subject areas. Figure 1 shows that 46% of grades awarded for A5: Learning and teaching process were very good, 53% good and 1% fair. There were no unsatisfactory grades awarded for this quality element over 2004-08.

The learning and teaching process is the core of colleges’ work so it is a positive finding that 99% of grades awarded for A5: Learning and teaching process were good or better. The definition of good in the SFC/HMIE quality framework used during this period was "strengths outweigh weaknesses". Such an evaluation represents a standard of provision in which the strengths have a significantly positive impact. However, the quality of learners’ experiences is diminished by aspects in which improvement is required. In 2004-08, 53% of grades for this quality element were recorded as good. With under half of the grades for A5: Learning and teaching process recorded as very good, colleges should take action to eliminate the weaknesses which were identified in the learning and teaching process and seek to enhance further the areas of strength.

Three subject areas recorded particularly high proportions of very good grades for this quality element during the review period. Special programmes recorded 73% very good grades, Care recorded 70% very good grades and Art and design recorded 65% very good grades.

4.1.2 Strengths and weaknesses

During each external review, subject reviewers recorded strengths and weaknesses identified for each quality element. The most frequently recorded themes for strengths in learning and teaching process were as follows:

• learners were enthusiastic about their learning experiences, showing high levels of engagement and motivation;
• teaching staff applied their specialist knowledge effectively to the teaching and learning process and generally kept their subject knowledge up to date. In doing this they were able to enhance the learning experience for learners;
• effective relationships between staff and learners contributed to enhancing the learner experience;
• learners displayed confidence in using a range of learning resources, including online and other ICT resources;
• staff identified appropriate learning goals for learners and planned activities to ensure learners were able to achieve their learning objectives; and
• staff encouraged learners to reflect on their learning and set targets to improve their learning. This promoted independence and learners taking ownership of their learning goals.

The most common theme from the weaknesses identified related to insufficient use by teaching staff of resources, including ICT. This theme accounted for 20% of weaknesses reported from the colleges reviewed during 2004-08. Others included:

• staff not systematically checking learners’ understanding or providing them with effective feedback;
• too narrow a range of teaching methods being deployed by staff;
• learners not being sufficiently engaged in the learning process; and
• poor or late attendance of learners causing disruption to the teaching and learning process.

4.2 Learner progress and outcomes

This section of the report focuses on the grades awarded for quality element A7: Learner progress and outcomes.

4.2.1 Distribution and analysis of grades awarded by subject area

Figure 2 shows the range of subject areas covered by the reviews, and the grades awarded for A7: Learner progress and outcomes.

Figure 2: The grades awarded for A7: Learner progress and outcomes by subject area over 2004-08

The grades awarded for the quality element A7: Learner progress and outcomes were 29% very good, 58% good and 13% fair.

The grades indicated that out of 293 subject reviews 87% of grades recorded were good or better. However with only 29% of grades recorded as very good, it is clear that the majority of subject areas reviewed for A7: Learner progress and outcomes had weaknesses or important weaknesses that colleges need to address.

4.2.2 Strengths and weaknesses

The most frequently recorded themes from the strengths recorded in A7: Learner progress and outcomes were:

• high levels of retention and attainment on specific programmes;
• learners took advantage of effective progression and transition arrangements into further study or employment; and
• learners generally made good progress in achieving their learning goals and gaining appropriate vocational skills.

The most frequent themes in the weaknesses recorded related to specific attainment and retention issues for individual programmes within each of the subject areas reviewed. Of the total number of weaknesses recorded in A7: Learning progress and outcomes, 83% related to low retention and low attainment in specific programmes and in key units. The next most frequent weakness identified was insufficient development of learners’ core skills, which accounted for 6% of recorded weaknesses.

4.3 Other significant factors

Subject reviewers concentrated on evaluating against the two quality elements A5: Learning and teaching process and A7: Learner progress and outcomes. However, they also commented on factors related to the other elements in the quality framework which had a significant positive or negative impact on the quality of the learners’ experience and outcomes.

In 2004-08, review teams reported 962 other significant factors. Of these 539 (56%) were considered to have had a positive impact (recorded as strengths) and 423 (44%) a negative impact (recorded as weaknesses).

The main positive feature across other significant factors reported during 2004-08 was the impact that high standards of accommodation had on the learning experience. In much of the provision reviewed, a learning environment that was in line with the vocational context met the needs of learners well and prepared them for future employment opportunities.

Other prominent positive features reported in other significant factors were that:

• the use of personal learning plans (PLPs) contributed to effective monitoring of learner progress and to setting appropriate learning targets;
• strong links with partner organisations led to better provision for learners; and
• staff identified the support that learners required and provided a range of appropriate support measures.

The main negative themes in the other significant factors were, in order of frequency:

• accommodation that was poorly designed, with insufficient space for learning activities;
• insufficient access to resources, including ICT resources, so learners’ progress was impeded;
• specific programmes that were poorly designed and did not match the needs of the learner group; and
• insufficient analysis by staff of performance indicator data and poor action planning to improve the learning experience and learning outcomes.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• All colleges have either resolved or are in the process of improving their accommodation to ensure that there is sufficient space for learning activities.
• There is now better access to resources including ICT for learners.
• Programmes have now been redesigned to better meet the needs of learners.

However:

• The majority of colleges still need to improve the analysis by staff of performance indicator data to ensure effective action planning leading to improvements in the learning experience and learning outcomes.

5. Analysis of cross-college reviews

5.1 Distribution and analysis of grades by quality element

The cross-college quality elements in the SFC/HMIE quality framework address the commitment and capacity of the college to support the quality of the learner experience and improve outcomes.

The cross-college elements in the 2004-2008 quality framework were as follows.

B1: Educational leadership, direction and management
B2: Access and inclusion
B3: Guidance and support
B4: Resources and services to support the learner
B5: Staff
B6: Quality assurance
B7: Quality improvement.

Figure 3 shows the distribution of grades awarded for B1-B7.

Figure 3: The cross-college grades awarded in reviews 2004-08

In 2004-08, 295 grades were awarded in the 43 colleges reviewed. The review of Sabhal Mòr Ostaig was bespoke and the only cross-college element evaluated was B1.

The table shows that 48% of grades awarded during the cross-college reviews were very good, 44% good, 8% fair and 1% unsatisfactory. In cross-college reviews over 2004-08, 92% of grades awarded were good or better.

The charts on the following pages show the proportion of grades awarded for each of the quality elements over 2004-08.

Distribution of review grades in cross-college elements in college reviews, 2004-08

B1: Educational leadership, direction and management

B2: Access and inclusion

B3: Guidance and support

B4: Resources and services to support the learner

B5: Staff

B6: Quality assurance

B7: Quality improvement

5.2 Strengths

5.2.1 Analysis of strengths identified in cross-college reviews: Educational provision: design, planning and management

This section brings together the strengths recorded for the quality elements B1: Educational leadership, direction and management and B2: Access and inclusion. In 2004-08 the strengths that were identified most frequently in these two elements were that:

• the senior management team and the principal shared a clear vision with staff and provided effective and enthusiastic leadership;
• college aims and objectives took account of partner needs and both local and government priorities;
• effective partnerships and links with stakeholders helped widen the range of learners taking part in college programmes; and
• college commitment to access and inclusion resulted in clear strategic plans.

5.2.2 Analysis of strengths identified in cross-college reviews: Learner services, resources and staff

This section covers the three quality elements B3: Guidance and support, B4: Resources and services to support the learner and B5: Staff.

The strengths most frequently identified under B3: Guidance and support were in relation to:

• managers who were effective in communicating their commitment to supporting learners;
• staff who were helpful, approachable and supported learners’ needs well;
• colleges that provided effective access to guidance and support to meet learner needs well;
• effective partnership arrangements that enhanced the helpful guidance provided by college staff for learners; and
• the early identification of learning needs, including core skills, that helped to support learners well.

For B4: Resources and services to support the learner, the most common strengths identified were:

• high quality, clean and secure learner accommodation, and resources that provided well for learner needs and for developments in the curriculum;
• facilities, including assistive technologies and accommodation, that provided well for learners with impaired mobility and for those with extended learning support needs;
• the implementation of college ICT strategies resulting in good access to VLE/ICT facilities that met learner needs well; and
• library and quiet learning spaces that catered well for learners and facilitated independent learning.

For B5: Staff, the common strengths identified were:

• CPD provision that met staff training needs effectively;
• staff review processes that identified training needs well;
• staff who were well qualified and had relevant vocational experience; and
• a strong commitment to teamwork among staff.

5.2.3 Analysis of strengths identified in cross-college reviews: Quality assurance, improvement and enhancement

This theme includes the quality elements B6: Quality assurance and B7: Quality improvement. The common strengths were:

• well-developed and comprehensive quality systems that helped improve the learner experience;
• staff who demonstrated good levels of knowledge of quality procedures;
• staff who were committed to improving the quality of learner experience; and
• colleges that had a strategic commitment to self-evaluation and staff who were involved in self-evaluation processes.

5.3 Main points for action

Main points for action are recorded by HMIE only for the cross-college B elements in reviews. Over 2004-08, there were 387 main points for action. Below, they are organised under the same themes as in the previous section:

• educational provision: design, planning and management;
• learner services, resources and staff; and
• quality assurance, improvement and enhancement.

The bar chart shows the distribution of main points for action for each of these themes.

Percentage of main points for action by theme 2004-08

5.3.1 Analysis of main points for action identified in cross-college reviews: Educational provision: design, planning and management

Twenty-nine per cent of all main points for action related to quality elements B1 and B2. The areas in which HMIE most frequently identified main points for action were that colleges should:

• improve communication links across college functions to inform strategic and operational planning; and
• develop approaches to target setting with outcome-based and measurable plans.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• Most colleges have now improved their communication links across college functions to inform strategic and operational planning.

However:

• The majority of colleges have not yet sufficiently developed their approaches to target setting with outcome-based and measurable plans.

5.3.2 Analysis of main points for action identified in cross-college review: Learner services, resources and staff

Forty-four per cent, or almost half of all cross-college main points for action, related to quality elements B3, B4 and B5. The most frequent areas for main points for action related to:

• systematic evaluation of the effectiveness and impact of CPD activities undertaken by staff;
• progressing action plans on building adaptations to support access for all learners;
• the systematic profiling of learners’ core skills; and
• ensuring that all staff are trained in all aspects of child protection and equalities.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• The majority of colleges now systematically evaluate the effectiveness and impact of CPD activities undertaken by staff.
• The majority of colleges now have plans in place to ensure effective physical access by learners to their services.
• The majority of colleges now profile learners’ core skills.
• Most colleges have now ensured that all of their staff are trained in all aspects of child protection and equalities.

5.3.3 Analysis of main points for action identified in cross-college reviews: Quality assurance, improvement and enhancement

Twenty-seven per cent of all cross-college main points for action related to quality elements B6 and B7. The three most frequent areas for main points for action were that:

• team action plans should include specific, measurable targets to help address weaknesses;
• the college should develop procedures to systematically identify and share best practice in learning and teaching; and
• self-evaluation procedures should be extended to cover all aspects of college services that impact on the learner experience and be linked to improvement planning activity.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• The majority of colleges still need to ensure that team action plans include specific measurable targets to address weaknesses.
• More than a few colleges have now developed procedures to systematically identify and share best practice in learning and teaching.
• The majority of colleges have improved their self-evaluation procedures to cover all aspects of college services which impact on the learner experience and have linked their procedures to improvement planning activity.

6. Sector-leading and innovative practice

Review reports identified and described a range of practice that was:

• sector-leading and innovative;
• particularly effective in terms of impact or outcomes; and
• able to be replicated, adopted and customised by other colleges.

During the external review process, such SLIP examples were identified in both subject and cross-college quality elements. In the subject elements, reviewers identified 79 examples of SLIP from the 43 colleges reviewed. The bar chart below shows the frequency of SLIP examples for each of the quality elements in which they occurred. For this analysis, quality elements A1-A4, A6, A8 and A9 are considered separately, not under the general heading of other significant factors.

The bar chart shows that in the 43 external reviews, 41% of SLIP examples were identified in the quality element programme design. Learning and teaching processes accounted for 29%, learner progress and outcomes 15% and guidance and learner support 6%. There were three SLIP examples recorded for the quality element accommodation for learning and teaching, and one each for equipment and materials, staff, assessment and quality assurance and improvement. There was one SLIP example recorded for quality assurance and improvement.

Percentage of SLIP examples by quality element

In cross-college reviews, 108 SLIP examples were identified. The bar chart below shows their distribution across the cross-college quality elements to which they are related.

The bar chart shows that almost two-thirds of cross-college SLIP examples related to the quality elements B1: Educational leadership, direction and management and B2: Access and inclusion. The quality element with the greatest number of SLIP examples was B2: Access and inclusion, with 34% of the total. There were only four SLIP examples recorded in B6: Quality assurance and three examples recorded in B7: Quality improvement.

The full range of SLIP examples identified in both the subject and cross-college reviews is available on the HMIE website at this link: http://www.hmie.gov.uk/GoodPractice/Default.aspx.

7. Some key messages from the 2004-08 reviews: summary

Colleges have a critical role to play in meeting the lifelong learning needs of individuals, their communities and the economy. In recent years colleges have further developed policies, quality systems, procedures and strategies which are designed to meet these needs and to improve the overall experience of learners.

The HMIE report Improving Scottish Education, 21 February 2006, summarised findings in college reviews over the 2002-2005 review cycle. It identified areas where colleges were doing things well and areas for improvement. The report outlined the need to:

• identify and use effectively the good practice which exists in colleges;
• ensure there is sufficient staff expertise, access to equipment and facilities, and variety in teaching approaches to ensure the effective contribution of ICT to the learning process;
• focus on self-evaluation procedures to improve the quality of learning and teaching; and
• make better use of performance indicator data to evaluate the effectiveness of programme delivery.

The key strengths and weaknesses which emerged from reviews between 2004 and 2008 are summarised below. Particular strengths identified in A5: Learning and teaching process and A7: Learner progress and outcomes were as follows.

• Learners were enthusiastic about their learning experiences, showing high levels of engagement and motivation.
• Teaching staff applied their specialist knowledge effectively to the teaching and learning process and generally kept their subject knowledge up to date. In doing this they were able to enhance the learning experience for learners.
• Effective relationships between staff and learners contributed to enhancing the learner experience.
• Learners were generally making good progress in achieving their learning goals and gaining appropriate vocational skills.
• A high proportion of learners progressed successfully into employment or further study.

However, the review process also identified weaknesses in A5: Learning and teaching process and A7: Learner progress and outcomes that were relevant to attrition from specific programmes or learners’ low success rates in award-bearing programmes. They most frequently related to:

• resources, including ICT, not being used sufficiently in support of learning;
• staff not systematically checking learners’ understanding or providing them with effective feedback; and
• too narrow a range of teaching methods being deployed by staff.

These strengths and weaknesses were underpinned by other significant factors that had a positive or negative impact on the learning experience.

Positive other significant factors included:

• recent improvements in accommodation and facilities;
• staff working effectively as teams, reflecting on learning and sharing ideas and good practice;
• teaching materials and resources of a high industry standard; and
• appropriate learning support for learners through provision of a range of highly effective measures.

Other significant factors which had a negative impact upon the learning experience included:

• insufficient access to resources, including ICT resources; and
• accommodation that was poorly designed for learning activities.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• There is now better access to resources including ICT for learners.
• All colleges have either resolved or are in the process of improving their accommodation to ensure it is appropriately designed for learning activities.

Many of the strengths recorded during reviews relate to senior managers’ initiatives in developing and implementing systems, procedures and strategies to improve the quality of the learner experience. In the cross-college quality elements in the reviews over 2004-08, HMIE identified major strengths where senior managers had:

• developed effective partnerships and links with stakeholders which helped extend learner participation by offering a range of programmes in a variety of modes;
• provided appropriate access to well-coordinated guidance and support which met learner needs effectively;
• provided accommodation which met learner needs and expectations well;
• implemented CPD procedures which met the training needs of college staff effectively; and
• implemented comprehensive and well-developed quality systems to improve the learner experience.

In other cases, colleges had developed processes and systems for quality improvement and enhancement but the impact on learners had been limited. Issues included:

• insufficient targeting of college and subject team action plans on specific issues to help address weaknesses in key areas;
• insufficiently systematic arrangements to identify and share best practice in learning and teaching;
• inadequate evaluation of the effectiveness of CPD activities undertaken by teaching staff, hindering plans for the improvement of the learner experience; and
• self-evaluation procedures and operational planning which did not adequately cover all cross-college and support functions that impacted on the learner experience.

Follow-up activities by HMIE have indicated the extent to which colleges have now addressed these issues.

• The majority of colleges where issues were identified during the review have not yet ensured that college and team action plans target and address weaknesses in key areas.
• More than a few colleges have now developed procedures to systematically identify and share best practice in learning and teaching.
• The majority of colleges now systematically evaluate the effectiveness and impact of CPD activities undertaken by staff.
• The majority of colleges have improved their self-evaluation procedures to cover all aspects of college services which impact on the learner experience and have linked the self-evaluation to improvement planning activity.

8. College staff evaluations of the review process

HMIE invites and encourages colleges to provide feedback about the review process and its outcomes through the use of questionnaires and engagement with the principal after both phases of the review are complete. All comments from college staff are considered by HMIE in adjusting current arrangements and in developing procedures for future reviews.

8.1 Evaluations from review phase one

Almost all respondents, in both the subject and college review process, rated the preliminary procedures, suitability of methods, deployment of reviewers, procedures employed and quality of feedback as good or better. Ninety-one per cent of respondents from the subject review process rated the usefulness of written feedback as good or better compared to 89% of respondents from the college review process.

A few aspects of the review process received less positive feedback from a few respondents involved in both the subject and college review process. These included the range of documentation required and the level of demand experienced by staff in preparation for and experienced during the review.

Comments from respondents

The following comments have been chosen from feedback received about the first phase of reviews during 2007-08 to be typical of the range of perceptions of college staff of the review process. The first group of comments reflect the predominantly positive pattern of evaluations recorded above.

• Given the breadth of supported learning, the reviewers did an excellent job in covering the number of classes and staff which they saw.
• Our subject reviewer was vey helpful in giving positive examples of changes which could be made for improvement. We felt the comments were supportive and helpful.
• Feedback was clear and concise, delivered in an encouraging and supportive manner.
• The review was conducted in an extremely friendly way. HMIs put me at my ease. The lengths of the interviews were rather short but fine. Many thanks.
• The reviewers were helpful, made staff comfortable and staff were left with a positive impression.
• I think the feedback was very useful and also very fair. Comments were treated seriously and will be acted on in the context of useful and constructive criticism.
• Overall a very worthwhile and very constructive exercise despite the heavy burden of paperwork.

However, there were also comments that were less positive or made suggestions for improvements. Examples are shown below.

• There was a very high reliance on written materials with less "professional review".
• A handbook or guide to the procedures identifying the key evidence requirements would have been useful.
• More use of electronic information would have been more effective and environmentally friendlier.
• Demands on staff to attend and/or facilitate meetings were excessive.
• In a smaller college like ours the one week puts a great deal of strain on normal operations.
• There is too long between the review and the formal publishing of the result. I’d have liked to see an executive summary of initial findings published quickly with the full report and SLIP material to follow.
• More written details about weaknesses would have made the task of constructing an action plan easier.
• Information for students prior to review could be more user friendly and in "plainer" English.
• Difficult as a UHI college to segment what we do for FE learners only.

These comments have all been taken into account in the design of new external review arrangements from 2008-09 awards.

8.2 Evaluations from the follow-through phase

Almost all respondents rated the suitability of methods employed by reviewers, the quality of the feedback given and the good practice explored as good or better. Asked to rate the follow-through phase in terms of helpfulness to the colleges, most respondents chose good or better.

Comments from respondents

The following comments have been chosen from feedback about the follow-through phase of reviews during 2007-08 to be typical of the range of perceptions of college staff of the review process on specific aspects of the quality of provision in colleges. They are presented in two groups, as in section 8.1 above.

• HMIE took time and trouble to understand the process behind the SLIP in this area and gave valuable feedback.
• The follow-through was conducted quickly and efficiently with minimal disruption for the section.
• Very helpful development sessions for staff.
• The follow-through phase was very helpful in terms of reinforcing our own thoughts or how we should move forward with the objective of improving learning and teaching.
• It was good to hear the opinion of an independent person on a process that we think is very good.
• The developmental workshops provided were greatly appreciated by all staff who participated.

However, there were also comments that were less positive or made suggestions for improvements. Examples are shown below.

• A greater degree of information on the criteria used and the process of deciding on the sector leading/innovative practice chosen would be beneficial.
• Perhaps a more "workshop" hands on, as opposed to a round table discussion, would be of additional help and benefit.
• Perhaps it would have been useful for a longer period of time to have been devoted to some of the activities scheduled.

The examples above include comments from college managers and staff that occasionally conveyed reservations about aspects of the external review process. Generally such reservations are the views of individuals rather than representing commonly-held or widespread misgivings. However, HMIE has addressed one recurring theme in particular.

Concern was expressed by a few respondents that time was spent identifying whether the good practice identified through the SLIP process was sector-leading and innovative or just very good practice. Also comments were made that there was insufficient time during the follow-through to explore the issues identified during the review. The new external quality arrangements for colleges (2008-12) should improve this situation. Issues relevant to the college and SLIP proposals will be discussed on an annual basis with HMIE rather than the current situation of every four years.

9. Glossary