[Previous] [Contents]
[Next]
HOW WELL ARE CHILDREN AND YOUNG PEOPLE PROTECTED AND THEIR NEEDS MET?: Self-evaluation using quality indicators
Part 3
Using the Quality Indicators for Self-Evaluation
This section of the document provides advice about practical approaches to self-evaluation and some examples of using the quality indicators to evaluate aspects of practice.
It will help us to:
- focus on best practice in understanding how to protect and meet the need of children and young people;
- ask key questions to help agencies measure their current practice against a benchmark of chosen quality indicators;
- look at how agencies might collect evidence of their current practice in order to evaluate its effectiveness and identify areas in which improvements are necessary;
- weigh up these strengths and areas for improvement as part of overall evaluation of the quality of services; and
- use the results of evaluation to plan for improvement and to report on what has been found.
Taking a broad view: Self-evaluation across all key areas
It can be helpful to stand back and consider the overarching question How well are children and young people protected and their needs met? All those involved in the provision of services may have a role to play.
Our three basic questions can guide us through the process of evaluation, taking a broad view.
How are we doing?
We might scan across all agencies and assign one of the six levels to each quality indicator. The evaluations would be based on our professional judgements and refer to the evidence which has been gathered in the normal course of our work. We do not need to look in detail at everything that is going on. We can concentrate on the main areas of activity centred round the indicator. We can then record the evidence alongside the relevant quality indicator.
Recording evidence involves us identifying what we know is happening not what we think should be happening. For example, we cannot put down as evidence that practitioners have all had training in how to respond to children needing help. We need to know what they do in practice, as a result of that training.
This broad look can be undertaken at a variety of levels within agencies or across agencies.
How do we know?
However we decide to work, the aim should be to promote objective and open discussion based on evidence. We can then assign one of the six levels to each quality indicator, perhaps using a grid similar to the example below.
No. |
QI and Themes |
Level |
Evidence to support assigned level |
1.3 |
Children and young people are helped by the actions taken in immediate response to concerns
- Professionals’ initial response to children, young people and families who need help
|
|
|
When we have finished we can see at once which quality indicators have been evaluated as good, very good or excellent. These indicate key strengths and should be recognised as such.3
We can also immediately identify those assigned as adequate, weak or unsatisfactory. These are, or should be, priorities for improvement within our development plan. We can take a closer look at these areas using the method described in the next section in order to identify contributing factors. If there are several areas for improvement, we need to select the most pressing when agreeing priorities for development.
What are we going to do now?
Once we have identified and prioritised the areas for improvement we need to identify what follow up action is required and whether that should be the responsibility of one particular agency or a collective responsibility that may require consultation with another agency or organisation or direction from the CPC.
Taking a closer look using the quality indicators
There are a range of reasons why practitioners or managers may wish to look at aspects of their practice in more detail. They may wish to focus on areas of weakness emerging from the broad look described above. An equally useful approach would be to evaluate how well agencies, singly or in partnership were meeting their own strategic aims in relation to protecting children and young people and meeting their needs. They could then select the appropriate quality indicators, which best related to that aim and collect evidence against these indicators.
Our three basic questions can guide us through the process of evaluation, taking a closer look.
How are we doing?
- We need to agree what we are going to focus on. We need to decide at the outset what it is we want to find out. We may wish to use one of the key questions as a starting point. It may be more helpful to generate some questions directly related to the context of our work. Although all self-evaluation should contribute to answering the over-arching question How well are children protected and their needs met? it would be very difficult to look in detail at all the quality indicators at one time.
- We need to define the area of activity. This can range from aspects of a range of services in an area, to sampling of work across a small team or within a very local area. It may be about a voluntary organisation looking at their role in protecting children and young people and seeking help from other professionals. It is often useful to involve one or more partners in the evaluation process so that views about aspects of quality can be shared and challenged in order to reach an agreed view.
- We need to gather any evidence we have already related to the area we want to focus on, such as performance information, previous reports or audits, or the views of stakeholders.
How do we know?
When we have decided what our focus will be, we need to take the following steps.
- Select the quality indicators which will give us most help.
- Use the illustrations to develop some questions to be answered during the evaluation process.
- Identify features to look for, referring to relevant national and local policies, and the vision statements within our own organisation.
- Decide what evidence we need to show that these features are present and effective.
- Collect the evidence and judge the quality of what we have observed.
We may wish to summarise strengths and weaknesses. We can also refer to the illustrations to come to a balanced judgement overall, about our performance. The quality indicators can be evaluated at all levels, jointly across all agencies within a local authority area, by single agency or by individual teams.
What are we going to do now?
The next step is to promote improvement. The outcomes of self-evaluation should enable us to:
- Identify key strengths and areas for further development;
- Provide feedback to professionals involved and share good practice in relation to the impact of services on the lives of children, young people and their families;
- Report on our findings if appropriate;
- Identify priorities to be included in improvement planning within our agency, or in joint planning structures for our area; and
- Identify areas where further information is required in order to make meaningful evaluations of the impact of services.
How are we going to find the evidence?
It is necessary to build up evidence from a range of sources about the practice related to a particular quality indicator. Each indicator will help us to define the kinds of evidence which might be helpful. For example, it may include a study of records of cases to examine information sharing across agencies or questionnaires. It may include interviews to find out what training or supervision arrangements are in place for practitioners.
In any approach to self-evaluation, we need to bear in mind the importance of relating processes to impact on or outcomes for children, young people and families. To assist in evaluating impact, it is very worthwhile for us to involve a range of stakeholders, including children, young people and their families in self-evaluation. Their views are a valuable source of evidence. This will be particularly important when we are reviewing across services and multi-disciplinary teams.
The following list suggests some sources of evidence.
- Review of case files of a range of children and young people.
- Interviews with key staff who have been involved with children or young people in a case sample.
- Interviews with parents/carers and/or children and young people in the case file sample, where possible and appropriate.
- Interviews with focus groups of staff from a range of relevant agencies.
- Interviews with focus groups of children in schools or other settings.
- Interviews with focus groups of parents, for example those with children in a family centre.
- Interviews with strategic and operational managers.
- Reading a range of documentation, including policies, procedures, improvement or service plans, self-evaluation documentation and training plans.
- Scanning information already available about the comparative levels of need, staffing and socio-economic need of the area.
- Information available from other inspections already in the public domain.
- Observations of practice, for example attending a sample of review meetings.
- Minutes of meetings.
Managers and practitioners working in a local area will be able to identify valuable sources of information in their own area.
For an evaluation to be robust, it needs more than one source of evidence so that judgements are confirmed by a variety of sources of evidence. For example, managers may report that they have clear guidance in place related to some aspect of work. However, the views of practitioners and evidence of whether the guidance is followed and its impact on practice would need to be taken into account before reaching a judgement about the impact of that guidance.
In undertaking self-evaluation in the complex area of protecting children and young people and meeting their needs, it is important to be aware of all the sensitivities involved. There are rarely right answers, and it is very important to develop a culture where professionals are confident to share views, successes and difficulties and learn from each other.
The following sections are practical examples of how the quality indicators can be used in a range of specific situations.
Example 1: Reviewing the help children and young people get
Introduction
The CPC in an area has had discussions about how well children and young people are protected in their area. They are aware that there are a significant number of children and young people they try to help who do not engage with services and that professional help has made little impact on their lives. They have recently become aware of research published by Edinburgh University and Childline Scotland which revealed that most children and young people would not tell an adult or seek help when they had difficulties.
"Many children feel unable to ask adults for the help they need. That’s what comes through in this research — the fact that it’s a hidden problem".4
How are we doing?
The CPC wants to look more closely at how they help children and young people. They want to find out if children and young people get the help they need when they need it and if services in their area help to reduce the likelihood of children and young people experiencing harm. They decide to use the quality indicators to plan an evaluation of the experiences of children and young people in their area.
How do we know?
In order to make evaluations of the effectiveness of help and the impact on children and young people, they select a small number of quality indictors and from these select the themes they wish to focus on as follows.
Q.I. 1.1 Children and young people are listened to, understood and respected
Theme: Communication
- Children and young people have good communication with professionals who know them well. They are listened to, understood and respected and their views are taken into account when decisions are made.
Theme: Trust
- Children and young people have a named person to help them and can see that person regularly if they choose. They can discuss issues in private when they want to. Children, young people and families trust the professionals who are helping them. Children and young people feel that the professionals involved in their lives are using their power to help keep them safe.
Q.I. 1.2 Children and young people benefit from strategies to minimise harm
Theme: Support for vulnerable children, young people and families
- When a child or young person needs help, there is generally an adult who knows them well, recognises their need and takes steps to help them.
- Support is provided frequently enough and at times, in places and in ways that are acceptable to children, young people and families.
- Children, young people and families benefit from help provided and are less at risk.
From these illustrations the CPC generates questions which they want answered, draw up a list of sources of evidence to help find out the answers to the questions and describe what they are looking for. These are shown below.
An important aspect of this example is the engagement of a range of professionals in talking honestly about their work, the risks they feel they sometimes have to take and the decisions they feel concerned about. It will require an atmosphere of trust and confidentiality to help a process of genuine self-evaluation. Professionals involved will need to decide how many children, young people and families they are going to consider closely to get the necessary evidence. They will need to address the issue of finding out from some children and young people why they do not engage with services.
In order to carry out this work, the managers of services, including any voluntary organisations involved, may need to free up some operational time for reviewing case files and talking to children and young people. They may wish to involve a voluntary organisation, to talk to children and young people, including children and young people who are at risk but not using services.
What are we going to do now?
This piece of work would result in a report to the CPC. The professionals involved may decide to write a report which describes practice and highlights its strengths and weaknesses with or without making specific evaluations. The report should identify suggested ways of improving services where weaknesses were found. It is likely that management teams in individual agencies would also find the report useful.
The CPC could use the report to plan improvements where they were clearly required. This may include redesigning resources and services to meet more closely what children and young people want. It may involve staff training and development or increasing resourcing in certain areas or types of the services.
Q.I. 1.1 Children and young people are listened to, understood and respected
What do we want to know? |
How do we find out? |
What to look for |
Do children and young people feel:
- listened to;
- understood;
- respected;
- that professionals know them; and
- there is someone to help them when they need it?
Do children and young people:
- have a named person; and
- see that named person regularly, in private and at time and place to suit them?
Do children and young people:
- trust professionals;
- feel they are helped by them; and
- their views affect the decisions made?
What kind of help do children and young people want? |
- Review a selection of children’s and young people’s records, selecting some that are known to have engaged well with services and some that have not.
- Look at records from all agencies which have had contact.
- Trace, through descriptions of contact, reports and minutes of meetings, the kind of interactions that have taken place.
- Engage people who have advocacy roles, e.g. from voluntary organisations, to talk to older children and young people about their experiences of services.
|
- Evidence that all professionals have regularly sought and recorded in files the views of children and young people.
- Evidence that children and young people have sought professionals, e.g. social workers and guidance teachers, to talk to.
- Evidence that they have at least one professional that they like and who they want to talk to.
- Information about what stops children and young people from seeking help.
- Evidence from files and from children and young people about whether they feel they have been helped by services and if not why not.
- Evidence that decisions have taken account of children’s and young people’s views and if not, the reasons are recorded.
|
Q.I. 1.2 Children and young people benefit from strategies to minimise harm |
What do we want to know? |
How do we find out? |
What to look for |
- Are all vulnerable children and young people in this area identified and known to professionals?
- Are they all being supported by professionals?
- Is help provided frequently and at times, in places and in ways that are acceptable to children, young people and families?
- Have the actions taken helped children, young people and their families?
- Do children and young people feel positive about the help they get and do they feel it has made their lives better?
|
- Look at records and files from a range of agencies.
- Check other records held by drug support agencies or GPs on parents who are substance misusers or are known to mental health services.
- Find out from records, which services, if any, these families use and what other help or support they are getting.
- Check a sample of recent domestic abuse reports, audit trail outcomes and check whether children were known to other services.
- Map the range of services available in the area across all agencies and voluntary organisations and identify service gaps or duplication.
- Interview children and young people or arrange for appropriate interviews through advocacy or voluntary organisation.
|
- Evidence that professionals are aware when children may need help, for example from their behaviour in school, or through observations from detached youth workers.
- Evidence that children, whose parents are known to services because of issues which could affect their capacity to parent, are getting help if they need it.
- Evidence that services are used and valued by the groups they are intended to support.
- Evidence about any children and young people who need help but are not using services.
- Evidence of children’s and young people’s views of the services available, whether they use them or find they get appropriate help from them.
- Evidence of any gaps in meeting children’s and young people’s needs.
|
Example 2: Sharing information to help children and families
Introduction
The professionals who work together with families of young children in an area meet regularly to share information, and discuss the development of services for vulnerable families in their area. They include the health visitor, the manager and family support worker from the local children’s centre, the head teacher of the nursery school, the community liaison officer from the local police, the team leader from the local children and families social work team and voluntary organisations working with families. They want to use their meetings in a more focussed way to improve the quality of services they are providing in the area. The group is aware of the findings of a number of inquiry reports that highlighted information sharing as an issue when children were not adequately protected.
How are we doing?
The group decides to use the quality indicators to take a closer look at how effective information sharing is within the local area and how the information is recorded. The group members get on very well and feel that they share information with each other but they want to check that information is shared across a variety of different professional disciplines and it is not dependant on relationships. They decide to review the records of five children that are known to be supported by a number of professionals.
How do we know?
They decide to use a range of quality indicators to evaluate whether relevant information is shared with other practitioners or not and how the information is recorded. They select the following themes from illustrations.
Q.I. 3.2 Information sharing and recording
Theme: Appropriate sharing of information
- Professionals clearly understand when they need to share information about children, their families or other adults who may have contact with children in order to protect children from harm.
- Professionals who are gathering information in order to assess the risks to and need of a child, ensure that they seek information from all relevant sources, including professionals who may be involved with other members of the family.
Theme: Joint understanding of information
- When information is shared, the professional communicating the information ensures that the recipient has fully understood the information, makes clear what is fact and what is opinion and ensures the expected actions. There is a record of what information has been shared, why and with whom.
Theme: Management and recording of information
- Information is recorded accurately, succinctly and is stored securely but is accessible to those who have a right of access. Where there are arrangements for sharing files or electronic systems, there are clear protocols in place which are understood by all staff, including staff who are managing databases. These protocols are followed.
The group decide on a number of questions they wish to answer and they draw up a list of possible sources of evidence to assist them to answer the questions. These are shown below.
This piece of work will involve a number of different professionals and will need commitment and engagement from all involved. The group need to decide who will review the records and who will co-ordinate the collection of the records in each agency. This may involve getting access to electronic files as well as paper files in some cases. Managers may need to free up time for staff to participate in the exercise.
What are we going to do now?
This piece of work would not necessarily result in a formal report being written by the group but would involve reporting the findings, both strengths and any weaknesses, to managers in each agency and providing feedback to staff involved in the exercise. If any weaknesses were found then recommendations would be made and implemented by all the agencies concerned. Good practice or any concerns found while undertaking the review would also be highlighted and shared with other practitioners and managers.
The CPC may also be interested in the findings of the review.
Any training issues identified while undertaking this exercise would be highlighted and addressed.
Q.I. 3.2 Information sharing and recording
| What do we want to know? |
How do we find out? |
What to look for |
- Do we5 clearly understand when we need to share information about the children we work with and their families to keep children safe from harm?
- When a child we know is being assessed do we make sure that information from all relevant services is included?
- When we share information with each other, at our meetings or more informally, do we make sure that the information is understood, that we are clear what is fact and what is opinion, and that we record what we have shared?
- Do we make clear what we expect when we tell someone something about a family we know, for example, if a nursery teacher tells the social worker that a child has been talking about her daddy when the father should not have access to her?
- To what extent do our records give a clear summary of key events, key people and the help the child is receiving from services?
- To what extent do we all understand and follow the protocols for sharing our files and accessing our electronic database?
- To what extent can we see benefits to sharing information about our children and families?
- To what extent do our regular meetings make a good use of our professional time, and help us protect children?
|
- Speak to all staff involved with each child in the sample.
- Review the files from all agencies involved with each child in the sample and their families, including any adults who have significant contact with the child.
- Examine case notes, such as minutes of meetings, contact records and reports.
- Speak to staff who may be delivering a service to the parents or carers of the child such as mental health staff or addiction staff.
- Speak to the family or other carers for each child in the sample.
- Check records on sharing of files and how we have been using our electronic database.
- Speak to the children and families involved to seek their views about any information which has been shared.
- Ask families about their experience of professionals working together, and in particular to what extent the range of professional interactions is assisted by professionals meeting together and sharing information.
|
- All staff have had basic child protection training. They have acted appropriately when they have had concerns about a child and they have shared information when it is necessary to help a child.
- When a concern has been raised about a child, more information is always sought from other professionals and family involved with the child.
- Information has been shared across agencies that are involved in protecting each child.
- For each family decisions have been made on the basis of fullest information and there is not information in any file which, if shared, could have helped practitioners do more to protect a child and meet their needs.
- Evidence of feedback from other agencies which demonstrates that the information shared has been understood, and acted on when necessary.
- Families feel comfortable about information shared, and do not feel as if they have to keep repeating information to different professionals, or that information has been shared behind their backs.
- Files are stored securely, records are readable and ensure that practitioners can access the information they need quickly and effectively.
- Arrangements for access to electronic files help us in our work and we use them appropriately.
- Evidence that our meetings and our regular sharing of information has made things better for families we work with.
|
Example 3: Submitting referral reports
Introduction
The police area commander and the local Children’s Reporter have had a discussion about the quality of "compulsory measures of supervision" referral reports submitted by the police to the Reporter. They both feel that police officers are not always clear about the purpose of such referrals, the type of information required by the Reporter and what the officers expect to happen following submission. They think it likely that police officers who have contact with children and young people, but who do not specialise in child protection, are not always effectively assessing the risks to, and needs of, children and young people prior to submitting referrals.
How are we doing?
The area commander and the detective inspector in charge of the police family protection unit decide to take a closer look at the police process for the submission of "compulsory measures of supervision" referral reports to the Reporter. In particular, they want to know how officers decide that children and young people need help and whether the referrals that are being made are appropriate and likely to help achieve best outcomes for children and young people.
How do we know?
In order to assess the quality of assessments of risks and needs and the appropriateness and likely effectiveness of referrals to the Reporter, they select one quality indicator and from this select the themes they wish to focus on as follows:
Q.I 3.3 Recognising and assessing risks and needs
Theme: Recognising a child or young person needs help
- Police officers who have contact with children, young people or families are alert to and recognise signs that children may need help or protection from harm.
- When a concern is raised with the police about a child or young person, a police officer or other relevant professional sees the child or young person, reassures them and establishes the nature of the concern and any immediate risks.
Theme: Initial information gathering and investigation
- When concerns about a child or young person are referred to the police, an immediate assessment of the risk of harm to that child or young person and any other associated children and young people, for example siblings, is carried out.
- In appropriate circumstances there is an initial referral discussion between the police, social work and health and any other relevant professionals to establish what is already known about the child or young person and their family and to come to a joint decision about what further action is required in the best interests of the child or young person.
- As part of their assessment, police officers consider, in conjunction with other relevant professionals, whether to make a referral about a child or young person to the Children’s Reporter with the rationale for referral/non-referral being recorded.
From these illustrations, the area commander and the detective inspector in charge of the police family protection unit generate questions which they want answered, draw up a list of sources of evidence to help find out the answers to the questions and describe what they are looking for. These are shown on pages 64-65. They decide this should be a readily manageable piece of work that addresses a range of issues related to the quality indicator and selected themes. An important aspect of this example will be the engagement of staff, particularly the non-specialist front line officers, in talking openly and honestly about their work and how they make these decisions in an operational setting.
In order to carry out this work, the area commander and the detective inspector in charge of the police family protection unit recognise that they may have to dedicate staff time in order to carry out the necessary audits and interviews.
What are we going to do now?
This piece of work would result in a report to the area commander who will almost certainly wish to share and discuss findings with the Reporter prior to developing proposals for change or improvement. Whilst this exercise was conducted in order to address issues in one particular agency, namely the police, the findings and any proposed changes or improvements may well impact upon other agencies.
The CPC may be able to use the report as a basis for planning improvements over a range of issues which may be raised in the report and which may affect more than one agency. An example of this might be the need for a risk assessment tool which could be used by all the agencies.
Q.I. 3.3 Recognising and assessing risks and needs
What do we want to know? |
How do we find out? |
What to look for |
- Are we satisfied that all our staff who have contact with children and young people or families are alert to, and able to, recognise the signs that children may need help or protection from harm?
- When a concern is raised about a child or young person, do our staff make sure that the child is seen and the nature of the concern established?
- Do our staff demonstrate ability to gather relevant information for the purpose of assessing risks and needs?
- When a situation arises which involves a child at risk or in need, do our staff take immediate action to keep them safe and protect them from harm?
- How exactly do our staff carry out risk assessments, which other professionals do they consult, how are they recorded and do they address risks to all children involved, for example siblings?
- Are there clear and comprehensive records which detail the actions taken by our staff when a concern is raised about a child or young person or when a risk or need is identified?
- Do the records include contacts and discussions with the child or young person and with other professionals and are decisions and outcomes resulting from those contacts and discussions also recorded?
- Are the children and young people referred to the Reporter ones who are likely to be in need of compulsory measures of supervision? Are the risks to, and needs of, the child clearly identified and recorded in the referrals made? Are any children and young people not referred who should be?
|
- Speak to a range of staff and their supervisors, including those in specialist departments, for example the drugs squad.
- Review a sample of referral reports to the Reporter.
- Review a sample of case files, including any contact sheets or other records, for example records of initial referral discussions or entries on computerised systems.
- Speak to the Reporter to ascertain more detailed information about his/her concerns about the quality of referrals.
- Speak to children, young people and their families or carers.
- Consult with professionals from other agencies, for example social work or health, to gain their perspectives on the effectiveness of joint decision making and on the appropriateness of referrals by the police to the Reporter and to their agency.
- Audit command and control incident logs.
- Review content of awareness raising/ training courses, attendance at training and evaluations of quality and relevance of training/awareness raising.
- Review force and local procedures and other guidance, for example inter-agency child protection guidelines.
|
- Evidence that all staff who have contact with children, young people and their families have an appropriate level of awareness in relation to child protection. In particular, they have the ability to recognise signs that children and young people may need help or protection from harm and that they take appropriate action. This should include officers in specialist departments whose primary responsibility is not necessarily focussed on child protection, but who regularly come into contact with children, young people and their families.
- Evidence that when a concern is raised about a child or young person or where it is apparent that a child or young person may be at risk or in need, arrangements are made for the child or young person to be seen without delay, either by a police officer or another relevant professional.
- Evidence that staff have gathered all relevant information available to them from both internal and external sources and have consulted with other relevant professionals before making a referral to the Reporter.
- Where there is more than one child or young person for whom there is concern or identified needs or risks, separate risk assessments and records are maintained for each child or young person.
- Evidence that comprehensive information collected in the course of assessment is recorded accurately and succinctly in either a manual file or on a computerised data base. The record contains details of all contacts with the child, young person or family and with other professionals and includes the decisions and outcomes resulting from those contacts. There is a chronology of events.
- Evidence that all relevant factors have been taken into consideration by staff when making an assessment of risks and needs, preferably using a recognised risk assessment tool, or approach which has been agreed with partner agencies.
- Evidence that all staff requiring child protection training/ awareness raising have received it and that the content has had an impact on their day-to-day work.
- Evidence that there is good communication between professionals including police, health, social work, education and the Children’s Reporter and that key decisions are made jointly.
|
3 A copy of a blank table of the QIs in this format is available from HMIE/SFCU website
4 Children’s Concern: Research briefing from Centre for Research on Families and Relationships, Edinburgh University and Childline Scotland. (April 2005)
5 We in this example refers to the group of practitioners undertaking the self-evaluation activity
[Previous] [Contents]
[Next]