Financial pressures in the 1980s and early 1990s led to system-wide evaluations by administrators and consultants who were especially concerned about cost effective achievements in schools. These pressures were very evident for both public and private schools. There have also been increasing concerns about student performance in core subject areas and this has led to system-level evaluations. Performance testing in literacy at specific age levels as well as testing in core subjects is now common in many countries. In most countries individual schools receive details of their students’ results and so school-level evaluative data are also available.
The development of curriculum frameworks and standards, couched largely as outcome statements, provides a major opportunity for systems to evaluate the performance of their respective schools. Thus, in many systems annual reports have to be submitted by individual schools together with external evaluations over longer periods, usually triennial. It is in the self-interest of many schools to undertake their own school-level evaluations and to use the results to target their market share of students. School-level evaluation, whether defined in terms of accountability or standards of performance, is a major focus for schools in the twenty-first
century.
‘Evaluation’ is a process of collecting and communicating information and evidence for the purpose of informing judgement and ascribing value to a particular programme (Simons, 1987). It can refer to small-scale activities involving a very limited number of clients (such as a teacher and his or her class) or to massive large-scale studies involving many schools and teachers (and other interested parties such as parents and community members). Neve (2001) examines the relative advantages of external school evaluation (for example by OFSTED inspectors in the United Kingdom) where the emphasis is upon accountability, setting standards and benchmarks, and internal school evaluations where the emphasis is upon self-evaluation, empowerment evaluation, reflection and the professionalization of teachers.He argues the case for a combination of external and internal evaluation. Specifically, external evaluation can:
stimulate internal evaluation – to motivate persons and organizations to do internal evaluation;
expand the scope of internal evaluation – by providing benchmarks and comparative data;
legitimize the validity of internal evaluations. Further, internal evaluations can benefit external evaluations by:
expanding the scope and examining unique elements;
improving the interpretation of findings;
increasing the utilization of the evaluation results.
McGehee and Griffith (2001) and Visscher (2001) acknowledge that large scale evaluations are becoming an important part of the education culture. Fullan and Earl (2002) undertook a large-scale evaluation of the National Literacy and Numeracy Strategy in the United Kingdom and noted that it is a prime example of the intricacies of national reform. Ainley et al. (2002) noted the renewed interest in large-scale evaluation in Australia with regard to literacy and numeracy.
School-level evaluation as part of the general field of evaluation can be undertaken as a small-scale or large scale activity. Skilbeck (1982) supports the use of small-scale activities rather than elaborate, comprehensive, managerial evaluations, and suggests that they should be at the level of ‘intelligent forms of reflection on experience, self-appraisal and forward thinking’. In his opinion, educators often amass vast quantities of unmanageable data, and this should be avoided by being quite clear about such questions as:
What do I need to know about this activity?
How can I most economically find out?
How can I use what I know?
What do I need to make known to others?
School evaluation differs from other kinds of educational evaluation in that it focuses upon how teachers and students interact over a particular curriculum or syllabus at one school site. It is not just an analysis of how students perform in a teaching/learning unit, nor is it just an analysis of the lesson plans which teachers use in instruction. Rather, school evaluation involves an examination of the goals, rationale and structure of teachers’ curricula, a study of the context in which the interaction with students occurs (including parent and community inputs) and an analysis of the interests, motivations and achievements of the students’ experiences. School evaluations also focus on the needs and interests of the constituent groups involved in the school community. Particular interest groups operating at the school level, mainly teachers, administrators, students and parents, may have very different views about the purposes of schooling. Consequently, evaluation studies have to reflect different orientations and not give undue emphasis to single dimensions such as the behaviour of individuals (students), an analysis of materials, or the behaviours of a school as a social institution. Rogers and Badham (1992) suggest that school evaluation is about accountability and development. Accountability is crucial to prove quality – to ensure that standards in a school are rising. Development is also most important because it establishes a positive staff climate – staff are more aware of the data that needs to be collected as an aid to certain developmental goals.
Wilcox (1992) emphasizes the developmental aspect also,along with four other important aspects of curriculum evaluation:
1. It is based on evidence which is systematically collected.
2. The evidence is seldom unambiguous and therefore needs to be interpreted.
3. Judgements of value are made about the entity being evaluated and its effects.
4. It is action oriented, intended to lead to better practices and policies.
The two fundamental questions to be answered before considering any evaluation are:
1. Why do you want to evaluate?
2. What do you want to evaluate?
In large-scale studies, the purposes of evaluation are usually related to policy concerns at the head offices about the widespread implementation of programmes into an entire school system. At the local school level, evaluation activities may be undertaken for a multitude of highly personal reasons. These could include:
concerns about providing better teaching and learning for students within a particular school community;
the need to examine the impact of a new programme or organizational processes;
. collecting and presenting information from teachers and administrators, students and parents
. analysis of information collected and making judgements
. strategic planning
. development – improving quality
. accountability – proving quality
the need to substantiate the value of a particular programme or organizational structure to parents and/or to local business;
response to dissatisfaction expressed by individual teachers or a group/ association. When establishing purposes of evaluation at the school level it must be realized that any teaching situation brings about some unintended outcomes. Any comprehensive evaluation study must therefore provide for the collection of data on side effects and unintended learnings. Because evaluations at the school level rely upon conviviality and cooperation, it is essential that disparate motivations such as those listed above are discussed by staff who, in a series of informal and formal meetings, may come to a consensus about what are the most important purposes for them in doing the evaluation (Thornton, 2001). Simons (1987) argues that one of the best ways to develop effective curriculum practices is to grant schools the authority to formally evaluate in addition to external agencies. However, in many cases individual schools cannot avoid external accountability forces – they are the driving force above and beyond the personal needs of a school community.
As an example, all government primary schools operating in Western Australia are required, under the School Accountability Framework: to produce, in partnership with their school community, a school plan setting out their objectives, priorities, major initiatives and evaluation measures;
to assess their performance in terms of standards of student achievement and the effectiveness of the school;
to make available to the public and to the District Director a School Report that describes the school’s performance;
to be accountable for the performance of the school – school staff to the principal and school principals to the District Director (Department of Education, 2002a, p. 5).
Yet, the accompanying documents for schools are couched in the language of ‘self-assessment’ and schools are encouraged ‘to see this document as a resource to augment their existing self-assessment practice’ (Department of Education 2002b, p. 8). Further, there is some scope for schools to select particular themes and a choice of tools.
In the terms of Schwab (1969) these factors are ‘commonplaces’ of curriculum and consist of ‘learner’, ‘teacher’, ‘subject matter’ (curriculum) and ‘milieu’. Any evaluation activity must necessarily examine the impact and interaction of these elements. The sources of information about these four commonplaces can vary considerably. For example, information about the school milieu might be obtained from parents, community members and employers; information about the subjects taught at school might come from school administrators, external subject specialists, publishers, superintendents and parents. The range and choice of sources of data relates back to the purposes of the evaluation, the scale of the activity, the time and funds available. Once the focus of an evaluation has been determined, it is then possible to plan the kinds of information needed. For example, the evaluators may decide that information about students should include data about their previous academic levels, ongoing information about their class performance and interactions
with the teacher, and information about their achievements. This type of information is obviously collected at different time periods and the examples listed above refer to all three types of data: that is, diagnostic data collected prior to the beginning of a curriculum unit to find out interests and achievement levels of students; formative data collected during the teaching of a unit to pinpoint aspects of the teaching that are mismatched and not being successfully implemented; and summative data, which are collected at the completion of a unit and focus upon specific student outcomes and achievement levels.
Techniques that can be used to collect diagnostic, formative and summative data about students are included . Similar techniques can be used for collecting information about teachers and teacher–student interactions.
Collecting evaluative data about teachers requires considerable support and goodwill. George et al. (1998) highlight some of the problems and issues. They suggest that the ideal situation is for teachers to work in peer panels comprising three to five teachers. The important considerations are that:
they choose each other and there are no superordinate–subordinate relationships;
matters that are discussed are private to them but generally focus upon skill development;
they agree to meet regularly, ideally once a week;
they give low-inference feedback to each other (observe/record/report).
They do not make high-inference judgements as this would interfere with their peer relationships. As teaching-partner observer or peer panels can use a variety of techniques to collect useful data over the various phases, ranging from informal observations to rating systems to the use of interviews and questionnaires. Self-reflection and analysis are extremely valuable activities for all teachers and especially important for school-level evaluation (Wroe and Halsall, 2001). Schon (1987) refers to the need for teachers to be reflective practitioners. He focuses specifically upon how and why teachers should reflect upon their
experiences.
The evaluative techniques can be used both in terms of self-evaluation and using a teaching-partner or peer panel. However, the most common techniques include some form of written recording sheet (e.g. keeping a diary) and a variety of observational techniques. Diaries represent a ‘shorthand’ method of recording the significant happenings of a teacher’s day. It is recommended that diaries should concentrate on one or two aspects that are considered most important. Points that may be useful as foci for diary entries include such questions:
Is my teaching behaviour having the desired effect in classroom management?
Has a particular seating arrangement encouraged the desired behaviour from the students concerned?
Has a particular teaching strategy improved the performance of a specific group of students?
Is a special project being positively accepted by the class or is there a lack of interest?
Observation is a direct, systematic way of determining what is happening in the classroom. Observations of classrooms can often be very revealing! For example, the literature contains examples of teachers who have complained that certain students in their class do not contribute to their lessons. However, observations by colleagues revealed that these same teachers did not encourage the students in question to participate and in some instances prevented their interaction with other students. There are often massive discrepancies between what teachers state they are teaching compared with what actually occurs in classrooms.
Several alternatives are available for the classroom teacher who wishes to collect his or her own observational data. These include using audiotaping or, if resources are available, videotaping. Student observations can also be sought via informal discussions and interviews or by the use of checklists and questionnaires. It should be clear that self-evaluation techniques for the teacher are fairly limited, and that far more data, including important additional perspectives, are available if colleagues on a school staff assist each other cooperatively with their evaluation activities. However, this requires colleagues to collect data about each other and to submit themselves to self-reflective activities. The challenge may be troublesome for some teachers unless peer panels (as described above) or similar pairings are organized. It is suggested that if teachers are willing from the outset to collect evaluative data about their own activities and their colleagues, then the feedback they obtain will enable them to be more successful and presumably more fulfilled.
There are, of course, many hidden assumptions involved in all this. Not all colleagues will want to submit themselves to all of the types of data collection and to peer and panel procedures. Teachers in a planning group have to be sufficiently empathic toward each other to accept feedback even if it is low-inference feedback. The kinds of evaluative activities, therefore, have to be carefully negotiated with the individuals concerned. Some readers might consider that the types of self-evaluation are too superficial and are likely to
lead to over-concentration upon the frequency of occurrence of activities rather than the quality of the actions. Also, time constraints are often so pressing that it is not always feasible to undertake many, if any, of these evaluative activities.
A combined qualitative/quantitative technique, which is widely used in the USA, in the United Kingdom and in other European countries (Visscher, 2001) is the performance indicator (see Figure 11.4). These can be directed specifically at teacher performance (especially teacher competence tests in the USA), at student performance (e.g. the General Achievement Test in Victoria, Australia) or at school-wide issues. Performance indicators are linked directly to specific objectives or goals for a school programme and are intended to indicate the extent of progress made towards a specific objective. Rogers and Badham (1992) suggest that performance indicators should be capable of being collected on several occasions over a period of time.
Depending upon the size and scope of school-level evaluation, persons involved may be a team of one or two external experts, the entire school staff (together with selected school council members) or just one classroom teacher taking up the role of an evaluator. The US evaluation scene is normally dominated by the experts who are hired as consultants to evaluate school district programmes and similar large-scale activities. The literature on evaluation contains numerous references to the characteristics of ‘good’ evaluators (Simons, 1987; Popham, 1995; Wood, 1991) and includes such attributes as technical competence, personal integrity and objectivity.
External, full-time professional evaluators are not very evident on the Australian scene. External evaluators, as members of a team to undertake school evaluations, are found in all states but they are mostly experienced teachers and school principals who serve on evaluation panels for short periods of time, including site visits of one or two days. In the United Kingdom, the Office for Standards in Education (OFSTED) has recruited a wide range of registered inspectors and inspection contractors who are in turn subject to inspection quality audits (OFSTED, 1997). Internal evaluators, by contrast, are persons who are involved in, and responsible for, duties in a specific school. A pair of teachers in a primary school or a small team of teachers from within a subject department at the secondary school level, might undertake small-scale evaluation activities. These individuals may turn to external experts for particular forms of assistance – for instance, in designing the appropriate data-gathering instruments, or in developing appropriate criteria for validating the evidence. On occasions, school staff may be able to obtain small grants to employ external consultants for particular tasks, such as initiating the evaluation exercise, coordinating the diverse activities or collecting some of the data (e.g. observing teachers in the classrooms). Checklists of specific questions are a very useful way of providing evaluators (individual evaluator or a team) with the necessary guidelines.
The management of schools, system wide or individually, brings attention to bear on performance issues and matters of evaluation. Various stakeholders want information about achievements (especially in terms of the students, teachers, subject matter and milieu) to justify the substantial financial expenses. In addition to accountability reasons, participants in a school community need to ‘sample the temperature’ of what is going on so that development plans can be targeted to areas of need. There are a range of techniques available for obtaining evaluative data about teachers, students and the milieu. However, if participants at a school are not committed to regular evaluation activities and are not willing to produce developmental, strategic plans based upon evidence obtained from these evaluations, little can be achieved.
Reflections and Issues
1. ‘Evaluations are designed increasingly to be used, to accompany or initiate changes in schools and central offices’ (Rogers and Badham, 1992). Do you agree? If this is the case what are the implications for the time taken and who initiates the evaluation?
2. ‘‘‘Value-added’’ measures indicate the educational value that a school adds over and above that which could be predicted given the backgrounds and prior attainments of the students within the school’ (Hill, 1995, p. 6). What are some exam ples of value-added measures? Comment on their potential successes and problems.
3. ‘In the last ten years we have witnessed a rapid growth in school self-evaluation models and practices . . . What is least clear and most controversial in this range of activity is who has control of the process, who has access to any product that emerges and whose interests are served’ (Simons, 1987, pp. 319–20). What groups do you consider are controlling school evaluation processes? Are you aware of successful evaluation efforts? What do you consider are some of the major inhibiting factors?
4. ‘Evaluation can be a constructive process leading to stronger professionalism, but only if teachers grasp the opportunity for reflection and growth that it presents.’ (Granheim, 1990, p.1). Do the evaluation approaches with which you are familiar allow teachers to ‘reflect and grow’? What are some important safeguards you would propose to allow this to happen?
5. ‘In the final analysis the evaluator’s role is to assess the educational quality of the curriculum policy or program. But (s)he can still do this democratically through dialogue and discussion with a variety of interest groups, including practitioners. Through such dialogue an evaluator can deepen and extend his or her own understanding of the nature of educational values and how they can be best realised in particular contests’ (Elliott, 1991, p. 231). How important is the dialogue and discussion between interest groups in a school evaluation? What techniques can be used to achieve it? Elaborate upon some of the restrictions.
6. ‘Evaluation is a form of inquiry whose end product is information. Information is power, and evaluation is powerful.’ (Guba and Lincoln, 1989, p. 56). Can school evaluations be powerful? Which stakeholders are most affected by school evaluations? How can their needs be communicated and respected? Use examples to illustrate your point of view.
7. The Education acts be legislated for the local management of schools. ‘Any school which seeks to use management information effectively for planning purposes will need to devise systems for integrating a review of:
. curriculum delivery and pupil outcomes;
. staff appraisal and development;
. use of finance and other material resources (Rogers and Badham, 1992, p. 85).
Describe how you would plan an integrated evaluation of these elements. What might be some potential constraints?
No comments:
Post a Comment