|
Educational Technology Evaluation Design
Introduction
Sun Associates, an educational technology consulting
firm, specializes in designing customized evaluation and
assessment procedures to evaluate and assess the impact
of educational technology on teaching and learning. Our
evaluation work is designed to take schools and
districts beyond their current strategic technology
plans and into an area where they can begin to assess
their progress in meeting the vision and goals of these
plans. Our work centers around a process which combines
practice profiling, the development of indicators of
success, and the creation of data collection procedures
which are to be used to perform a formative progress
evaluation of a school or district's educational
technology initiative. In many cases, our clients choose
to combine our work in designing an evaluation procedure
with an audit of current technology implementation, and
various reporting options. These options might include a
final written report which details the developed
evaluation strategy and provides a summative evaluation
for the activities of the school/district's technology
efforts.
Framing the Task
Many of Sun Associates' clients have made dramatic
commitments to implementing and integrating technology
in all areas of their educational environments. These
schools and districts are making steady progress towards
fully integrating technology tools in all areas of
teaching and learning. Nevertheless, as with all
well-managed implementation efforts, questions of
performance must be asked. These questions are necessary
not just to find out what is and is not working, but
formative evaluation of performance is a critical factor
in fine-tuning and improving future implementation
efforts and planned activities. In this way, evaluation
is a key component of all educational technology
planning and implementation efforts. The question then
turns to one of how to best accomplish this evaluation.
Activities
Sun Associates works with school districts across the
country to design and implement evaluation strategies
which inspire reflection and change. Our process for
doing this is rather simple and roughly breaks down into
the following steps:
1. Development of evaluation priorities with the
participating district.
We understand that the evaluation should focus
specifically on the activities of a school or district's
current technology initiative. In many cases, this means
that evaluation will be taking a hard look at the impact
that staff development has had on technology
implementation. In all cases, we resist looking at only
technology infrastructure. Districts are fully capable
of "counting computers" on their own. Our task is to
help schools and districts articulate their original
vision and teaching and learning goals into viable
evaluation priorities and questions. In other words, we
help districts answer the question "What difference
is all this technology making in terms of teaching and
learning?"
2. Work with a representative team of district staff -
classroom teachers (users), administrators, trainers,
and others as identified by client - to develop practice
profiles of what constitutes effective implementation of
the chosen priorities.
This is the heart of the evaluation process as it
defines the indicators to be used to evaluate success in
the chosen priority areas.
For example, if we focus on just one aspect of
technology implementation - staff development -we may
ask "What does truly effective teacher use of technology
look like?" In this way, the team identifies its optimum
desired outcome for use of the particular technology.
One of the major roles of Sun Associates consultants, as
facilitators in these discussions, is to insure that the
team does not under or over estimate their desired
outcomes. This is why the process requires a facilitator
experienced in both the realities of the chosen district
and the full range of possibilities for any particular
technology.
Indicator development does not stop with defining
indicators for optimum success. Rather, the team is
asked to define indicators which describe "less than
optimum" levels of achievement that when arranged in
order would culminate in the optimum success. Arranged
in a matrix, these indicators form a rubric which
will be used to measure the school or district's success
and progress in meeting desired outcomes.
3. Develop data collection procedures
Once indicator rubrics have been developed for all
desired elements of priority areas, the school or
district evaluation team develops lists of data and
other supporting evidence that needs to be collected to
measure the district's performance within the rubric. We
facilitate the development of data collection process,
and are sure to add in various data collection
procedures that we are aware of and that the team might
overlook.
It is important to note - and this is a point that
always needs to be made to the project team - that data
is not restricted to quantitative information and/or
simple survey data. Rather, data can and should include
things such as interviews, observations, and a variety
of stories about the experiences involved with
technology integration.
In addition to the above, Sun Associates can create a
final written report that collects all of the
indicators, rubrics, and data collection
devices/objectives. These materials can be organized
into an evaluation plan for the district and presented
as such to the district's Technology Committee, School
Committee, or other policy/oversight board.
Summary
In summary, Sun Associates works with schools and
districts to create and implement educational technology
evaluations which have the following features:
-
The evaluation
methodology makes use of a number of indicators that
measure degrees of success in all key technology
program areas. In particular, we focus on helping
our clients understand and evaluate the impact of
technology on teaching and learning.
-
Under the guidance
and facilitation of Sun Associates consultants, a
representative team of school or district personnel
work to develop their own unique indicators of
success. This team is also trained to develop data
collection priorities and requirements.
-
The indicator
development process is one which is designed to
inspire considerable reflection as to what
constitutes "effective" technology integration. By
requiring stakeholders to set their own criteria for
success - as guided and monitored by a knowledgeable
facilitator and other outside informational
resources - this evaluation process has the effect
of strengthening a school or district's
understanding of technology integration and
specifically how it impacts teaching and learning
.
Sun Associates' Qualifications
Sun Associates is an educational technology consulting
firm specializing in issues related to the integration
of technology in primary and secondary schools,
districts, and classrooms. We take a broad approach to
technology implementation which emphasizes the need to
link the efforts of teachers, students, administrators,
parents, and other community members in a common
understanding of the uses and value of classroom
technology. This approach is realized through services
centered around:
-
Assistance to school
and district strategic technology planning
committees
-
Evaluation of school
and district educational technology implementation
-
Assistance to
educators on choosing and using educational software
and hardware
-
Assistance to
educators conceptualizing and creating on-line
information, servers, and other electronic
publications
Several examples of the firm's recent work are listed
below:
-
Evaluation of
Lexington (MA) Public Schools Year Two Technology
Plan Implementation
evaluation of the impact and effectiveness of the
unique train-the-trainers approach that this
district is taking for developing basic technology
skills in all of its elementary and middle school
teachers. The evaluation process involves developing
indicators of success and rubrics which the district
will use in its evaluation and on-going technology
planning efforts.
-
Evaluating the Impact
of Technology: A Workshop Series for Kentucky School
Districts-
an
on-going series of two-day workshops for school
districts in Kentucky to develop procedures and
plans related to the evaluation of technology
implementation efforts. Held in partnership with the
Appalachia Educational Laboratory, these workshops
provide hands-on experiences related to
understanding the basic types of educational program
evaluation; the links between strategic technology
planning and the evaluation of technology
implementation; and developing processes and
procedures for evaluating district educational
technology implementation.
-
An Evaluation of
Educational Technology Implementation for Warren
County (KY) Independent Schools
- a six month study, with a 150-page final report
and various presentations for a 15-school district
in south-central Kentucky. This study examined the
current use of educational technology across the
district. The goal of this project was to assist the
district in assessing the effectiveness of current
technology use and the implementation of its
district technology plan.
-
Virgin Islands
Territorial Five Year Master Technology Plan
- guidance and technical assistance to the
territorial government in the creation of a master
plan for technology implementation in schools and
administrative offices. Specific services have
included providing assistance in the overall design
of the plan, facilitating planning committee and
subcommittee meetings, editing and critiquing the
overall plan, and conducting Territory-wide
workshops on school technology planning.
-
West
Windsor-Plainsboro (NJ) Public Schools Educational
Technology Plan
- guidance and technical assistance in the creation
of West Windsor-Plainsboro's district-wide
technology plan. Specific services include
facilitating the development of a vision for
educational technology; developing and administering
a district technology needs assessment; developing a
professional development plan; and overseeing the
development of all key strategic plan components for
submission to and approval by the New Jersey state
department of education.
Timeline
The time a school or district takes to develop and
implement an educational technology evaluation varies
greatly. Significant factors that determine the amount
of time required are:
-
The size of the
school or district
-
The scope of the
evaluation effort - i.e., how many evaluation
questions/indicators are to be developed and
explored?
-
The amount of
time/effort a district is able to devote to
technology evaluation and on-going planning
Most of Sun Associates' evaluation projects take
approximately six months to complete. During this time,
a school/district evaluation committee would meet
monthly. The first several meetings are full or half-day
events and are used to orient the evaluation committee
to the process and to begin to develop indicators. Later
meetings are generally short, after-school, meetings and
are used to check progress and coordinate efforts. Data
collection takes several weeks and is usually dispersed
over several months.
Budget
Like the evaluation timeline, the budget for an
evaluation project varies. Significant factors that
determine budget are:
-
The size of the
school or district
-
The amount of
time/assistance Sun Associates provides in the data
collection phase
-
The degree to which
Sun Associates assists the evaluation team in
writing a final report
|
