CIEE Study Centers Academic Consortium
Board Program Review
A. Introduction
CIEE operates all of its programs and services following four basic principles: quality, service, responsibility, and efficiency. The Academic Consortium, represented by the Academic Consortium Board, oversees all of the CIEE Study Centers by conducting periodic evaluations of the programs and reporting its findings to the Consortium.
The purpose of program evaluation is to provide members of the Academic Consortium with the reasonable assurance that the programs which the Consortium member institutions are sponsoring are of high quality, and in particular of high academic quality. We ask what needs are served by the program in this location, what is the program's role in CIEE's overall plan, is the program academically sound, is it viable, do activities support the goals of the program, are local resources well used, does it have adequate staff support locally and from CIEE home office, what are the program's plans for the future, what are the program's internal processes for evaluation of the quality of its academic and support services, etc. But the baseline remains: is the program serving the needs of two primary stakeholders, the Consortium sponsoring institutions and these institutions' students?
B. General Principles
Each program must have a clearly defined purpose—a mission statement. Our review process measures against that stated purpose.
Review must be regular and systematic. Based on an internal review conducted by the staff, the ACB monitors program quality every year. This process is described in a separate document. The focus of this document is the comprehensive evaluations of each program that are undertaken every three to ten years.
Self-evaluation is the best guarantee of program quality and improvement. For this reason, the ACB reviews are conducted with the full participation of the staff and students of the program under review. The members of the review panels, however, will be external to the program.
The review process must not impose an undue burden on members of staff nor impede the normal pattern of teaching and assessment.
Evaluation provides an opportunity to prepare for the future. Therefore, the final evaluation report should be a useful planning document.
C. Terms of Reference (Nature
and scope of the review)
All reviews shall be conducted in the context of the strategic objectives of the CIEE Academic Consortium Board and the stated purpose of the program. Each review normally shall report on:
- the quality of the academic activities including teaching and assessment of student work;
- the appropriateness of the academic activities;
- the quality and appropriateness of services (pre-departure and on-site orientation, academic and personal/intercultural counseling, housing, field trips, internships, etc.);
- the quality and appropriateness of the students who participate;
- the quality of the program's facilities (office and class space, library resources, equipment);
- the program's plans for improving its teaching and services;
- the program's relationship with the host university;
- the perceived satisfaction of the Consortium member institutions;
- the perceived satisfaction of the student participants;
- safety and security issues;
- the program's compliance with CIEE's policies;
- the quality of administration and management of the program; and
- the relationship of this evaluation to previous evaluations.
Prior to the beginning of a review, the Chair of the ACB, CIEE Program Director, the Chief Academic Officer of CIEE and the Vice President of Programs of CIEE, will meet to decide whether any special terms of reference should be set for the evaluation, and whether there are any strategic issues or particular problem areas which should be drawn to the attention of the evaluation panel. The Chair of the ACB may add or amend terms of reference for any review.
D. Review Panels
There are two types of program evaluations – interim and full. In 2004, the ACB agreed to conduct an interim evaluation where a program is new and has yet to enroll more than a small number of students; for programs where a significant issue arose during a full ACB evaluation and where quick follow up is appropriate; and for programs that may not require a full evaluation as regularly as others. Interim evaluations are typically a single person evaluation by a serving ACB member.
For full evaluations, the evaluation team will typically be composed of three members appointed by the Chair of the ACB and the ACB Evaluations Coordinator. Whenever possible the Chair of each evaluation team will be a member of the ACB; the other members will be appointed from among the faculty/staff of Consortium member institutions. Balance between teaching faculty and study abroad administrators will be sought in each evaluation committee, and members will be chosen in order to maximize the team's linguistic and area expertise and disciplinary expertise which match the program's curriculum. Balance should also be sought regarding gender of team members and types of institution—large versus small, public versus private, etc.
The Program Director and other home office staff, the Resident Director of the CIEE Study Center under review, the current student participants, and the faculty/administration of the host institution will be integral contributors to the review process, but will not be members of the review team.
E. The Review Process
The Review takes place over a period which can be as long as six months. Several steps lead up to the visit, which is the culmination of the process.
Review of Existing Information
The first step is a thorough review of existing program information and previous evaluations. Each review team member will be provided by CIEE with the following information about the program in either a pre-departure binder of information or by e-mail attachment not less than two months prior to the site visit. The Table of Contents of the binder will refer members to specific location of information for any items that are not included in the binder.
The Binder includes the most recent Course Catalog that includes the program’s statement of purpose, admission standards, nature of courses, credit formulae, general calendar, and details regarding housing and the and co-curricular Program.
Binder Table of Contents
Section I: Overview
A. Contacts, Logistics, Schedule
This document will be updated on regular basis as the team prepares for departure. The goal is to have all logistical information contained in this one document.
B. Plan for Program Review
A copy of this document is included.
Section II: Staffing
A. CIEE staff
A listing of Portland based and on-site staff directly involved in the program.
B. Job Descriptions of on-site staff
Current job description(s) for resident staff are included.
C. CV’s of on-site staff
Current CV’s for all on-site staff
Section III: The Program
A. Program History
A brief history of the program
B. Terms of Reference Update
An update (when appropriate) to the standard terms of references indicated earlier in this document. This document specifies any issues with the standard terms of reference that the team should focus on during the visit. This will also encompass any issues raised in the last year either through the program monitoring process, by someone visiting the site, etc.
C. Previous ACB Evaluation Report
The previous evaluation report printed from the CIEE website
D. Previous Monitoring Reports and/or Critical Component Reviews
Each year the ACB monitors each of the CIEE Study Center programs. (The exception to this is when a program has been evaluated, i.e., a monitoring report is not done the summer before or the summer immediately following a full on-site evaluation.) Monitoring reports were written each summer up until 2002. In 2003, the process changed such that the ACB monitor and the Program Director now discuss the program based on documents provided by staff. Any issues raised by the monitor are then reflected in the Annual Study Center Review written each summer by the Program Director.
E. Annual Study Center Review and/or Resident Director Reports
New in 2004, the Annual Study Center Review replaces the end of session reports written by the resident director and published on the CIEE website. The Program Director solicits feedback from staff throughout the organization (including resident staff) as well as from the ACB and other Consortium member institutions in the drafting of these documents.
F. CIEE Course Syllabi
The most recent course syllabi are provided for all CIEE courses. The syllabus includes a reading list, requirements to be met by each student, and methods of evaluation (testing, papers, participation, etc.)
G. CV’s of CIEE Faculty Members
The most recent CV for all CIEE teachers for the program being reviewed.
H. Recent Visitors to the Program
Reports from Consortium member visits to the site in the last two years.
I. Study Center Bulletin
The most recent CIEE Study Center Bulletin. The Bulletin is the program update that is published each December and lists any changes to the current catalogs that impact the following summer, fall and academic year programs. (The program catalogs are published each August for the following spring semester.)
J. Host Institution Information
An organizational or institutional profile, including faculty and faculty credentials, students, nature of accreditation or recognition, catalog or equivalent, grading scales, etc., are provided
K. Sample CIEE Academic Record for the Study Center
The CIEE Academic Record is the grade report sent to the sending school. It includes a cover sheet on grades and equivalencies, information on the program and the grading system as well as any documentation received from the program site (host institution documents, RD generated reports, etc).
Section IV: Students
A. Student Acceptance Materials (Handbook, Forms, Insurance)
Copies of acceptance materials from the most recent semester.
B. Student Data and Analysis (last 3 years)
Summary statistics of participants including home institution, academic majors, ethnicity, and gender breakdown by semester are included. This historical information will also include a scan of CIEE Academic Records, to determine the number of credit hours generated, and to see how students perform (aggregate GPA or equivalent)
C. Full Student Evaluation Reports (3 years)
CIEE began on-line student evaluations in spring 2003. Student evaluation reports, including student comments, are made available for spring 2003 and recent semesters. For programs under evaluation through spring 2005, CIEE will include copies of individual student evaluations for prior semesters. After spring 2005, student evaluations reports will be provided.
Questionnaire to Sponsoring
Institutions
After reviewing these materials, the team conducts an electronic canvas of the top sending institutions, the School of Record for the program, and a sample of other sending institutions. The team leader should if possible also conduct telephone interviews with faculty and staff at some institutions. Team members will review the questions which were mailed with the letter to institutions, ask if they are satisfied with the program, and ask the institutions to identify areas of concern to which the review team can be alerted, and to assess the degree to which institutional needs are being met. We hope that the responses are based on respondent debriefings of students who have returned, on their conversations with faculty who have these returnees in their classrooms, and on their own knowledge of what constitutes a good international program. Responses will be collected and collated by the chair of the review team.
Sample Questionnaires will be sent to the team by the Program Director.
Invitation to students and
staff of the program
In preparation for the visit, the team will
invite students and staff in the program to make written
submissions to the review panel, and to meet with the
team during the site visit. This invitation is usually
issued about three weeks prior to the site visit.
Sample Invitations will be made available to the Team by the Program Director.
Site Visit
The site visit is the most important step in the process. The visit allows team members to gain direct experience with the participants, staff, and professors. Normally, a site visit by the review team will be of four days duration, and will consist of intensive interviews, discussions and deliberation, formulating conclusions for the report, and writing of the report. Review team members will be required to review all information before arrival, be available exclusively for the review during the site visit, and be available after the site visit for further consultation in the final drafting of the report.
The Team Report to the ACB
At the end of its visit, the team prepares a report for the ACB. Since team members are busy professionals with very full workdays back at their home institutions, it is essential that the report be drafted on site before the members go their separate ways. The last day of the visit is usually devoted to writing the report. The chair of the review team will be responsible for coordinating the conduct of the review and writing the report to the ACB. The report will address each of the terms of reference and make such recommendations for change or commendations for good practice as are appropriate. It will include an executive summary. Reports will be submitted to the Chair of the ACB, the ACB Evaluations Coordinator and the CIEE Program Director within two weeks of the completion of the review. The Team Report is intended to be complete, detailed, and frank in every respect. It is not a public document, although parts of it will later be made public, and will be read in its entirety only by staff and by the board members.
F. The ACB Evaluation Report
The Team Report is discussed at the next meeting of the ACB. Based on the Team Report and on discussions with staff, and on any other information that may be available, the ACB will produce an Evaluation Report intended for public distribution after approval by the full ACB. The production of these two closely related but not necessarily identical reports is an important part of the review process. In most cases, the Evaluation Report will contain most, if not all, of the same text produced by the team. However, the ACB may make some changes if it feels that some material may not be appropriate for a public report (e.g., references to specific individuals, critical comments about a host institution, etc.) or may be misunderstood. The ACB will distribute the Evaluation Report to the Academic Consortium and it will also be posted on CIEE’s web site.
G. After the Review (Follow
Up)
The Reviews are important documents and serve a variety of useful purposes including the provision of objective information to prospective students and to sending schools. In addition, these reviews constitute the basis for improvement of the program. To assure that the review recommendations will be used to maximum advantage, the Annual Study Center Reviews will comment on the progress of the improvements. It is likely that the Program Director and the ACB Team Leader will continue discussion of the recommendations and the ACB Team Leader will monitor the program in the year following the review.
Finally, the ACB Evaluation will form the baseline of the next scheduled evaluation of the program as well as for the yearly monitoring process that occurs between evaluations. Over several years, a dossier of the program will thus be built which will document how the program measures up to CIEE's commitment to quality, service, responsibility, and efficiency.