Quality Management and Study Abroad
This is the fifteenth in a series of “food for thought” pieces from CIEE. The themes vary but all deal with study abroad for U.S. undergraduates. We present our ideas not as the only viable ones but rather to stimulate discourse in furtherance of the study abroad enterprise. Previous topics include:
– How Are We Doing?
– Standards
– A CIEE Eye for the Study Abroad Guy...or Girl
– Parents, Pills, & Pandering
– A Research Agenda for Study Abroad
– What’s It All About?
– Numbers
– Mirror, Mirror on the Wall
– Down With America
– Beware the Wolf in Sheep’s Clothing
– Leadership, Management, & Study Abroad
– Be Careful What You Wish For!
– Moving Beyond the Gap
– A Good Time
We welcome your comments and requests for additional copies at any time to editor@ciee.org.
A recent survey of concerns and priorities conducted by the Forum on Education Abroad lists program quality as the number one concern of most study abroad professionals. We certainly understand that there is pressure on campuses to ensure that the quality of an educational experience abroad is comparable to the quality of the educational experience on campus, and that there is a great deal of variability about how this is addressed.
Quality management varies widely among institutions, but there seems to be little difference in the standards applied to faculty led and staffed programs, irrespective of program location, i.e. on campus or overseas. College and university faculty teaching abroad, as when they teach at home, have wide latitude in the conduct of their courses. For programs conducted by campus faculty, there is typically no quality control mechanism.
For universities and colleges that use foreign institutions and/or third party providers for their study abroad needs, most campuses have an approval process for adding programs to the portfolio offered to students. On some campuses, there is a single committee that handles all such approvals. On other campuses, individual departments often have sway in the approval process. There are also many hybrid approaches to quality management and program approval. Suffice it to say that while there are variations between institutions, in general and on most campuses, some process exists to ensure that overseas programs available to students meet campus-level quality standards.
None of this effort deals with the fact that on many public campuses the school is required, within broad limits, to accept transfer credit from other U.S. accredited institutions; in essence, transfer credit. Therefore, students can pick programs overseas that use a school of record in the U.S. and get credit for these courses, almost irrespective of any quality management initiative and whether or not the school and the study abroad office believe that the program is a good one. All of us are familiar with programs of this type that are less sound in a variety of ways, and not necessarily the best part of the study abroad landscape.
More than a decade ago, we began to address the quality issue at CIEE in a number of ways. While much of what we do is familiar to people within the field, the details of our approach and the values underlying it are often lost in the day to day shuffle of life. This “Our View” seeks to reiterate our approach to quality management in study abroad–both in terms of the values driving the process and the operational implementation of quality management methodology.
Underlying our quality management system initiatives are a series of values.
First, we do not believe that the best measure of effectiveness is to ask “Mirror, Mirror on the Wall….” In a simple sense, none of us are good judges of our beauty, our effectiveness, or, in a more serious vein, our program quality. While student and internal program evaluation is important, useful, and part of any decent quality management effort, without external, impartial, and independent feedback, it is simply impossible to know what a program is delivering and whether that delivery meets program goals and expectations. It’s pretty easy to fool yourself; but, in our experience, a knowledgeable field audit team rarely misses their mark.
Next, secret quality management isn’t quality management at all. If one wants to make quality the core of an activity, visible, public, and transparent quality management reporting is essential. If people believe that their actions and views will never meet the light of day and public scrutiny, they behave differently than when they know that judgment and feedback is going to be open and available to all interested parties. Total transparency, with the exception of sensitive personnel and similar types of information, is essential. All CIEE evaluations, by students and by evaluation teams, are available on our website.
Finally, programs have a life cycle and a tendency to change with time. Therefore, for evaluation to work, it needs to be continuous. When new elements of a significant nature are added to a program; when important changes in leadership and affiliation take place; and/or when the cultural setting, goals, and/or conditions under which the program operates change, evaluation focused on the impact of these changes is essential. In our value set, the process of ongoing evaluation of program elements is as important as more complete program evaluations that take place every three to eight years.
Supporting these values is much easier to say than do. We’ve developed and refined our process, over 15 years of trial and error, and have been aided by many study abroad professionals, both internal and external to the organization. If we’ve learned anything, it’s that time, money, and professional know-how are needed to make a quality management effort work.
At the core of our effort is the CIEE Academic Consortium Board (ACB). This group of study abroad professionals is elected by CIEE Academic Consortium members and charged with oversight of program development, evaluation, and ongoing monitoring. The group meets formally twice a year but also participates in the process of constant updating, ongoing program monitoring, and consortium outreach throughout the year. The people whose names appear at the end of this piece give generously of their time, know-how, and energy to complete the work of the ACB. They are independent and, in some cases, non-users or very limited users of CIEE programs. They are selected for their expertise, ability to lead evaluations, diversity, and so forth through an ongoing nomination and election process.
Formal evaluations of programs take place every three to eight years. With ongoing monitoring of all programs, some very stable programs with little change do not need to be evaluated as often as those that are new, changing, and/or in other ways require attention. Each year, the ACB develops an updated two-year schedule of programs to be evaluated. A member of the ACB is charged with leading an evaluation team and, with the help of members of the Consortium, two outside independent faculty members with expertise in the content of the program are recruited to participate in the evaluation. Prior to the site visits, evaluation teams review program goals and objectives, curriculum and program conduct, student evaluations, school feedback, program monitoring reports, and other relevant data. Evaluation teams visit programs for five intensive days of observation, meeting with students, staff, and faculty, and inspecting facilities, housing, and so forth. It’s a grueling process and includes writing a first draft of the evaluation report on site so that feedback is captured fresh. A typical report runs twenty or more pages and involves a set of elements; classes, facilities, etc., uniformly addressed at all sites. Reports are reviewed by the ACB, discussed, and published on the CIEE website. Recommendations from the report are reviewed and responded to as part of ongoing monitoring, so that each issue raised on a site visit is addressed.
Between full evaluations, as noted above, every program is monitored annually. Each member of the ACB is responsible for a group of programs and a monitoring report on key program elements is completed, rating the program on each core component. Components can be ranked as fully acceptable or requiring attention, and/or intervention, at which point action is taken. Overall, the combination of monitoring and full program evaluation feed one another to ensure that programs are always in the quality limelight. And, from time to time, an interim evaluation takes place, often related to a new or very small program, or to see the impact of a change in programming outcomes. These interim evaluations have served as a useful adjunct to the full evaluation and monitoring process, and are conducted by one or two present, or in a few cases past, members of the ACB who are familiar with the process and have special language or other skills related to the program in question.
This is not exciting, light up the sky, flash of brilliance kind of work. It involves enormous attention to detail, a great deal of time and commitment on the part of the ACB and the staff that support them, and a significant budget allocation to pay the expenses of meetings and travel. However, it pays off in a variety of ways:
- Program quality is a concern and must be addressed if study abroad is to continue to grow in scope and depth.
- No program is perfect. Every program has opportunities for improvement, change, and development.
- The ongoing process described herein ensures that programs meet their full potential year after year. It takes a long time for every stakeholder in this process to appreciate the value of the effort. Let’s be honest, resident directors, faculty, and other parties of study abroad aren’t always that excited about having a team evaluate “their” program. The interesting thing is that once they’ve been through the process, they usually become its biggest supporter.
- There are many ways to whitewash this process, i.e. to make bad results look good and/or try to ensure that all news is good news. The effort it takes to do this is often greater than simply doing it right, letting the chips fall where they may, and addressing whatever issues arise. If you take the time to have a physical, you should listen to the results and take action, not look for an explanation to wish the results away. The more the culture of the organization embraces the process, the more everyone benefits and the easier it is to have open, positive discussions about program quality.
- The ACB members do an amazing job. Over the years, virtually all of them have pointed out that while they give a lot, they get a lot. Working with their colleagues, they develop insights into program design and management, have a chance to think critically about program execution issues, and develop strong networks upon whom they can rely for information across a wide variety of issues.
We do not mean to suggest that our way is the only way to conduct program evaluations. Various schools and other program providers will address these issues in different ways with greater or lesser impact. Rather, we present this information based on many years of work defining, refining, and developing a process that clearly serves the needs of some very discriminating stakeholders. We know a good many institutions have used components of our process for their own programs, and we welcome others to do the same. Detailed information on the evaluation process is available here.
In these times, when money is tight and each program must deliver what it promises to be effective, quality management is a key activity in the ongoing growth and development of study abroad. As those in the field think about their own efforts, we hope that our view will help them further develop their own evaluation processes.
2009 CIEE Academic Consortium Board Members
Paul DeYoung, Reed College
Joël Gallegos, University of North Carolina at Charlotte
Brian Harley, Purdue University
Joe Hoff, Oregon State University
Nancy Kanach, Princeton University
Cheryl Matherly, The University of Tulsa
Kathleen McDermott, Columbia University
Ellen Sayles, Oberlin College
Kathleen Sideli, Indiana University
Neal Sobania, Pacific Lutheran University
Michael Stohl, University of California, Santa Barbara
Susan Thompson, University of Nevada, Las Vegas
Past CIEE Academic Consortium Board Members
Elizabeth Andretta, Georgetown University
Al Balkcum, University of Minnesota
Thomas Bartlett, Johns Hopkins University
Adrian Beaulieu, Providence College
Terrance Bigalke, Beloit College
Paul Brown, Clark Atlanta University
Julius Coles, Morehouse College
William Cressey, Georgetown University
William Davey, Arizona State University
Lawrie Davidson, Murdoch University
Michael Delaney, University of Colorado at Boulder
Duleep Deosthale, University of Alabama at Birmingham
Carol Dickerman, University of Michigan
Valerie Eastman, Scripps College
Jane Edwards, Yale University
David Fenner, University of Washington
Margery Ganz, Spelman College
Allen Green, Wesleyan University
Elizabeth Hayford, Associated Colleges of the Midwest
Kathryn Hubbard, University of Washington
Chuanren Ke, University of Iowa
Maryelise Lamet, University of Massachusetts at Amherst
David Larsen, Arcadia University
Jacqueline Levine, University of Rochester
Thomas Manley, Pitzer College
Ruth Mason, Gustavus Adolphus College
Ivy McQuiddy, University of Texas at Austin
Frank Miller, Columbia University
Michael Monahan, Macalaster College
Kirsten Moritz, Brown University
Charles Ping, Ohio University
Thomas Ricks, Villanova University
Brenda Robinson, Grand Valley State University
Gary Sibeck, Loyola Marymount University
Nancy Stubbs, University of Colorado at Boulder
John Sunnygard, University of Texas at Austin
James West, University of Washington
Julian Wheatley, Massachusetts Institute of Technology
Ana María Wiseman, Wofford College
Peter Wollitzer, University of California
Kristi Wormhoudt, The Pennsylvania State University