Are Elite College Courses Better?

DENVER -- The public -- and heck, many people in higher education -- widely assume prestigious colleges and universities provide the best quality education. That's why employers often want to hire their graduates and why many parents want their children to attend them.

And the assumption partially explains the fascination from the media and others in recent years with massive open online courses from Harvard and Stanford and other elite universities: the courses were believed, rightly or wrongly, to be of higher quality than all other online courses precisely because they came from name-brand institutions.

But what if the richest and best-known colleges and universities don't provide the highest-quality education? Would the perceived value of degrees from those institutions decline, and would colleges that were shown in fact to provide higher-quality courses be held in more esteem than they are now?

The push to measure student learning outcomes and other attempts to gauge which institutions, programs and courses most help students learn have been motivated, in part, by skepticism about the assumption that the most famous and selective institutions deliver the highest-quality learning. But the quest for proof to the contrary has at times seemed quixotic.

Researchers at Teachers College of Columbia University and at Yeshiva University, however, believe they are developing a legitimate way to compare the educational quality of courses across institutions -- and their initial analysis, they say, "raises questions about the value of higher-prestige institutions in terms of their teaching quality." They are cautious about asserting that they have proof, and experts on learning challenge some of their assumptions and warn against reading too much into them.

But the study and their approach -- which were previewed here during a session at the annual meeting of the Association for the Study of Higher Education -- are likely to raise questions, and at the very least start an interesting conversation about what and how we define quality in higher education.

Pushing Back Against Prestige

The new research is the work of Corbin M. Campbell and Marisol Jimenez of Teachers College and Christine Arlene N. Arrozal of Yeshiva University’s Benjamin N. Cardozo School of Law, supported by a fellowship from the National Academy of Education and the Spencer Foundation. You get some hints about their perspective from the paper’s working title: “The Mirage of Prestige: The educational quality of courses in prestigious and non-prestigious institutions.”

The researchers work from the presumption that historically, quality and “prestige” in higher education have been defined much more by the “signaling” aspect of an institution or degree (the extent to which employers and others see it as evidence of a student’s potential for employment or leadership) than by proof that it has actually been “transformative,” cognitively and otherwise, to the students who have gone through it.

A whole infrastructure has mimicked and reinforced this bias, the researchers argue, with rankings such as U.S. News & World Report elevating the values of the high-prestige institutions (selectivity in admissions, research over teaching in faculty work, high institutional spending) and influencing the behavior of many students, many institutions, and some governments and other funders.

And by favoring the traits that gain colleges and universities currency in the rankings and all that follows, the researchers posit, colleges and universities adopt trappings and practices (getting more selective, etc.) that strengthen their signaling potential at the expense of those that make them more likely to focus on transforming students through quality education.

“Given that the prestige structure in higher education has bifurcated the signaling and transformation missions, we consider the possibility that higher-status institutions (via the rankings) may fulfill the signaling mission, but institutions that are lower in status may better fulfill the transformation mission.”

So how do the researchers go about trying to define and measure the quality of education, arguably a holy grail? By sending actual faculty observers into nearly 600 classrooms at nine colleges and universities with various levels of prestige and having them judge the teaching quality and academic rigor of the courses they offer, using a common rubric on which the observers have been trained for about 30 hours. The nine institutions -- three with high prestige, two medium prestige and four with low prestige -- were a mix of public and private, teaching and research intensive.

(Teaching quality was defined as instruction that displayed the instructor’s subject matter knowledge, drew out students’ prior knowledge and prodded students to wrestle with new ideas, while academic rigor was judged on the “cognitive complexity” and the “level of standards and expectations” of the course work.)

The researchers acknowledge many limitations in their approach (about which more later), and characterize the study as only a “first step toward examining the relationship between prestige and in-class practices.”

But they found that on only one of the five measures, cognitive complexity of the course work, did the elite colleges in the study outperform the nonelite institutions.

On two, standards and expectations of the course work and the level of the instructors’ subject matter knowledge, there were no meaningful differences by prestige level. On two others, though -- the extent to which the instructors “surfaced” students’ prior knowledge and supported changes in their views, the lower-prestige institutions outperformed the elite ones. (Drilling down, there were differences between the prestige levels for the public institutions in the study, but not between prestigious and nonprestigious private nonprofit ones.)

“This is particularly surprising given the substantial variation in prestige across institutions included in this study: low-prestige institutions were largely unranked and broad access, while the high-prestige institutions were national institutions, highly ranked and highly selective,” the researchers write.

Cautions Aplenty

In the session at which the prestige paper and two others were presented at the higher ed research meeting here, Karen Kurotsuchi Inkelas, an associate professor at the University of Virginia who was charged with responding to the studies, identified some potential weaknesses in the analysis.

While a total of 587 courses were examined -- “astounding work,” Inkelas said -- in each case the assessors observed only one class section. “Can you really know whether it achieves goals by attending one class?” she asked. “If I’m teaching a 15-week course, does one class really represent the quality of my teaching?”

Campbell, the lead researcher, said it would be wholly inappropriate to judge any individual instructor based on one observation, since she or he might have had a bad day. But looking across hundreds and hundreds of courses, it’s reasonable to think that average performance holds up, she said.

Inkelas also questioned the extent to which the raters themselves had subject matter expertise, such that they were in a good position to judge the expertise of the instructors. Campbell said the researchers "did our best" to match the subject matter of the raters to the classes they observed.

A member of the audience (and this reporter) asked how the researchers’ definition of “quality” meshed with the national push to try to judge institutions’ performance based on student outcomes.

Campbell said her colleagues’ approach was an attempt to “push back on the outcomes movement a little bit,” since colleges have so little control over many of the economic and other measures on which policy makers are trying to judge them.

“One thing institutions do have control over is using the practices that we know have been related to student learning, and to do more of them,” she said. “This really is malleable by institutions, so I’d like to think there could be more buy-in. ‘Measure me on something I can actually do, actually change,’” she said. “Part of our responsibility as a field is to think about better, more complex ways to think about” quality.

George S. Kuh, the Indiana University researcher who is a strong advocate for more focus on student learning outcomes, was not at the ASHE meeting (though he was honored with a lifetime achievement award in absentia on Friday). But via email, he also questioned the researchers’ decision to measure not actual student learning, but classroom techniques that may or may not produce more learning.

“There is little to no evidence that what instructors do is a precursor to what students do or learn,” Kuh said in an email. “That is, how is student behavior affected by the study's measures of teaching quality? The guiding assumption is that observed measures of -- for example -- cognitive complexity of readings or lectures somehow spurs greater levels of student complex thinking and behavior. In the absence of evidence of actual student performance … we are left to assume that the measures of teaching quality used in this study really do represent educational quality (i.e., better student performance/more learning, greater proficiency in applying learning and so forth). Probably in some instances, but likely not in others.”

But Kuh and others who reviewed the research also praised the researchers for their efforts to get inside what one called the “black box” of instruction and learning, and for persuading hundreds of faculty members to let outsiders peer into their classrooms and judge their work. (Not to mention that they convinced three highly selective institutions to participate, albeit anonymously, in a study in which there was arguably nowhere for them to go but down in perception.)

As a new starting point for a strand of research aimed at gauging the quality of instruction and education across colleges and universities in a rigorous way, they suggested, this paper could be important.