Applying Values to Our Values
Data may be objectively measured, but to tell a story using data — to extract meaning from a data set about a larger set of questions — requires judgment that is inevitably subjective. The answers one arrives at depend very much on the questions one asks, as well as the context of the line of questioning.
This is illustrated by the humble baseball card, that traditional compendium of statistics on player performance. The emphasis placed on different baseball statistics has changed over time, and the analysis used to determine the merit of a player depends on the values of whoever is interpreting the data: the relative importance of batting average versus on-base percentage; whether to invest much meaning into the nuances of WAR (“wins above replacement”); the intertwined data on a pitcher and on the defensive prowess of the team around him. In other words, our measure of the value of a player results from our application of a particular set of values — ideas that we consider important — to the numerical values or statistics. The data themselves lend very little meaning on their own.
This fundamental principle — that the ability to use data to answer important questions depends on the system of values or priorities that we bring to the problem — addresses both the potential and the limitations of the College Scorecard, unveiled online by the White House on Saturday. On the surface, the Scorecard assesses colleges and universities much the way a baseball card assesses players. Each institution is characterized by three “headline” statistics — average net cost, graduation rate after six years and median salary 10 years after matriculation. This is much like the classic trifecta of batting average, home runs and RBIs for a position player that for years graced the front side of a baseball card. But, just as you can turn over a baseball card to get more details, a few clicks on “View More Details” on an institution’s College Scorecard page will reveal demographics, student debt information and much more. By doing searches with filters on different parameters, one can sort and compare institutions by key statistics. This is a powerful tool that will enable all of us to understand the metrics of higher education in greater detail.
Not surprisingly, Kenyon performs well in this review. We are above average in the headline statistics, and when you dig deeper you will discover that we have a relatively low level of student indebtedness and our graduates have a very low default rate on their student loans. Yet, while I appreciate having access to this great resource of information and I appreciate that Kenyon performs reasonably well on the Scorecard, I find the choices made on the headline priorities puzzling. Nowhere among these top statistics is a metric that measures what a student actually learns in college, in many ways the core of an institution’s mission. Instead, the salary-after-matriculation is a statistic that is more closely related to field of employment (i.e., engineers are paid more on average than school teachers) than it is to the central question of how well the school educates its students. The headline statistics reduce the college experience to the simplest return-on-investment analysis possible: how much it costs students, the rate at which they graduate and how long it takes them to recover that financial investment.
Fundamental questions are missing here. What did students learn? Were their lives enriched by the experience? Have they become leaders in their communities? Have they had an impact on our world by creating art, making scientific discoveries, building businesses or holding elective office? These qualities may seem harder to measure, but they are not impossible.
In this Sunday’s New York Times, Frank Bruni described one approach to this, a study led by Gallup and Purdue University that aims to correlate components of the undergraduate experience with workplace engagement and post-graduate well-being. An initial report from this study, called “Great Jobs Great Lives,” was released in December 2014; an update (with survey data from an additional 30,000 college graduates) will be released later this month. This approach — rigorous in method — broadens the definition of return on investment beyond the simple financial rubric used by the College Scorecard and begins to capture the career and personal benefits that may arise from an excellent college education.
What makes a first-rate college experience? As Bruni describes, the study reveals that graduates “fared better if, during college, they did any one of these: developed a relationship with a mentor; took on a project that lasted a semester or more; did a job or internship directly connected to their chosen field; or became deeply involved in a campus organization or activity (as opposed to minimally involved in a range of things).” In other words, a fulfilling and successful life may be better predicted by how students invest their time while at college than by median salary after matriculation.
This Scorecard is a start, but a more helpful Scorecard would include more comprehensive information about the opportunities available to students, and our aim should be to describe the college experience as accurately as possible.