Warren, Steven F.

Biodiversity Informatics

37     “Let’s Play Moneyball!”: Analytics, Accountability and the Future of Research Universities  Steven F. Warren, Vice Chancellor for Research and Graduate Studies,  University of Kansas  esearch universities make massive investments in research. Many of these in-vestments are obvious and easily accounted for. These include infrastructure (building, specialized equipment), the complex management of external grants, and other easily measured costs. Arguably the largest relatively undocumented uni-versity investment is the “release time” from teaching provided to most tenure line faculty members. The purpose of the release time to allow the faculty member to con-duct research and scholarship. This release time often accounts for 40% of a faculty member’s full-time 9 month appointment. Conceptually it’s typically the difference be-tween teaching 4 courses per semester (a typical load in a regional undergraduate cam-pus that has as its primary mission teaching as opposed to teaching and research) and 2 courses per semester. Furthermore, this investment is an excellent one in the majority of cases in which faculty use this “research time” to actively engage in important and meas-urable scholarship. But what about fac-ulty members who are “inactive schol-ars”? I am referring to full time university faculty members who generate little or no meaningful evidence of scholarship and acceptable creative active scholarship over significant periods of time while still receiving the benefit of this release time.  There are at least two reasons that re-search universities should be concerned about tenure line faculty members who are inactive scholars. First, there may be an ethical issue if these individuals main-tain graduate faculty status that allows them to chair or serve on PhD level doc-toral student committees. These faculty committees are charged with supervising the training of future research scholars. One could argue that an inactive scholar (e.g. someone who has not generated published scholarship in perhaps five years or more) should not automatically qualify for continuing doctoral faculty status simply because they have held this status since they became a faculty mem-ber. That is, automatic qualification should be reserved for active scholars. The ethical issue is that we should want our PhD students to be supervised by committees consisting of active scholars. The second reason is the obvious expec-tation that if you receive release time, you are expected to use it as intended unless given explicit permission to do other-wise. If not, this behavior (or lack of it) is in violation of the implicit and explicit R38  employment agreement that exists be-tween a full time tenure line faculty mem-ber and his employer. It can be hard for administrators to determine whether a given faculty member is using the pro-vided release time for research. Research is often done off campus and in fact many faculty use their home offices or studios as places to engage in their scholarship. Thus, we tend to honor self-report in-stead of direct monitoring of faculty members. This is as it should be. But it can be abused. Nevertheless, scholarship is virtually any type of work that does generate some kind of product. Most of-ten these are easily measured publica-tions. But even in areas such as the visual and performing arts, there are “prod-ucts” than can be measured.  In the past the problem of “inactive” scholars at research universities was most evident to their colleagues. It was the subject of rumors, and perhaps had a negative impact on an inactive faculty member’s salary over time because they didn’t receive raises. However, in the world of electronic publication we now live in, the evidence of this problem is more transparent. That is, it can now be identified by outside groups that harvest information on the productivity of fac-ulty among other things, and then sell these analyses back to universities to help them improve, etc. These aggregators can also sell the same data to other groups such as state legislators and university governing boards. There are cases where this has happened and subsequently cre-ated problems for universities in a few states. Even highly productive flagship research universities are not immune to this problem.  Two Scenarios Consider a scenario in which data on the scholarly productivity of all doctoral program faculty on a state-by-state com-parative basis is made available for pur-chase. Suppose individuals in your state legislature get this data. Suppose your state does poorly (the problem of inactive scholars does not respect state lines). Per-haps the data show that more than 10% of doctoral program faculty at your univer-sity have not published in the last 5 years or more. Suppose your legislature takes this as evidence that a significant number of your faculty members are “inactive scholars”. How would you respond? What actions would you take? This has apparently already happened in a few states.  Or consider another scenario that is perhaps even more problematic. Suppose that other research universities that you compare yourself to, use this kind of ex-ternally captured faculty productivity data to reclassify or remove unproduc-tive scholars from their doctoral faculty and assign them larger teaching loads. Then, as a result of these actions and per-haps others taken over a period of 5 to 10 years, these universities significantly im-prove the overall scholarly productivity of their doctoral program level faculty. Perhaps they also use tools like Academic Analytics to identify departments and programs that need “new blood”, and perhaps some that need to be merged with others, reorganized, or even elimi-nated. They also use these tools to recruit new faculty with a high likelihood of suc-cess and to help retain truly productive scholars. In other words, they use the data to play “Moneyball” and to improve 39  themselves with wise, data based person-nel decisions in ways like the Oakland A’s used productivity data to build a highly competitive major league baseball team on their relatively small budget. Fi-nally, suppose that this strategy, after 5 or 10 years, leads these universities to pass your university by and move up on a wide range of ranking while you sit on the sidelines and maintain business as usual. Can this happen? I suspect it is al-ready happening. So what are you going to do? Are you going to play Moneyball too? Or put your institution at a long term risk because you choose not to play and thus gradually slip behind what had pre-viously been your “peer group”.  These two scenarios are plausible in the world we now live in. What can you do to avoid them? Consider these possi-ble steps: 1. Get the data on your university and your competitors and develop an in depth knowledge of it. Make it easy for deans and department chairs to use this data. Perhaps offer a con-sultation service whereby “analyt-ics counselors” will conduct studies for departments and colleges at their request. Create incentives to get people using this data.  2. Start using the data to make deci-sions about hiring, retention, reor-ganization, etc. That is, make it an active planning tool.  3. Work closely with deans, chairs, and faculty to create a broad under-standing of the serious downside of ignoring this type of data. Not act-ing in the face of the changing world is essentially sitting on the sidelines and possibly watching the relative decline of your university in terms of its effectiveness and competitiveness.  4. Put in place policies aimed at elimi-nating problems like unproductive tenured scholars. The root of this problem may lie in your tenure and promotion system. Better data can inform that process too. In addition, your policies on post-tenure review and differential allocation of effort can make it relatively straightfor-ward to reassign unproductive scholars to higher service and teaching loads, or other activities. 5. Use analytics data to make budget-ing decisions. Make it a meaningful part of the scene.  Cautions Having a huge amount of data is a separate issue from using data wisely. Einstein among others is famous for ob-serving that much of what we can meas-ure is of little real value just as many things we can’t measure are what really matters. Indeed, having a high publica-tion rate and having a high impact and value can be remarkably unrelated. A number of the most influential scholars in history produced only a very small num-ber of publications. Furthermore, lots of papers published in “high impact jour-nals” are never cited in the literature. Nevertheless, the right data, wisely used and qualified can help us identify schol-ars who are no longer active. Further-more, it is necessary that we evaluate scholarly productivity within the fields/disciplines where it resides and against the standards of that field. Other-wise you are simply comparing apples and oranges. Publication patterns differ 40  greatly across various disciplines. Fi-nally, some fields (e.g. the visual and per-forming arts) present significant chal-lenges in terms of evaluating the impact of creative activities in a valid way. This doesn’t mean it can’t be done, but it does mean we need to take great care and tread lightly in these areas.  Final Thoughts Analytics and big data are already having a significant impact on higher ed-ucation in all sorts of ways. Furthermore, we are still in early stages of this big data revolution. There is no turning back from this and no returning higher education to what some consider its “monastic ways”. Indeed, we need to embrace analytics and big data or we will be run over by others that do embrace them. But this is not just about playing defense in an age of rapid change. These new tools present great op-portunities for improving the perfor-mance and impact of higher education in general and research in specific. They are tools that can actually level the playing field for public research universities. That is, it can help lesser endowed institutions become the Oakland A’s of research uni-versities. Billy Beane, the manager of the Oakland A’s portrayed in Moneyball (the book and the movie) is still using sabra-metrics to make the A’s remarkably com-petitive despite having a total annual budget that is less than 1/3 of the New York Yankees payroll. In fact, when I checked the paper this morning, the A’s were leading their division.   References Lewis, Michael. (2003). Moneyball. W.W. Norton & Company     

"Let\u27s Play Moneyball!": Analytics, Accountability, and the Future of Research Universities

The University of Kansas: Journals@KU

English

"Let's Play Moneyball!": Analytics, Accountability, and the Future of Research Universities

https://journals.ku.edu/merrill/article/download/7839/7437

"Let's Play Moneyball!": Analytics, Accountability, and the Future of Research Universities

Abstract

Similar works

Full text

Available Versions

Biodiversity Informatics

The University of Kansas: Journals@KU