Tenure, publishing and Tony

As Tony Hirst has blogged, his recent promotion case was unsuccessful. I'm obviously disappointed by this, for his sake, and because it was our first attempt at pushing through a digital scholarship case.

We don't know why it was unsuccessful yet (detailed feedback will follow I suspect), but today I was reading an excellent report from the Center for Higher Education Studies at Berkeley titled "Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Valuesand Needs in Seven Disciplines". There are several points it raises which I think reflect on Tony's experience and others like him. Here are some key quotes with my interpretation of each of them:

"the peerreviewed journal article is the primary mode of scholarly dissemination in the sciences and the quantitative social sciences, while the more interpretive, historical, and qualitative disciplines rely heavily on the university press monograph with a varying mix of journal articles, critical editions, and other publications. These traditions, which rely heavily on various forms of peer review, may override the perceived “opportunities” afforded by new technologies, including those falling into the Web 2.0 category."

So, even if we acknowledge there are benefits to using new technologies, the cultural traditions may prevent us taking advantage of them. This was a theme I explored in my earlier presentation on researchers and new technologies and which I go into detail in for my forthcoming book.

"In most fields, however, a stellar publication record in prestigious peer-reviewed outlets usually counts significantly more in advancement decisions. At some institutions, scholarly contributions such as data curation or multimedia websites are considered to be forms of “service” or “teaching” in a scholar’s academic portfolio, or they may receive credit when presented in a peer-reviewed publication that “discusses” the resource or data set."

Even when other outputs are recognised, the peer-reviewed journal trumps them all to the extent that they may placed in an ill-fitting separate category.

"The degree to which peer review, despite its perceived shortcomings, is considered to be an important filter of academic quality, cannot be overstated."

This stuff is deeply entrenched.

"enthusiasm for the development and adoption of technology should not be conflated with the hard reality of tenure and promotion requirements (including the needs and goals of final archival publication) in highly competitive and complex professional environments. Experiments in new genres of scholarship and dissemination are occurring in every field, but they are taking place within the context of relatively conservative value and reward systems that have the practice of peer review at their core."

This reinforces the point I made in my research talk - the environment we have created works in direct opposition to innovation and adoption of new technologies. This is a dangerous situation.

If we take Tony as an example the way he works raises some interesting questions about what we mean by research. He typically explores new technologies and data visualisation in particular. A random sampling of recent posts include:

• An analysis of twitter connections between UK politicians
• A representation of online communities who use the same hashtag
• An interrogation of the Mendeley software to show users by institution
• Sharing his own promotion case
• A presentation on ‘data driven journalism’

Each of these is intended to promote discussion, and has suggestion for implications, for example in how higher education can make effective use of data. None of the posts arise from a specific research project, and each of them is fairly small in terms of time and resource. The use of a blog though (instead of publications) allows Tony to engage in this ongoing experimentation, as it has an outlet, and it simultaneously encourages it also since discussions will arise on the blog (or in other places such as Twitter). Taken as a whole then, the blog itself represents the research process, and in this context it is difficult to say that it is not demonstrating the REF definition of research: ‘a process of investigation leading to new insights effectively shared.’

A paper on scholarship has recently been approved by the OU Senate and there is undoubtedly a desire to engage with and recognise new forms of scholarly activity. But what the report demonstrates is that cultural practices are almost inextricably bound up with publishing. And when you unpick what publishing means it leads you to some difficult considerations. This is Bellow's Law in action. When a case like Tony's comes across it simply doesn't fit well with the existing framework. This is, I guess, one of the perils with being a pioneer. Not that this is much consolation.

Comments

[Paul Lefrere]

Well said, Martin. An additional irony is that Tony's pioneering work in all areas has considerable potential to be exploited commercially (eg for data-mining and sense-making), and could be a god-send to universities that need to find new income streams to fill the gap left by next week's official pronouncements on university funding. If people like Tony leave higher education for greener pastures, those income streams will be much harder to find.

[Rebecca]

Although the OU Senate discussed and approved the definition of five separate types of scholarship, there was little consideration of what digital scholarship could mean to the university. The paper gives 26 examples of 'approved scholarship'. 'Rated blog' is one of these - with no indication as to what rated might mean in this case - whereas six examples refer to refereed/peer-reviewed journal articles.
There's also a very strong emphasis on textual presentation of results which appears to exclude video and podcasts, for example, and also nothing about the collaborative construction of multimedia representations of research and its results.

[antonella esposito]

Someone said that today - due to the current tenure system - a philosopher such as Schelling wouldn't been recruited as a professor (at 18 y-o). Too brilliant.

It's a discouraging story: if this happens at the OU, I wander if there is any hope for change in 'conventional' universities...

In these weeks I am working on an early literature review for my MRes dissertation, just focusing on scholarly practices on the move.

Faculty resistance with respect to scholarly communications’ change is also being confirmed by large scale surveys (http://www.ithaka.org/ithaka-s-r/research/faculty-surveys-2000-2009/faculty-survey-2009), which are unable to identify "a clear trajectory to change", indifferently among humanists or scientists:

“Despite several years of sustained efforts by publishers, scholarly societies, libraries, faculty members, and others to reform various aspects of the scholarly communications system, a fundamentally conservative set of faculty attitudes continues to impede systematic change” (Schonfeld and Housewright, 2010: 2)

Furthemore, the key quotes you extrapolated from the excellent report by Harley et al.'s (2010) are also relevant because they foreshadow a contrast of new practices' adoption against competitive advantage.
In fact, the investigation was undertaken in a group of prestigious US universities. With respect to the transition of scholarly communication practices, the authors comment that

“it is premature to assume that Web 2.0 platforms geared toward early public exposure of ideas or data, or open peer review, are going to spread among scholars at the most competitive institutions”. (Harley et al., 2010: 15)

Moreover, this impasse seems to include the so called “millenial” learners as future scholars (graduate students, postdoctoral scholars, or assistant professors), who “adopt the behaviors and norms of their mentors to advance their careers (Harley et al., 2010: 15; see also similar findings for UK in JISC/British Library, 2010, http://www.researchersoftomorrow.net/).

The only 'category'of potential innovators of scholarly practices is the 'middle bracket' of the 'associate teachers', who are less worrying for their own career. However, I am afraid that it can deal again with the syndrome of "Lone Ranger", translating from Bates who refers it to the isolated teacher carrying out her/his e-learning pilots.
What can the impact of these innovative scholars be on scholarly practices if they aren't able to involve the early majority of new researchers?

[Martin]

@Paul - yes, then we'll probably hire him back as a consultant...

@Rebecca - yes, they're not sure what this stuff looks like, yet when we give them a good example, it isn't recognised.

@Antonella - thanks for the response. I agree entirely - look back at the research post I link to here (which references the JISC and other studies), where I argue that the situation we have in HE is the opposite of most other industries, where the new blood are positively discouraged from any innovation. This strikes me as a dangerous situation for research to be in. As you say the only people who can innovate are those profs with tenure, and given that they've often been successful operating in the traditional manner, they are unlikely to do this, so the pool of innovators (with regard to technology and communication anyway) is made unnaturally small. Your thesis sounds interesting, be good to see where it goes, good luck with it.

[Dougclow.wordpress.com]

This is very disappointing - not just for Tony, but for the message it sends about digital scholarship at the OU, which is in opposition to the other messages being sent.

I think Tony posting his case and results (and presumably feedback when he gets it) may begin to give us an idea about the application of Bellow's Law to the promotion process. And good thing too.

[Joel Greenberg]

The award winning S216 Environmental Science DVD represents 22 hours of student learning and has been used by thousands of OU students. It is a fabulous resource and very relevant in 2010. The main academic behind it told me that he had not bothered to put it on his CV as he would acrue no professional credit for it. He got promoted by becoming Head of Department. So this is not a new problem - just a bit depressing that it still exists in HE.

[Tony Hirst]

Here are my immediate comments on the Senate paper, which i passed on to a couple of members of Senate who had requested comments... I'll be posting them again on a commentable version of the Senate doc (I hope) as soon as it becomes public.

Tony Hirst - Response to scholarship paper

In "Sorkin vs. Zuckerberg", a review of "The Social Network"in The New Republic (Oct 1st 2010
[ http://www.tnr.com/article/books-and-arts/78081/sorkin-zuckerberg-the-social-network ]http://www.tnr.com/article/books-and-arts/78081/sorkin-zuckerberg-the-social-network ), Lawrence Lessig observed:

"But the most frustrating bit of The Social Network is not its obliviousness to the silliness of modern American law. It is its failure to even mention the real magic behind the Facebook story. In interviews given after making the film, Sorkin boasts about his ignorance of the Internet. That ignorance shows. This is like a film about the atomic bomb which never even introduces the idea that an explosion produced through atomic fission is importantly different from an explosion produced by dynamite. Instead, we’re just shown a big explosion ($25 billion in market capitalization—that’s a lot of dynamite!) and expected to grok (the word us geek-wannabes use to show you we know of what we speak) the world of difference this innovation in bombs entails.

"What is important in Zuckerberg’s story is not that he’s a boy genius. He plainly is, but many are. It’s not that he’s a socially clumsy (relative to the Harvard elite) boy genius. Every one of them is. And it’s not that he invented an amazing product through hard work and insight that millions love. The history of American entrepreneurism is just that history, told with different technologies at different times and places.

"Instead, what’s important here is that Zuckerberg’s genius could be embraced by half-a-billion people within six years of its first being launched, without (and here is the critical bit) asking permission of anyone. The real story is not the invention. It is the platform that makes the invention sing. Zuckerberg didn’t invent that platform. He was a hacker (a term of praise) who built for it. And as much as Zuckerberg deserves endless respect from every decent soul for his success, the real hero in this story doesn’t even get a credit. It’s something Sorkin doesn’t even notice"

In a similar vein, whilst the Scholarship paper to Senate is to be welcomed, it arguably misses the point about the new forms of emerging scholarship.

From my perspective, "the new scholarship" is predicated on ideas relating to openness: discoverability through web search on the one hand, and information flow via publish-subscribe networks (aka asymmetric follow networks), particularly on the web, on the other.

Looking at the document, it is not clear what the defining charateristics of "scholarship" are, although the authors appear to have an unstated assumed definition in mind (exactly what that is, I'm not sure...)

The Senate paper requests the acceptance of a statement (paragraph 5) about the distinctiveness of scholarship at the OU. This paragraph suggests that the OU's distinctiveness is bound up with scholarship relating to open and distance education, which it surely is, though to my mind the practice of this scholarship might also be a characteristic feature if pursued at least in part according to the "new scholarship" traits suggested above.

In [ [ http://blog.ouseful.info/2010/09/14/so-wtf-is-data-scholarship/ ]http://blog.ouseful.info/2010/09/14/so-wtf-is-data-scholarship/ ] I quoted a short unit on Connexions (What Is Digital Scholarship? [ [ http://cnx.org/content/m14163/latest/ ]http://cnx.org/content/m14163/latest/ ]) by the American Council of Learned Societies Commission on Cyberinfrastructure for the Humanities & Social Sciences which suggests:

"In recent practice, “digital scholarship” has meant several related things:

"- Building a digital collection of information for further study and analysis
"- Creating appropriate tools for collection-building
"- Creating appropriate tools for the analysis and study of collections
"- Using digital collections and analytical tools to generate new intellectual products
"- Creating authoring tools for these new intellectual products, either in traditional forms or in digital form"

The piece goes on:

"It may seem odd to some that creating collections and the tools to use them should be counted as scholarship, but humanities and social science research has always required collections of appropriate information, and throughout history, scholars have often been the ones to assemble those collections, as part of their scholarship. Moreover, scholars have been building tools since the first index, the first concordance, the first scholarly edition. So, while it is reasonable to regard (d) as the core meaning and ultimate objective of “digital scholarship,” it is also important to recognize that in the early digital era, leadership may well consist of collection-building or tool-building. In addition, tool-building is dependent on the existence of collections, and both collections and tools get better and more general as there is more use of digital information. If we hope to see new intellectual products, we should give high priority to building tools and collections. Finally, it is worth noting that although (a), (b), (c), and (e) require a great deal of cooperation, it is still imaginable that (d) can be the work of a single individual."

It is my contention that the *approach* we take to scholarship (scholarship in the sense of creating through doing a new practice of scholarship) may be as important a contribution to the evolution of scholarship as the "unique" focus - distance and open education. Indeed, the process may well lead in to new forms of supporting and delivering open and distance education, as well as contributing to practice in the arena of open science (and other forms of open research).

In identifying several "proposed scholarship types" (paras 6-10), the paper identifies particular "scholarship domains", none of which include the practice of new forms of scholarship (which I would argue is scholarship, as the American Council of Learned Societies commission did). Just as Lessig commented that the producers of The Social Network missed the point by using an old model to try to describe a new model, not least in the area of "metrics", I think the authors of the paper are making a similar mistake in their treatment of what defines scholarship in a networked and digitally mediated age. For example, there seems to be a focus on metrics around formal entries into the "recorded body of human knowledge", without recognition of how web-based access to information has significantly changed what it means to make something citeable, quotable, referenceable, (re)desicoverable and recoverable/physically accessible. There is also an assumption that formal academic publications lead to sensible "impact" related metrics, although it is not clear what it meeans to make an "impact" in real terms. In my mind, impact is related to the extent to which someone has influenced the behaviour of someone else, particularly in the area I focus on - the development of digital tools for understanding and exploiting web based digital communication systems.

For pretty much everything I post on the web, I can tell you how many times it was viewed, often with an indication of the number of unique individuals who have seen the resource, identify other reseources that link to the resource, and make it available for free and in an open way via a unique URI that dereferences to a unique location on the internet. The reources can also be associated with commentary and other weak signals or "votes" that the resource has "impacted" someone in some way.

A significant contribution the OU could make would be to the evolution of metrics relating to reputation and quality of scholarly activity, models that may be extensible to rating scientific research as well as the construction, validation and verification of (informal) educational awards and qualifications.

When discussing "Further Issues for Comment", I would like to have seen some reference to the openness in the practice of shcolarly activity. The potential benefits for this are themselves uncertain, though they may appear from the most unlikely directions and in the most unexpected ways. (For example, see this review of a brief, informal and ad hoc mini-project that moved things on in open science, albeit only slightly!, over the course of a weekend: A little bit of federated Open Notebook Science, C Neylon [ http://cameronneylon.net/blog/a-little-bit-of-federated-open-notebook-science/ ]

On the question of the evaluation of scholarship (para 12), the reference to externality (first bullet point) suggests that "all forms of scholrship should have an existence externally and be capable of recognition in a wider domain". To me, that is an argument for openness (where by open-ness I mean the publication of content in an open and discoverable way, ideally unencumbered by access or license obstacles; it should also mean the freedom to "remix"). It is unclear from the paper what the OU values as evidence of "externality", and I would like to see this clarified. (I would also argue that publication in the University of Poppleton Journal of Iffy Reports does not really count.... Even the impact of publication in a "major" journal is moot if no-one who reads that journal is in a position to be influenced by the publication...)

THe second bullet point of para 12 on outcomes suggests that they should be "subject to judgements of excellence by peers". Again, I would argue this is in accord with the principle of open publication even of informal works and work in progress. It is not clear how the OU would ascertain the extent to which these judgements might be made on the one hand, and recognised on the other.

That the outcomes of scholarship should be "capable of use and elaboration by others" is a significant reason  why I blog in an open and incomplete fashion, although to my knowledge this is not recognised by the OU; (if it is, I would like to know how)

And sa far as scholarship outcomes having an "impact", I would like to know what definition of impact is being used. As mentioned above, for me impact is closely tied to the ability to influence. In my own work, this might be measured in terms of "conversions" from people who read my blog or follow me on Twitter to their inviting me to talk at an event they are running. As to whether the contributions need to be academic, I would like to know what is meant by academic, and whether this precludes wider impact, such as the dissemination and adoption oor uptake of ideas in the wider academic, academic related area as well as business, local government, news media/jourmnalists, informal learning communities, and so on, not only with the OU and UK HE, but also internationally. I would also like to know how it is intended to measure the "significant difference" that scholarship might make to the academic contribution of the OU particulalry in the new world of ubiquitous online digital communications and access to knowledge.

On the role of funcding (para 16), the emphasis is on drawing in large project grants. For my own part, I attend a lot of events (by invitation) and there seems to be an opportunity for finer granularity metrics. The ladder appears to go:
- fee waiver
- travel expenses
- accommodation
- speaker fee
Whilst many conference invitations I receive are probably at best described as "academic related", and not subject to my submission of peer reviewed papers, they are the result of people choosing to invite me (often to events others have paid to attend) to hear what I have to say, rather than read out something I have already said.

The Appendix to the paper expands on several examples of the five scholarship types. Again, I bring to mind Lessig's review of "The Social Network", for it somes to me that the examples are very much influenced by the way things have already been done in a publishing environment that was more contrained that the online publishing and communication(s) environment available to us today.

For me, publications should be discoverable (often in a timely way), referenceable and dereferenceable/available/recoverable (e.g. in the sense of persistently linkable to), and able to influence others (which includes notions of them building on the (partial) works).

Recapping a point made above, the examples seem to be more about viewing impact in terms of things that have already been said in a particular place with the ability to influence others through what you might have to say to them in some sort of ongoing, invited or pre-emptively started conversation with them.

A lot of the examples given reiterate the primacy of formally published, "completed" works. By contrast, there is only a single mention to "ongoing"/persistent publications or communication channels such as well-respected blogs. The time taken to develop and persist a blog with good reputation should not be underestimated, nor should the potential audience size in at least two respects: 1) through publish-subscribe models, blogs and other syndicated publication channels have an always available route to an audience. In the OU, myself and Martin Weller both have blog subscriber numbers of the order of 2000 subscriptions for example. Everything I publish on OUseful.info will be seen by approx 200 people within 24 hours (the "reach" of the blog, cf. its subscriber numbers), and maybe actually read by 5-10% of them - it's hard to know for sure;-) In addition, content on a high ranking website can pull in pre-qualified visitors i.e. visitors likely to be interested in the topic at hand, for example because they were searching for it via a search engine, or clicked through to it from a link on a  page that had already grabbed their attention.

It also seems to me that there is little emphasis on notions of ongoing engagement with communities and domains of interest, or the effort required to gain influence in those domains (eg through search engine rankings, or high connectivity (or significant hub and/or authority values in a HITS sense) in online networks. That is, the paper empahsises traditional models of reputation which, even if they are still relevant in academia today, are unlikely to be meaningful (eg in sense of being able or likely to influence others) in the wider community. (The OU used to be very supportive of community engagement; what I would like to do is see us really innovating and driving the extent to which we can influence and contribute to ever more communities, both formal and ad hoc, even ephemeral communities) through our online reputation and presence.

[hapdaniel]

A university that does not have at its core the concept of lecturers and students attending educational sessions in person? That will never catch on.