The Keynote Equivalent?

I've been thinking a lot about recognition of digital output in formal systems, in short, can you get promotion for doing all this online stuff? One of the great things about being online is that as soon as you start to openly think these things through, people start to point you at stuff (mainly Tony). It seems a lot of people have been doing the same (and getting further than me).

For example Eric Schnell says that

The New Media Department and The University of Maine amended their promotion and tenure guidelines (all the way) back in 2007 with redefined criteria in the form of alternative recognition measures. Their documents identify nine alternatives to the standard 'article in a peer-review journal' model

He then lists the nine alternatives: Invited/edited publications; Live conferences, Citations; Download/visitor stats; Impact in online discussions; Impact in the real world; Net-native recognition metrics; and Reference letters.

I think the inclusion of download/visitor stats and net-native recognition metrics is significant here. Similarly, Geoffrey Rockwell has a wiki for a project on evaluating digital work in the humanities. The types of digital output they list as relevant are:
Online peer reviewed publication; Scholarly Electronic Editions; Specifications; Research Tools; Hypermedia/Hyperfiction; Instructional Technology/CALL; and Research blogs.
They also provide a list of questions for those evaluating such work (I'm not sure I agree with the list, it seems focused on a specific product, rather than, say overall significance to a community).

Gerry McKiernan blogs about a study which examined 39 different impact factors for science. The authors of the study (Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6)) state:

To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact.

After reviewing a number of impact factors, including social network measures, they conclude that:

the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

So what we currently have is too simplistic in an online world.

Which brings me on to the idea of equivalence. I wonder if we could follow Eric Schnell's line of creating equivalents for existing, and well understood, criteria. Let's take the Keynote speech as an example. Being invited, and giving, keynotes is often listed as one of the marks of esteem if you are seeking promotion. The reasons are twofold I believe:

1. Reputation - it demonstrates that you have gained significant standing in your field to be asked regularly to give a keynote talk at a conference.
2. Impact - if you are giving the keynote then everyone at the conference hears it, and you can therefore claim a significant impact in your subject.

The important element then is not the keynote itself, but what it signifies. If we start with this basis, then we can think of online equivalents. For example, if I give a talk and then put up a slidecast of that presentation, a certain number of views might equate to impact (how many people would hear a live presentation?). If the presentation is retweeted, linked to, embedded, then this might give an indication of reputation.

I don't think we can provide simple translations (500 views + 5 embeds = 1 keynote), which means there needs to be both a case made by the individual, and an interpretation by the panel, but by focusing on the existing criteria and considering what it is they are meant to demonstrate fundamentally, we can then talk about online equivalents.

Comments

[Mark Smithers]

I wonder whether 'reputation' web sites may emerge that give an independent indication of a person's online reputation. Already sites like twitter.grader.com will provide a ranking. Maybe in future someone like Google will provide this functionality.

[Nicolas Holzapfel]

I don't think anyone could disagree with that viewpoint and I certainly hope academic institutions see it that way too; not just for the sake of giving fair recognition to internet-savvy academics but also because it would encourage more academics to share their expertise online, making the world of academia more accessible and more influential.

[Steve Ransom]

I think you give us a number of important issues to consider here. Thanks.
The only area that I would be more cautious about is that of formulas and mere numbers of viewers as evaluative criteria. A presentation posted on a service like SlideShare is not really equivalent to a keynote at all. With a keynote, the speaker has been intentionally selected based on a proven track record/reputation of some kind. Attendees are attending the conference based upon their discipline(s) of interest and position. The audience is neither random nor global in the open sense. WIth an online presentation, it is possible that much of the audience is clicking out of curiosity, but there is no way to gauge whether the presentation was viewed in its entirety and viewed by people who had a serious interest (hence not viewed in its entirety). The silliest or crudest YouTube videos may have hundreds of thousands of views if gone viral, yet perhaps that says more about the viewers than it does about the individuals posting that sort of content. All I am thinking about here is that the posting of, linking to, and retweeting of content does not necessarily indicate excellence or rigor when the audience is global and potentially unlimited (and I know you did not state this). The same holds true with peer-reviewed publications. Although they are problematic when hidden behind subscriptions and membership that essentially keep them from wide readership and dissemination, there is something to be said for the quality/rigor that they require/ensure.

That being said, I think you are quite correct that we need to seriously reevaluate how we determine merit and recognition in a highly connected and digital age. The traditional notion of scholarship really needs to be carefully reworked.

[Martin]

@Mark - yes, something that gives a metric of 'scholarly activity' would be useful, but it'd come with lots of caveats.
@Steve - yes, see some of my previous posts on metrics. We should see them as part of a package, not the replacement for peer review. I'd disagree with you on a keynote being superior though - often the keynote is selected on the basis of being a mate of someone who is organising the conference and the audience doesn't sign up for the keynote, they have no choice in the matter. So you could argue that views, links, tweets to a slideshare are more significant because they exercise an element of choice. Obviously they are not the _same_ thing but by looking at the underlying reasons for why we value something such as the keynote we can then extrapolate back up to the online world and find activities that meet those criteria also.