The Ed Techie

The Keynote Equivalent?

I've been thinking a lot about recognition of digital output in formal systems, in short, can you get promotion for doing all this online stuff? One of the great things about being online is that as soon as you start to openly think these things through, people start to point you at stuff (mainly Tony). It seems a lot of people have been doing the same (and getting further than me).

For example Eric Schnell says that

The New Media Department and The University of Maine amended their promotion and tenure guidelines (all the way) back in 2007 with redefined criteria in the form of alternative recognition measures. Their documents identify nine alternatives to the standard 'article in a peer-review journal' model

He then lists the nine alternatives: Invited/edited publications; Live conferences, Citations; Download/visitor stats; Impact in online discussions; Impact in the real world; Net-native recognition metrics; and Reference letters.

I think the inclusion of download/visitor stats and net-native recognition metrics is significant here. Similarly, Geoffrey Rockwell has a wiki for a project on evaluating digital work in the humanities. The types of digital output they list as relevant are:
Online peer reviewed publication; Scholarly Electronic Editions; Specifications; Research Tools; Hypermedia/Hyperfiction; Instructional Technology/CALL; and Research blogs.
They also provide a list of questions for those evaluating such work (I'm not sure I agree with the list, it seems focused on a specific product, rather than, say overall significance to a community).

Gerry McKiernan blogs about a study which examined 39 different impact factors for science. The authors of the study (Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6)) state:

To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact.

After reviewing a number of impact factors, including social network measures, they conclude that:

the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

So what we currently have is too simplistic in an online world.

Which brings me on to the idea of equivalence. I wonder if we could follow Eric Schnell's line of creating equivalents for existing, and well understood, criteria. Let's take the Keynote speech as an example. Being invited, and giving, keynotes is often listed as one of the marks of esteem if you are seeking promotion. The reasons are twofold I believe:

1. Reputation - it demonstrates that you have gained significant standing in your field to be asked regularly to give a keynote talk at a conference.
2. Impact - if you are giving the keynote then everyone at the conference hears it, and you can therefore claim a significant impact in your subject.

The important element then is not the keynote itself, but what it signifies. If we start with this basis, then we can think of online equivalents. For example, if I give a talk and then put up a slidecast of that presentation, a certain number of views might equate to impact (how many people would hear a live presentation?). If the presentation is retweeted, linked to, embedded, then this might give an indication of reputation.

I don't think we can provide simple translations (500 views + 5 embeds = 1 keynote), which means there needs to be both a case made by the individual, and an interpretation by the panel, but by focusing on the existing criteria and considering what it is they are meant to demonstrate fundamentally, we can then talk about online equivalents.

Arguments for social media engagement

In a comment on a previous post my colleague Chris Jones argued that establishing an online identity wasn't suitable for everyone:

What of the bulk of other academics interested in research and publication? They may move to open access routes for publication but will they want to spend their time in developing a public persona? I am not sure they or I will. I haven't yet developed a blog, though I follow others, including yours. I am not sure that digital scholarship covers all or even the main aspects of intellectual endeavour. Sometimes it is a lone academic quarrying away obscurely on a narrow point that makes a difference. Some of the dynamics of intellectual life require a position outside of the public gaze.

I've pulled Chris' quote out because it is worth discussing in detail I think. Chris is not someone who objects just for the sake of, or from a resistance to technology, but from a more considered standpoint. His argument is that he has a very well established, and effective, way of engaging in scholarly discourse and activity, and in order to develop the type of online identity and reputation I was suggesting he would need to give up doing something. I think it is easy for those of us who spend a significant amount of time in the online, participatory media world to think its attractions are obvious, and undeniable. But Chris raises some valid objections here that bear consideration. I think there are four responses one might make, which range in levels of evangelism.

The argument of recognition - the softest response is to say that I am not proposing that everyone should, or needs, to develop an online identity. What I am arguing is that there is a monopoly on the types of activity and outputs that are currently recognised in the scholarly system, and that we should be able to expand upon this to include the more digital scholarship work, alongside the normal work.

The argument of simplicity - this takes the proposal a touch further and says that not everyone needs to spend a great deal of time and effort in becoming a 'podstar' or blogging celebrity, but there are sufficient number of easy to use tools available such that everyone can easily create an online presence. This doesn't really require you to give up any other activity, it is a byproduct of what you do anyway. Creating a presence using Tumblr for example is a quick way to have an online identity.

The argument of benefit - in his autobiography Martin Amis asks of the humourless 'how do they raise children? How does it get done without a sense of humour?' I sometimes feel the same about social network/web 2.0/participatory media (call them what you like) and being an academic. How does it get done without using these tools every day? I can't imagine finding resources as readily without the links and filtering my network provides, or the analysis they provide, or the feedback I get from them, or the responses to help they give me. I set out a few benefits in the talk on online identity. For me this is one of the strongest arguments - it makes things better.

The argument of imperative - this is the most forceful argument, and an extension of the argument of benefit. Not only does online activity provide you with benefits as an academic, but it is fundamentally key to what you should be doing. The argument here goes something like 'how can you be relevant to students, operate as a scholar if you're not engaged with the most powerful research/communication medium ever?' I'm not sure I'd go this far, because you can use various online services effectively within your research without it necessitating establishing an online identity, but, if one thinks this social media stuff has any relevance, then I maintain that the only way to really understand it is to do it.

Have I missed any, and are any of these really compelling enough? And do they apply across all domains eg. could I argue the same for, Art History, say?

We're all studio bands now

Marieke, over at RemoteWorker, asked me to do a guest blog post around this issue of being a remote worker and establishing a profile at your work place, which you can read in full on her blog.

I mixed up several issues, the first being the effectiveness of being an online worker, where I made a (tongue-in-cheek) comparison with the power-breakfasters of yore:

In the 1980s in the heyday of the Yuppie, there was talk of the ‘power breakfast’ when Masters of the Universe would meet at breakfast to do business to show how tough they were. This is nothing – nowadays I get up and reply to some tweets, put a comment on someone’s blog, respond to comments on my own blog that have come in overnight and maybe even produce a quick publication in the form of a blog post. I have done global networking before I’ve changed out of my pyjamas. And I’m pretty normal in this.

I also wanted to explore the notion that our online presence is somehow an improvement on our real identity, ie the online me is superior to the 'real' (or at least the real-time) me in many senses:

I know that some people now don’t bother attending my ‘live’ presentations (or are unconcerned if they miss them) because they know I will put them up later. And more than that, I feel that I have time to correct the presentations (although they still remain a fine example of amateur hour) so that the recorded version may well be superior to the live one. I think this may pervade across all of my online identity: my blog has more interesting things to say than I do, and my twitter stream is wittier than I am. I have become like one of those bands of whom people bemoan ‘they’re not as good live’. And as we perfect our online skills, maybe this is the fate that will befall us all. In which case, don’t ask me to give a keynote, I’ll just record you one instead.

I was just exploring some ideas here, so pop over to Marieke's blog to give it a read.

Blogging and academic identity

Blogging and academic identity

View more OpenOffice presentations from mweller.

I gave a talk recently about blogging (twittering, etc) and how we create online identities for ourselves. The slidecast is above. I wanted to explore some ideas, the main ones being:

• It is about identity, not technology X
• Your identity will be constituted from several different tools/services
• Your configuration and emphasis of those tools is part of what makes the identity (as well as what you put in them)
• An online identity is becoming default for academics now
• All this is driven by really easy and diverse ways of sharing
• There are numerous benefits to you as an academic

I concluded with two propositions, which you might like to disagree with:

1. Soon, your online identity will be your academic identity
2. There is an online identity of some form out there for everyone

I hope you find the presentation useful - I have to say I've come to loathe the sound of my own voice and my incoherent rambling so much, I think I may desist from doing slidecasts. Whatever my online identity is, it's not as a raconteur.

Some more thoughts on metrics

Following on from my last post on the possible use of metrics to measure online digital reputation, here are some more thoughts.

Andy Powell took me to task in the comments, arguing eloquently that the metric is so obviously flawed that it is positively harmful. I've been pondering this, and here are some reflections, both for and against. As the methodology is the property of the consultants, I don't know exactly how the algorithm works, so am guessing from the results.

I can see at least three possible problems with the methodology:

1) A deadly attractor - given the search term used ('distance learning'), the OU unsurprisingly dominates the space. I am guessing that close association with the OU then boosts other sites. So, for example, if I had this blog, but was a Professor at, the University of Glamorgan, say, then my ranking wouldn't be as high.
2) An echo chamber effect - a lot of the sites tend to reference each other (me, Tony, Brian, Grainne, for example). You then get a positive reinforcement effect. They claim to have adjusted for this, but I'm not convinced.
3) Bias in initial setup - the blurb seems to suggest that they find the influential sites by searching and analysis, but there must be some priming. The list is heavily UK-centric for example. This may be a result of the term used (see below), but until we know how each search is initiated I think we have to suspect some initial influence depending on the starting parameters.

Search term - as I mentioned in the original post, the search term is significant. 'Distance learning' is very niche - it's not 'online learning' or 'elearning'. Had it been then the list would have been very different. This (plus the things above) may account for some notable absences from the list - why no Stephen Downes for instance, who we would surely think of as the hub par excellence?

Results may be revealing - Andy argues that the presence of Brian Kelly demonstrates that the list is nonsense as Brian doesn't really blog about distance learning. But I think it may be telling us something interesting. It can't just be random, and why is Brian higher than Grainne, for example? It could be telling us that people who write about distance learning tend to reference Brian, even if he doesn't write about it directly. In this sense Brian does have 'influence'.  It could also show us that people who are writing about distance learning online are writing more about IT than pedagogy. This in itself is revealing isn't it?

It's not just popularity - popularity is a factor, and in some respects is a proxy for influence. But we all know that popularity can be gained for all sorts of unacademic things. So popularity is a factor, but only within a given context - it's not about the overall number of subscribers say, but the number of links relating to a given term, and its semantic cousins. So, if I was the expert in modern interpretation of Macbeth say, then I would expect to have the leading amount of links relating to this topic, even though the number of links would be small overall, because it is a niche subject. I wouldn't be a popular blogger relative to every other subject, but within this very specialised subject then I would be 'popular' relative to other subject sites.

Gaming - any algorithm is subject to gaming, as is any system. Exams and the REF are subject to gaming, but the key is to put in enough checks to make gaming difficult, detectable, and ultimately not worth the effort. Any such algorithm would need to be sophisticated enough to avoid obvious gaming.

A metric would only be a partial solution - I think we'd probably always want an element of peer-review in any analysis, and wouldn't rely solely on an automatic measure. But rather we could view an algorithm as part of a portfolio of evidence an individual might present.

We've got to start somewhere - my take on this is that the output may have problems, but it's a start. We could potentially develop a system focused on higher education, which is more nuanced and sophisticated than this. By analysing existing methodologies and determining problems with them (such as the three I've listed above) we could develop a better approach. I hold out hope that we can get interesting results from data analysis that reveals something about online scholarly activity.

Connections versus Outputs

<image No Trumpets Please! by Mr Jaded http://www.flickr.com/photos/jaded/1907907/>

Warning! This post may contain own trumpet blowing!

Laura Dewis sent me a report the lovely OU Online Services people had prepared in collaboration with consultants (MarketSentinel). They were interested in examining the broader influence of various web sites and looking at sentiment mining. The idea from an official communications perspective being you can see how well regarded your institution is in different sectors, and maybe influence that perception.

But from my perspective the analysis they performed could be tweaked to provide a measure of an individual's influence or prominence in the online community of their particular topic. As you will know, I am interested in the concept of 'digital scholarship' and in trying to get the type of online activity many of us engage in recognised as scholarly work.

The problem has been a mismatch between what has traditionally been measured to indicate academic standing in a particular field and the type of activity that takes place online. In short, the formal systems such as the RAE or the REF in the UK, are focused on outputs. They make a nod towards impact, or reputation, but it's really outputs that they want - and we largely know what these look like in this world: Refereed  papers in respected journals; Keynote speeches; Gaining large research bids; etc. These don't map well onto the digital world - is 1000 blog posts good? What about 1000 subscribers? Is having a large twitter network a sign or standing? All too easily such measures fall down. So I have been looking for something more robust which might act as a metric for measuring an individual's reputation in their subject area.

The report the Comms team provided me gets some way towards this. They chose the subject area of 'distance learning' really as a test to see how it worked - one would expect the Open University to come out well in this. Here is the blurb from the company on how they determine influence:

"a stakeholder of a topic is “an entity (individual or organisation) who is sufficiently  referenced in the context of the topic” .
When performing a MarketInfluence study our computer systems initially collect any documents on the Internet (web pages, Word, pdf or PowerPoint documents) which match a defined search phrase.
These documents (often hundreds of thousands) are then analysed. The analysis is especially focused on identifying who references whom.
Based on these references it is possible to calculate influence."

Influencers

They then apply a number of filters to prevent self-citation. From this they drew up the following list of top 100 influencers in 'distance learning':

I come out 4th, Brian Kelly 6th and Tony 10th. I'm sure that Tony and I both gain some credit benefit from being associated with the OU, which drags up our distance learning status, but even so the mix of individuals in with large professional output such as the Guardian, JISC and BBC is interesting - the space has become democratised to an extent.

Betweeners
They then have a measure of friends in common, what they term 'betweenness':

"Stakeholders with high Betweenness are “stations” where information (on the issue in focus) is passed via in order to reach the constituency of said stakeholder. "

Here is the table for those with high betweenness:

So Brian, Tony, Alan, Grainne and I are all good conduits for information (in this narrow domain). Individuals seem to work better as betweeners than organisations it seems.

Hubness
They reference Gladwell's Tipping Point and his notion of 'connectors', to suggest some sites/people have a high level of 'hubness': "Hubness is the characteristic of disproportionately linking to those who are authoritative on a given topic."

Again, people tend to be better hubs than organisations. Oh, and from now on you MUST address me as 'your hubness'.

It is still problematic, and could be gamed. I don't know enough about the algorithms used to assess this. One would also need to be careful about the search term used - 'distance learning' is quite a niche, UK term I think (had it been 'educational technology' we would have had a very different list). But, all this isn't (just) about ego massage - it strikes me that if we could develop such an algorithm so that we could easily enter any subject domain, this would provide a useful tool for measuring an individual's online influence/reputation/status etc in their field. This would then provide evidence for justifying this type of work and in seeking promotion. It could offer us the Alt-REF I was after. At the moment this work belongs to the consultants, and we would want to tweak it for academic use, but it does suggest that such an analysis is possible.  A JISC project to develop the service for all academics?

What this gets at is that online activity is different - it is less focused around outputs and more around overall activity and reputation. And it does begin to back up what I've always felt - that this stuff isn't just peripheral, playing around, but increasingly is significant to individuals and organisations. I wouldn't want to try, but one could think about it in monetary terms - how much is this influence worth if you were to try and buy it (through advertising or other means?).

Bless my cotton socks I'm in the news

(yes, I have been waiting a long time to use that title)

A couple of weeks ago, a journalist from The Guardian contacted me to talk about how a tweet from Stephen Fry had sent over 10,000 visitors to the OU's Devolve Me site. The angle was initially the power of new media, but quickly morphed into educational use of new twitter. I stressed the importance of one's personal network and new forms of digital scholarly activity. I didn't really have a single message though, and journalists like to have something to hang a report around, and it seemed clear that she wanted to go beyond the Fry story.

So she spoke to Brian Kelly, and in the meantime the CLex report of the use of new technologies in higher education came out, and the piece became about more general use of web 2.0 and higher education. It was published today in the Education Guardian. After all the hype and scare stories around twitter, it's a good, well balanced piece.

It was interesting to see the piece develop. I wonder though if there isn't a little hypocrite dwelling at the bottom of my heart: I praise new media, talk about the death of traditional news, and then get a little flush of excitement when a national newspaper prints my name. Is therapy required?

A literacy, like a ghost, must be spoken to a little

I did a presentation for an online session on digital literacy hosted by Josie Fraser last Friday. Here is my slidecast:

Digital literacy for educators

View more presentations from mweller.

It was well attended with some of my favourite online people and we had some good discussion.

A few things occurred to me:

i) We forget the literacies we wear in life  - it's a broad term, and we all come to it with different sets of skills and concerns. If you're interested in schools, then digital literacy has a whole different set of connotations and issues than for higher education. It is debatable even whether lumping all of these groups together is useful.

ii) It is the best of terms, it is the worst of terms - we debated back and forth about the term, and whether we should even debate the term, whether it should be literacy or literacies, etc. In general there was consensus that a) we shouldn't get bogged down too much in terminology, b) as the term is out there we may as well stick with it, c) there are issues with it.

iii) There is a literacy of the head and a literacy of the heart - conventional literacy is reasonably well defined, and while it is not an either/or, we can at least set some definite targets. Should digital literacy be specific or aimed at more general skills? For example, is it a checklist of technology mastery eg have captured and uploaded video to video sharing site, or more general skills eg using appropriate technology to communicate effectively.

I have some reservations (see my post about being a spoilsport) that whenever we cast our bespectacled academic eye over such a subject we over-analyse it. The result is then something which can be implemented but has had much of the initial spirit that made it attractive removed from it, in the process of pinning down exactly what we mean. This is a dilemma we will need to work around carefully. Don't get me wrong, it may seem that I don't carry in my countenance a letter of recommendation, I really think it is an important debate and task. Just want to make sure it's not a tedious checklist and instead is a joyous explosion of creativity (and flowers no doubt).

(Apologies for extended, laboured Dickens theme, no real message from it, just playing).

Technorati - grounds for divorce

	Elbow - Grounds for Divorce
	Adblock
	Found at skreemr.com

(play song while reading)

Technorati has always had it's problems, but like in any relationship, you can overlook them for a while. However, they've been adding up lately, and I think we've reached the stage of irreconcilable differences. Here are the grounds for divorce:

1. It's broken - speaking just as this user, I know it doesn't work reliably. Exhibit A) - my authority hasn't changed for 20 days and even if I ping it manually, it still says 20 days ago. Exhibit B) - there are several people who I know have linked to me that don't show up in Technorati. As it works by calculating the linked references to your blog, this renders it pretty hopeless. Exhibit C) - my authority when it was working seemed to fluctuate randomly. There was a two months period where it dropped from around 160 to the current 79.
2. It is pre-Twitter - Technorati works on references from other blogs. The problem is that a lot of the referencing has moved on to Twitter (and Friendfeed, etc) now. A lot of that quick linking people used to do in short blog posts they now do via microblogging. And Technorati isn't picking these up (is this because of the use of shortened urls, or just that it doesn't look at Twitter accounts?). It's therefore operating with only half the information.
3. It lacks granularity - there isn't even an education channel. By lumping us all in together a small, individual blog is 'competing' with output from large companies with teams of writers. Take a look at the Technorati top 100 and you'll see that they are mostly either online magazines or professional bloggers. This is fine, but it needs a layer of categorisation over it to become meaningful.
4. It measures the wrong thing - this isn't really Technorati's fault, more an issue of the blogosphere. I would like to do, or find, research that confirms this, but my hunch is that what people link to is different, or at least not necessarily the same as, what they find interesting and valuable. People often link to funny stuff, or technical tips. Some of the posts that I have gained the most links to have been short ones that say 'I found technology Y, it looks interesting.' Others might link to this and say 'I found this (via Martin)...'. But your 2000 word discussion on Wittgenstein gets less linklove. However, it may still be useful and relevant to people. This is particularly problematic for higher education where you want to promote the Wittgenstein skits and not the tech trainspotting.
5. It's subject to gaming - all too easily really. As it measures link references the best thing to do is to engage in a real ding-dong fight with someone, preferably someone with a higher authority than you. This gets good links.

Now, you could say, 'who cares?'. Tony often chastises me for mentioning Technorati as he thinks it's all about ego (before pointing out that his rating is higher than mine ;)). But my interest has always been about whether such bottom-up metrics have a place in the world of the digital scholar, and Technorati has been the exemplar for these. If we have to consign Technorati to the tech bin then the question arises: does this mean any such approach is fundamentally flawed or do we need better metrics?

Doug Clow (who is rapidly becoming a digital scholarship guru) gave a talk recently where he concluded that we can really only rely on peer review. I don't fully agree - I think as scholars peer review will be the main component when assessing a colleague's contribution (say, for promotion), but given all the rich data we have online, surely there must be some reasonably reliable use we can make of it to at least add to the mix. I think PostRank, for example, does a better job of measuring your impact, although some of the same criticisms still apply.

So, I won't give up on metrics completely, but me and Technorati - we're through.

Year of Future Learning - an experiment

I want to try something out, so would like people's views on it. My idea is to try and get some focused activity going around the concept of the learner of the future. Think of it as either a distributed research process or adding a layer of intentionality over some of what we do in the blogosphere.

I have this feeling that between us there are often very good conversations, excellent resources, great ideas, etc but we don't always take the next step in synthesising or collecting these. 

Here is what I have in mind: A loosely coupled, distributed research 'project'. We declare 09 the 'year of future learning' (#YOFL or #FLY - take your pick). I accept that 'the year of X' is a bit naff, and comes with all sorts of anti-arguments, (what about next year?) but as a means of focusing our energies, providing opportunities for review, and giving a broad enough umbrella it provides a useful framework.

This is a means of aggregating existing resources and events by tagging them - in the Weinberger sense, we may as well put our leaves on as many branches as possible, so for instance there will be sessions at Northern Voice and Alt-C which could be made part of the YOFL simply by adding the tag.

But I see it as much more than just another meme or tag, I also want to use it as a means of instigating some new open, distributed activities. I have in mind the following:

• A set of interesting questions generated by the community
• Connected blog posts around particular topics - blog-carnivals Tony has pointed me at
• An open, special edition journal
• Combined research results
• A workshop format that anyone can run, and share results via a wiki
• 'Live' events eg George Siemens' Elluminate sessions
• 'Flash debates' in twitter around a specific topic, which are then summarised
• Face to face sessions at various conferences
• A wiki/blog/site collecting all of this material together
• An end of year review

 I want to avoid any perceptions of elitism  so the idea is that this is open to anyone. I've emailed a few blog chums to see if they would be interested in doing some of the above, to give it some traction, but essentially it's open. If there is a topic of burning interest to you, run an activity and it'll be pulled in.

In addition, my colleague Grainne Conole also has lots of links into various communities, so we will be running a number of face to face activities also. This involves 'talking to people' - not sure what that is, probably some cool web 2.0 service.

The experiment for me is whether the online community can work this way, or whether it really does need a more directed, formal research project. At the moment I think we have good conversation. Conversations and dialogue can be spontaneous but they can also be focused, with a specific aim. Similarly, we have a loose-knit community of sorts, and communities can be around general areas of interest, but they can also have a common goal ('overthow a president', 'find Dave a date').

It is this second, more intentional layer that I was hoping we might foster this year. I like finding other people's posts and riffing off (or ripping off) ideas, but I want to explore whether the edublog community can do a bit more. Whether we can have purposeful conversations, aimed at addressing a specific problem.

To kick off I have created a wiki here for setting out the big questions.

I am go to aggregate anything with #YOFL and #FLY tags at some point. I make the solemn promise to summarise these once a month and try to put some interpretation to them, although anyone else is also free to do the same of course.

I will also set in motion a few activities, such as those listed above.

So, what do you think? Oh, here's a logo: