ACM Conference on Computer Supported Cooperative Work and Social Computing
26 February, 2013
San Antonio, TX
Wikipedia-Supported Cooperative Work session
Jonathan Morgan – Tea & Sympathy: crafting positive new user experiences on Wikipedia
New user experiences on Wikipedia are intimidating – unintuitive markup, support impersonal, overwhelming load of rules. Everything about new user experience seems to be intended to drive people away. Women’s experiences as minority, dissatisfaction with abrasive norms, wish for more collaborative participation – good parallel for newbies in general, same issues.
Teahouse: created to meet needs of new users, want to help address deficiencies for newcomers. Tried to figure out what they want and how to support that. Newbies usually hesitant due to concern over humiliation, intimidated by making first move to socialize when you don’t know norms.
Most of the experience isn’t scripted. Welcome – personal welcome, another host gives a tip with less overwhelming links than they would get for a “getting started” comment in Wikipedia proper. Very different sensibility than Wikipedia, and the difference lies in the norms. Set up list of guidelines for interaction – not enforceable rules – if they work, hope that it’s because they remind Wikipedians what it was like to be a newbie.
Engaging new editors effectively, what they valued about the Teahouse wasn’t usability but sociability. Human-human interactions were more salient. Looked at retention as well, Teahouse guests made more edits on average than non-Teahouse control groups, more of them becoming high volume contributors. They stuck around longer, and were substantially more active than those who didn’t do Teahouse. Wikipedian hosts enjoyed it too.
Still going strong after a year, looking at using badges, data set available upon request.
Aaron Halfaker – Making Peripheral Participation Legitimate: Reader engagement experiments in Wikipedia
Power law participation even among small number of regular editors. LPP lens – how do newcomers enter communities of practice? Initial new member tasks should be simple, low risk of causing problems in community, and productive so it’s not useless – same thing as in Bryant et al 2005, but the participation experience has changed a lot since then.
Newcomer’s first edits are now much more complex, with higher rate of failure (immediate reversion), and it’s getting worse. Wikipedia is very, very complicated in terms of rules – lots and lots of policies. Verifiability policy alone is a 6-page document.
Tried setting up new task for newbie that’s much easier to do – added suggestion box on bottom for feedback, which will hopefully help editors improve articles and also get new people involved. It’s simple and low-risk, but is it productive? 3 experiments to find out.
RQ1: How do requests for participation affect quality and quantity of feedback – some engineered toward reader concerns, some for editor concerns. Found that engineering request toward reader concerns boosted contribution rate by 45% over just asking for rating, no loss of utility.
RQ2: How does prominence of the request affect quality and quantity of feedback? (placement @ end of article versus top, affects visibility) The more prominent button that scrolled along with the article boosted feedback contribution rate by 108% (on top of prior 45%) with no loss of utility – very surprised by this, ran a bunch of confirmatory experiments.
RQ3: How does presence of feedback form affect new editor conversion? Could cannibalize primary contributions (edits) but could be a stepping stone. Found that asking people to edit after feedback submission increased new conversions by 151%, but 20% drop in probability of editing (after?) that first week.
Conclusions: no tradeoff between quantity and quality of participation; inviting readers to convert increased rate of new editors but at lower success rate; need to balance value of contribution against cost of moderating. If that equation is wrong, it won’t work. Article feedback tool almost being taken down because Wikipedia has no way to moderate the feedback.
Stuart Geiger – Using Edit Sessions to Measure Participation in Wikipedia (or: Edit Counts Considered Harmful)
Started w/ discussion of cross-methods collaboration – what each of the authors see when they look at Wikipedia (very funny comparison). Bringing together ethnographic understandings with more computational approach to iteratively, inductively develop a quantitative measure.
Big difference in how academics evaluate participation and how Wikipedians do. Many Wikipedians think edit counts are inadequate, even harmful – researchers focus on discrete units of work but those don’t capture the work experience. Need to expand our idea of what counts as work. We measure work in hours in organizational settings – 20 hours of work for SVs, not 20 questions answered or 20 problems solved. Time-based metrics more relevant.
So, edit session: graph of Jimmy Wales’ edits over time. Wikipedians edit in short punctuated bursts, usually in short periods of time. Tasks therefore segmented, can use same techniques as counting edits, look at time between 2 sequential actions, when less than cutoff of an hour apart, as time spent in editing session.
Determined cutoff by looking at histogram of time between edits. Also found 3 distributions – within-session breaks, intra-session breaks, and what Wikipedians call “wiki-breaks” – long-term departures from community – usually a few months.
Looking at Wikipedia in terms of time spent editing, there’s actually a lot more growth in participation that edit counts suggest. Total concrete bursts of editing on Wikipedia (not including research, off-wiki interaction, etc.) add up to over 100M hours. That’s over 11,700 human years – greater than the population of Tuvalu and 14 other small countries!
Limits to measures like these – it misses lots of work that doesn’t directly make an edit (e.g., background research), kind of creepy and invasive as every interaction in Wikipedia is an edit, acts a little like a diary study. Similar issues elsewhere, but edit sessions contribute to a more holistic measure.