Jump to content

Wikipedia:Pending changes/Metrics/Preliminary Analysis

From Wikipedia, the free encyclopedia

Analysis by Howief

[edit]

Note: Per Ocaasi's request, here is some basic analysis on the Pending Changes data. This analysis is meant to serve as a starting point to help focus the data for community discussion. Since individual reviewers have the deepest experience with the feature and how it works with specific types of articles, I've kept most of the commentary on the quantitative aspects. The exception here is the discussion of most frequently edited articles, where I pointed out some obvious commonalities. Hopefully the cuts of data in this analysis will be useful. If not, there are the full tables which people may use to look at the data in detail.

Throughout this analysis, I've focused on metrics that we've frequently used to evaluate the success of Pending Changes:

  • Participation: approximated by the number of anonymous edits an article under Pending Changes receives
  • Quality: the quality of the anonymous edits. This is approximated by the % of anonymous edits that are not reverted
  • Work: the amount of work required from reviewers to manage the incoming anonymous edits. This is approximated by the number of anonymous edits an article receives, the idea being that anonymous edits require review, which requires work from reviewers.

Howief (talk) 23:52, 20 August 2010 (UTC)[reply]

First, I made a simplifying assumption. The amount of time that an article is under pending changes for varies and this variation results in outliers later in the analysis. For example, slag was under pending changes for only 38 seconds. I decided to exclude articles that have been under pending changes for less than one week (152 out of 1647, or 9.2% of total). The total number of articles that have been under pending changes for at least one week is 1495.

By Average Daily Anon Edits

[edit]

Comparing articles by the total number of edits would not take into account the fact that some articles have been under pending changes longer than others. So normalizing by the number of days under pending changes can help us compare articles that have been under Pending Changes for differing amounts of time. Here is a histogram of articles by the number of days the articles were under Pending Changes:

Ave Edits/Day # of Articles % of total
>10 3 0.2%
5-10 14 0.9%
2-5 70 4.7%
1-2 150 10%
.5-1 206 14%
.25-.5 238 16%
.1-.25 226 15%
0-.1 303 20%
0 285 19%
Total 1,495 100%

One of the goals of Pending Changes is to open up semi-protected articles to anonymous editors. Yet from the data, we see that 84% of the articles under pending changes receive, on average, less that one edit per day. Much of this may be due to the natural edit velocity of the articles that have been placed under Pending Changes. Note that this is not a judgement on the value of the edits as a single edit may prove very valuable to the quality of an article. All this analysis tells us is that the frequency of anonymous edits is relatively low for a large percentage of the articles that were under Pending Changes.

Of articles that were under Pending Changes for at least one week, here are the 10 articles that received the most number of edits per day from anonymous users:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Total Drama World Tour 20.77 493 23.7 71.8%
Spain national football team 8.00 158 19.8 65.8%
Toy Story 3 12.47 146 11.7 51.4%
Australia's Next Top Model, Cycle 6 14.21 140 9.9 65.7%
Juan Pablo Pino 14.00 132 9.4 17.4%
Wayne Rooney 8.05 70 8.7 12.9%
SummerSlam (2010) 14.33 114 8.0 27.2%
The Twilight Saga: Eclipse 30.00 224 7.5 42.0%
Jason Leopold 7.63 56 7.3 17.9%
Manchester City F.C. 10.81 77 7.1 26.0%

It's interesting to note that almost all of these articles are about current events. Toy Story 3 and The Twilight Saga: Eclipse were released around the time of the trial. Total Drama World Tour and Australia's Next Top Model, Cycle 6 are TV shows that were active during the time of the trial. The World Cup gives rise to Spain national football team, Juan Pablo Pino, Wayne Rooney, and Manchester City F.C.. And SummerSlam (2010) is a WWE Wrestling event that took place on Aug. 15, 2010.

Here are examples of articles that received between 1-2 edits per day:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Thomas Edison 7.16 12 1.7 0.0%
Club Penguin 42.20 70 1.7 14.3%
The Simpsons 33.35 55 1.6 12.7%
Sum 41 44.67 73 1.6 49.3%
2010 Central Canada earthquake 42.30 69 1.6 34.8%
Nicolas Mahut 41.54 67 1.6 67.2%
Republic of Macedonia 27.28 44 1.6 15.9%
Oksana Grigorieva 22.40 36 1.6 50.0%
Baseball 44.28 71 1.6 4.2%
World War II 10.64 17 1.6 0.0%

And the following are examples of articles that received no edits during the time they were under pending changes:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Sonny Fai 45.29 0 -
Richard Goldberg 45.29 0 -
Wildcliff 45.29 0 -
Steven Geller 45.29 0 -
Robert Garside 45.28 0 -
Tran Van Huong 45.28 0 -
William B. Black (Illinois politician) 45.28 0 -
Suryoyo Sat 45.23 0 -
Tig Notaro 45.23 0 -
Steve Hoffman 45.23 0 -

If we take the articles that have at least one anonymous edit per day while under pending changes (237 total articles), we can then look at the quality of the edits. Articles which have at least 50% of anonymous edits unreverted:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Alvin Greene 17.00 25 1.5 88.0%
Uruguay national football team 33.14 111 3.3 82.0%
Total Drama World Tour 20.77 493 23.7 71.8%
France 34.43 150 4.4 68.0%
Nicolas Mahut 41.54 67 1.6 67.2%
Sonisphere Festival 46.39 66 1.4 66.7%
Angel Locsin 17.65 18 1.0 66.7%
Spain national football team 8.00 158 19.8 65.8%
Australia's Next Top Model, Cycle 6 14.21 140 9.9 65.7%
John Isner 41.54 89 2.1 65.2%
Australia 44.28 90 2.0 64.4%
The Hunchback of Notre Dame (1996 film) 45.23 51 1.1 62.7%
Gary Glitter 43.30 45 1.0 62.2%
Madonna (entertainer) 42.23 118 2.8 60.2%
This Is It (Michael Jackson concerts) 45.44 62 1.4 58.1%
Big Brother 2010 (UK) 14.00 56 4.0 55.4%
2010–11 Arsenal F.C. season 28.61 34 1.2 52.9%
2010 Formula One season 25.29 55 2.2 52.7%
Toy Story 3 12.47 146 11.7 51.4%
Oksana Grigorieva 22.40 36 1.6 50.0%
List of programs broadcast by Cartoon Network 21.39 30 1.4 50.0%

On the opposite end of the spectrum, the following articles that have at least one anonymous edit per day, none of which were unreverted:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
List of serial killers by country 9.20 32 3.5 0.0%
Osama bin Laden 9.16 31 3.4 0.0%
David Henrie 7.26 19 2.6 0.0%
Batman 11.21 25 2.2 0.0%
Chicken 32.90 64 1.9 0.0%
Emile Heskey 15.32 29 1.9 0.0%
Robert Kotick 8.58 15 1.7 0.0%
Brad Pitt 10.92 19 1.7 0.0%
Thomas Edison 7.16 12 1.7 0.0%
World War II 10.64 17 1.6 0.0%
Westboro Baptist Church 34.14 49 1.4 0.0%
Dinosaur 43.55 60 1.4 0.0%
Akshay Kumar 12.33 16 1.3 0.0%
Apple 34.38 42 1.2 0.0%
Danielle Staub 18.91 23 1.2 0.0%
South Slavic languages 16.54 20 1.2 0.0%
Christopher Columbus 29.12 34 1.2 0.0%
Penguin 11.48 13 1.1 0.0%
Victoria of the United Kingdom 13.58 15 1.1 0.0%
Turtle 19.36 21 1.1 0.0%
Moon 20.08 21 1.0 0.0%
Amy Winehouse 11.48 12 1.0 0.0%
Cheetah 11.62 12 1.0 0.0%
Obesity 45.79 46 1.0 0.0%

By Total Anon Edit Count

[edit]

Another way of cutting the data is by anonymous edit count (rather than average anonymous edits per day). The following is a histogram based on the total number of edits during the trial:

Total Anon Edits # of Articles # of Total
>200 3 0.2%
100-199 32 2.1%
50-99 108 7.2%
25-49 190 13%
10-24 302 20%
5-9 192 13%
1-4 383 26%
0 285 19%
Total 1495 100%

Of articles that received at least 50 edits during the period, the following articles had over 50% of the anonymous edits unreverted:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Uruguay national football team 33.14 111 3.3 82.0%
Total Drama World Tour 20.77 493 23.7 71.8%
France 34.43 150 4.4 68.0%
Nicolas Mahut 41.54 67 1.6 67.2%
Sonisphere Festival 46.39 66 1.4 66.7%
Spain national football team 8.00 158 19.8 65.8%
Australia's Next Top Model, Cycle 6 14.21 140 9.9 65.7%
John Isner 41.54 89 2.1 65.2%
Australia 44.28 90 2.0 64.4%
The Hunchback of Notre Dame (1996 film) 45.23 51 1.1 62.7%
Madonna (entertainer) 42.23 118 2.8 60.2%
This Is It (Michael Jackson concerts) 45.44 62 1.4 58.1%
Big Brother 2010 (UK) 14.00 56 4.0 55.4%
2010 Formula One season 25.29 55 2.2 52.7%
Toy Story 3 12.47 146 11.7 51.4%

And of the articles that received at least 50 edits during the trial, the 20 articles with the lowest percentage of unreverted anonymous edits:

Page title Time under Pending (days) Total anon edits while under Pending Anon edits/day % nonreverted/total anon edits
Will Smith 18.81 58 3.1 6.9%
Frank Lampard 46.46 58 1.2 6.9%
Star Wars 46.27 59 1.3 6.8%
Cat 46.43 153 3.3 6.5%
John Terry 46.44 51 1.1 5.9%
Ranbir Kapoor 49.86 53 1.1 5.7%
The Beatles 44.26 129 2.9 4.7%
Miloš Krasić 46.46 67 1.4 4.5%
Baseball 44.28 71 1.6 4.2%
World of Warcraft 39.65 98 2.5 4.1%
Jessica Alba 43.32 50 1.2 4.0%
Down syndrome 43.55 53 1.2 3.8%
Pig 43.33 58 1.3 3.4%
RuneScape 37.03 121 3.3 3.3%
Salman Khan 46.40 93 2.0 2.2%
Atheism 39.64 51 1.3 2.0%
Wonders of the World 45.78 51 1.1 2.0%
William Shakespeare 22.89 64 2.8 1.6%
Chicken 32.90 64 1.9 0.0%
Dinosaur 43.55 60 1.4 0.0%

Hopefully these cuts of data will help focus the discussion of Pending Changes. Please feel free to add additional commentary. Howief (talk) 23:52, 20 August 2010 (UTC)[reply]

Addition by Chris.urs-o

[edit]

I believe that the main goal of Pending Changes is that destructive edits of IPs are not going live. So that the frustrated IPs will do less vandalism. I do not understand why an IP does vandalism, if it does not get visible after all. The articles on the table were current events, the IP behaviour changes too much over time :( Of course, IPs could have tried to falsify the results of the Pending Changes trial to dump the feature :( --Chris.urs-o (talk) 09:57, 13 September 2010 (UTC)[reply]

Page title Pending Changes after Total IP edits before Pending Rev of these IP edits nonreverted/total IP edits Pending Changes: nonreverted/total IP edits
Spain national football team "21:16, 11 July 2010" 143 (10 days) 15 89.5% 65.8%
Toy Story 3 14:49, 23 July 2010 156 (10 days) 30 80.8% 51.4%
Australia's Next Top Model, Cycle 6 21:01, 21 July 2010 200 (10 days) 68 66.0% 65.7%
SummerSlam (2010) 18:11, 21 July 2010 167 (10 days) 45 73.1% 27.2%
The Twilight Saga: Eclipse 21:24, 25 April 2010 32 (10 days) 17 46.9% 42.0%
Jason Leopold 10:57, 28 July 2010 7 (10 days) 2 71.4% 17.9%