¹¹institutetext: IIT Kharagpur, Kharagpur, India - 721302
²²institutetext: Adobe Research, Bangalore, India - 560087
³³institutetext: TU Darmstadt, Darmstadt, Germany - 64289

When expertise gone missing: Uncovering the loss of prolific contributors in Wikipedia

Paramita Das 11 Bhanu Prakash Reddy Guda 22 Debajit Chakraborty 11
Soumya Sarkar 33 Animesh Mukherjee 11

Abstract

Success of planetary-scale online collaborative platforms such as Wikipedia is hinged on active and continued participation of its voluntary contributors. The phenomenal success of Wikipedia as a valued multilingual source of information is a testament to the possibilities of collective intelligence. Specifically, the sustained and prudent contributions by the experienced prolific editors play a crucial role to operate the platform smoothly for decades. However, it has been brought to light that growth of Wikipedia is stagnating in terms of the number of editors that faces steady decline over time. This decreasing productivity and ever increasing attrition rate in both newcomer and experienced editors is a major concern for not only the future of this platform but also for several industry-scale information retrieval systems such as Siri, Alexa which depend on Wikipedia as knowledge store. In this paper, we have studied the ongoing crisis in which experienced and prolific editors withdraw. We performed extensive analysis of the editor activities and their language usage to identify features that can forecast prolific Wikipedians, who are at risk of ceasing voluntary services. To the best of our knowledge, this is the first work which proposes a scalable prediction pipeline, towards detecting the prolific Wikipedians, who might be at a risk of retiring from the platform and, thereby, can potentially enable moderators to launch appropriate incentive mechanisms to retain such ‘would-be missing’ valued Wikipedians.

Keywords:

Wikipedia Missing editor Prolific contributor Platform moderation.

1 Introduction

Wikipedia has emerged as an immensely popular digital encyclopedia, over the last decade. This is primarily attributed to the “be bold” policy, which permits anyone to contribute on almost all wikipages. Encouragement toward global collaboration generates hundreds of millions of views monthly which resembles Wikipedia as a reliable source of information, irrespective of socio-economic backgrounds and cultures [39, 26]. The workhorse behind this success story is a large pool of voluntary editors. These group of people maintain Wikipedia pages behind the scenes which includes creating new pages, changing contents, making sure that the fact provided is appropriate etc. It is imperative for the survival of this platform that this resource is nurtured so that they can operate with due diligence.

In these lines there have been several works, which attempt to understand diverse roles editors play in the community [45, 42, 33]. These studies revealed valuable insights about editors’ motivation, group behaviour and productivity, thus further reinforcing our knowledge about community health. A complementary direction of exploration includes developing personalised recommendation system to identify appropriate pages and mentors for a newbie Wikipedia editor so that seamless onboarding to the platform can be achieved [8, 43]. However notwithstanding former efforts, there has been a steady attrition of Wikipedia editors [1]. One possible reason could be personal disillusionment from the project. However, some notable reasons are increased bureaucracy and incivility [25, 12]. In particular, to maintain the high quality standards of encyclopedic content, Wikipedia community has gradually become impermeable for the editors, resulting in unreasonable abuse of power such as blocking of users, deletion of good faith edits, and unexplained conflict arbitration [23]. Out-flux of contributors has been a chronic problem for other knowledge sharing platforms as well such as Yahoo! Answers, Baidu Knows etc. [15, 22, 14].

Loss of Wikipedians is a loss of Wikipedia: Editors, or Wikipedians [35, 40] are the soul of Wikipedia; without their active participation, Wikipedia will eventually become stagnant. As mentioned earlier, the loss of editors and simultaneously editor retention [17] is a Wikipedia-wide problem.

Related wikiprojects: Several wikiprojects [4], hosted by Wikipedia aim to construct a stronger bonding among Wikipedians and also figuring out several measures [2, 3] to establish the objective of editor retention. There exists corresponding wikiproject [5] dedicated to find the cause of editor depletion and early identification of editors at risk of forsaking Wikipedia. Often expert editors withdraw themselves because of the discontent with Wikipedia’s policies and widespread norms resulting them into semi-retired or permanent-retired often not disclosing the prime reasons behind the retirement. As an example, the Wikipedia project (WP:MISS)¹¹1https://en.wikipedia.org/wiki/Wikipedia:Missing_Wikipedians maintains a list of experienced Wikipedians who have made no edits to any article after a fixed time point in their life and are defined as missing Wikipedians. All of these editors contributed at least 1000 edits in their lifetime and can be considered as prolific editors. Even after being inactive, many of them are still featured among top editors²²2https://tinyurl.com/3ewzb468. As of June 2020, this list includes 1226 such editors whom we refer to henceforth as missing Wikipedians or missing editors interchangeably. In contrast, the pool of editors who are continuing to contribute as active participants till date, will be referred as active editors hereafter. Recently, a wikiproject named as Community Health Metrics: Understanding Editor Drop-off³³3https://tinyurl.com/3e6jj6zz has been initiated to understand the editor drop-off across multiple language versions of Wikipedia. The project aims to understand various dynamics in editors’ life-cycle with a special attention to the veteran editors of the community who are at the risk of leaving Wikipedia. Our work is in line with the objectives of these projects - identifying the decline of prolific editors whose contributions are measured significant in the community. We observed that majority of editors faced some form of impediment from the Wikipedia community that curbed their interest in further editing. Finally, these missing editors have left the platform and there is no assurance of their return⁴⁴4https://tinyurl.com/35n624a6. Most of these editors do not reveal the explicit reason for leaving Wikipedia. Some editors have pointed out typical instances of disagreement, bureaucracy on their personal user talk pages that influenced them for announcing retirement. Among many example reasons that we observed on editors’ user pages, a typical example is the following-

“I have left Wikipedia. I do not see it as acceptable to have advertisements, whether they be for brand identity or for a product, on Wikipedia.” –Im*****v

The above text shows an editor’s disagreement with the community for compromising NPOV policy⁵⁵5https://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view of Wikipedia as a prime cause of his/her withdrawal from Wikipedia.

Editor attributes as a cue?: Our hypothesis is that the early signals of retirement of a missing editor could be hidden in his/her last trail of editing activities. In other words, the editing activity patterns of the missing editors would have difference with the existing active editors. Further, the typical sentiments, emotions expressed on user talk pages should strike different attitude of missing editors compared to the active group of editors. Due to the challenges and hindrance of the platform, missing editors might follow different profile of quality as compared to active editors. Based on this hypothesis, in this paper, we develop a framework for the discovery of key editors who have stopped content generation in Wikipedia. For this purpose, we first identify 1146 editors who have no edits in the calendar year of 2020 and denote them as missing editors. We next select a set of 2569 editors whose editing activity (i) are ongoing and (ii) match the overall editing activity level of the missing editors. This second set is denoted as active editors. We compare the two sets of editors on a longitudinal scale in the attempt to identify various properties related to their activity patterns and language usage that could significantly differentiate them. Our main research questions are as follows.

RQ1: To what extent activity/linguistic/quality features of editors can help us discern missing from active editors?

RQ2: Using the discriminative editor activities can we predict currently prolific editors who have a possibility to leave the platform in the near future?

Our contributions: We make the following novel contributions in this paper.

•

We curate a first ever dataset of missing editors, a comparable dataset of active editors along with all the associated metadata that can appropriately characterise the editors from each dataset. (see section 3)
•

First we put forward a number of features describing the editors (activity and behaviour) which portray significant differences between the active and the missing editors. (see section 4)
•

Next we use SOTA machine learning approaches to predict the currently prolific editors who are at the risk of leaving the platform in near future. Our best models achieve an overall accuracy of 82% in the prediction task. (see section 5)
•

We perform rigorous ablation studies to provide further insights into our results. We discuss various nuanced observations that get manifested in the course of our study. An intriguing finding is that some very simple factors like how often an editor’s edits are reverted or how often an editor is assigned administrative tasks could be monitored by the moderators to determine whether an editor is about to leave the platform. (see section 5)

To the best of our knowledge this is the first work which proposes an automatic approach to predict, early on, the editors who have a propensity to leave the platform soon. We believe that the moderators can use these attributes to launch suitable platform governance [18] measures to appropriately incentivize these editors and, thereby, retain them on the platform. Further the observations and insights that we obtain from our results can he useful in designing the specific incentive strategies [6].

2 Related work

The reason behind the popularity of Wikipedia is often attributed to the hundreds of thousands of volunteers from all around the world, but several tens of thousands of Wikipedians and their collaboration are very crucial for the generation and maintenance of healthy and informative content [41, 37, 24]. Researchers are trying to overcome different challenges disseminated by open source of knowledge and information especially the quality and trust issues [36, 10] of this gigantic platform.

With the constant effort of regulation and maintenance of encyclopedia content, it appears that Wikipedia is at the peak of its popularity; however, experts noted that it is at the danger of sharp decline of its active editors [20, 44]. The revert of edits is highly effective in controlling vandalism [19], sometimes it becomes harsh to the editors, especially the newly joining ones [21]. The topic of volunteer retention [25] is one of the basic concerns of all peer-production systems, tied to their preliminary survival. Hence, researchers of Wikipedia shed light on the retention of new contributors [11, 31, 27]. However, works on early identification and retention of prolific editors at risk of retiring are scarce. Therefore, in this work we set out to develop an automatic framework that can accurately identify editing activity signatures that point to the early signals resulting in the abdication of an editor.

3 Dataset description

Wikipedians are the prime elements of our experiment. In order to curate two sets of editors – missing and active, we follow a two step approach.

List of missing editors: First, we prepare a list of editors who are mentioned as missing by Wikipedia. We extracted the list of missing Wikipedians⁶⁶6https://en.wikipedia.org/wiki/Wikipedia:Missing_Wikipedians and their last date of contribution using XTools⁷⁷7https://www.mediawiki.org/wiki/XTools/API. Wikipedia has mentioned 1226 such editors in the list at the time of our dataset collection (June, 2020). From this list we remove those that we found still contributing in 2020 despite being declared as missing by Wikipedia. This results in 1146 unique editors who further have not edited in the calendar year 2020.

List of active editors: Next, we collect a set of 2569 editors who are still active, i.e., are still editing different Wikipedia articles. In order to build a comparable set of active editors, we performed the following.

•

First, we collect the list of top 60 pages (top 20 from each namespace – ‘Main’, ‘Wikipedia’ and ‘Talk’) that have been most frequently edited by each missing editor. This leads us to an initial set of 68,760 Wikipedia pages.
•

Next we look into the 100 latest revisions of each of the 68,760 pages and the editors who contributed to the latest revisions are extracted from the revision history of the corresponding pages. Further, for every missing editor, only top 10 active editors are selected based on the frequency of their edits on the above mentioned top edited 60 pages. Formally, for a missing editor $M$ , the editors $A_{1},A_{2},...,A_{10}$ are assumed to be the most active if they have the highest edit counts in the last 100 revisions of the top edited pages $P_{1},P_{2},P_{3},...,P_{60}$ by the editor $M$ . These editors are placed into the pool of active editors. For every missing editor we perform this exercise leading us to a set of 5213 unique active editors. We confirm that none of these editors are listed in the missing Wikipedians page.
•

As a final step, we compute the average number of edits per day for the active editors and the missing editors. We next compute the mean of the average edits per day for the missing editors (say $m$ ). Next we compute the L2-norm of $m$ with the average number of edits per day of each of the active editors. We consider those active editors for whom the L2-norm is within one standard deviation of $m$ . This results in 2569 active editors. The difference in the distributions of the average edits per day for the missing and active editors so chosen is not statistically significant (Mann Whitney U test, $p=0.14$ ) thus making them comparable.

4 Feature space design

In this section we describe our feature space which we develop to differentiate the active from the missing editors. The features can be categorised into three broad classes–

Activity features, Quality features, Linguistic features.

Resources for the work is available at: https://github.com/paramita08/Missing-Active-Wikipedians

4.1 Activity features

The features we propose in this category describe the activity patterns of editors. The choice of these features is motivated by the fact that these would reflect different aspects of editor’s interactions with the community over time. These features, we believe should be useful to capture the temporal change in the behavior of the editors who have a possibility to leave the platform. We extract the details of the latest 50 non-automated edits of every user using the User API of XTools⁸⁸8https://www.mediawiki.org/wiki/XTools/API/User#Non-automated_edits and identify the following activity features.

Edits in different namespaces: Among different namespaces⁹⁹9https://en.wikipedia.org/wiki/Wikipedia:Namespace mentioned, we consider the editor contribution in 4 namespaces - article pages i.e., namespace 0; article talk pages, i.e., namespace 1; administrative main pages - Wikipedia, i.e., namespace 4 and the administrative talk pages - Wikipedia talk, i.e., namespace 5. Majority of contribution of editors is restricted in these respective namespaces [30] and we denote the four namespaces as F1, F2, F3 and F4 following the order as mentioned above. Our hypothesis is that the frequency of edits in the first two namespaces (i.e., 0, 1) will be affected drastically for the missing editors in their latest series of contributions. We have included these two namespaces to observe if the missing editors exhibit a decline in participation in Wikipedia’s policy decisions. As shown in Figure 1 (a), the average count of edits in all the namespaces except namespace 5 is higher in case of active editors. This trend implies a slow decline in contribution from missing editors possibly pointing to their absence in near future.

Major and minor edits: A minor¹⁰¹⁰10https://en.wikipedia.org/wiki/Help:Minor_edit edit is the one denoting minor changes such as typographical errors, formatting errors, reversion of definite vandalism etc. that the editors believe need no further reviews. In contrast, any contribution that changes the article content and needs to be reviewed for the acceptability to the community of editors is denoted as major edit. We hypothesize that an increase in the number of minor edits as compared to major edits can bear an early signature of their disengagement, finally abdicating the platform. Hence, we included the count of edits in both categories - major (F5) and minor (F6) of the latest revisions in our feature set. The cumulative edit count of major and minor edits in Figure 2 show that for major edits, active editors exhibit a steady state in the count over time but the decline is extremely unstable for the missing editors and almost always below that of the active editors. On the other hand, close to their retirement, the missing editors seem to be engaging themselves in an increasingly more number of minor edits as compared to the active editors.

Length of the edits: The average length of the edits is captured by this feature in terms of the bytes added or deleted per edit. Further, we classify this feature into four categories - (i) addition in major edits (F7), (ii) deletion in major edits (F8), (iii) addition in minor edits (F9), (iv) deletion in minor edits (F10). In each of the cases, we compute the average bytes added (or deleted) in case of a major (or minor) edit over the latest contribution of the editors. Once again the hypothesis is that the length of the edits done by the missing editors shall diminish over time. An interesting observation that we have here is that the missing editors seem to engage more in deleting minor edits than adding minor edits (see Figure 1(b)).

Span (in months): We compute the time span (denoted as F11) in months taken by each editor to complete the latest revisions of contribution. The span is expected to be larger in case of missing editors compared to the active editors indicating a possible loss of overall interest in the platform. We find that the average span in case missing editors is 2.04 months while for active editors this is 0.82 months.

ORES score of the edits: ORES¹¹¹¹11https://www.mediawiki.org/wiki/ORES, the web-service API by Wikimedia Foundation is used in automating several wiki management tasks such as assigning scores to individual edits of editors and predicting the edits to be damaging or good-faith. We compute the average ORES score (represented as F12) for each of the editors in our dataset over their contribution span. In addition, we consider the count of the good faith (denoted as F13) vs the damaging (denoted as F14) edits for every editor in our dataset. On average we did not see any difference in the quality of edits done by the two classes of editors. Further both the classes have a very small number of damaging edit contributions.

Refer to caption — (a) The average count of edits in different namespaces of active and missing editors.

4.2 Quality features

Although the content of any Wikipedia article is factual, its quality heavily depends on the dexterity of the editors. We formulate the following quality features that could be potential indicators of the future churn.

Count of reverts: Reverting is the way of reversing a prior edit or undoing the effects of one or more edits, which typically results the article to be restored to a version that existed sometime previously. Reverts are discouraging to any editor, irrespective of their experience [23, 21] and can potentially impede an editor toward making future contributions to the platform. To investigate this, we scrape the number of reverted edits of every individual editor in their respective top 50 most edited pages in the main namespace only. This is since almost all editors make their largest chunk of contributions to the main namespace. From Table 1 we observe that missing editors have experienced a larger number of reverts which is in line with our hypothesis that more reverts potentially disengage editors from the platform. Further we revisit a few nuanced cases in which reverts and the comments with the reverted edits had a negative impact on the missing editors. As shown in the Table 2, a missing editor received several foul comments with the reverted edits which were posted on her talk page publicly. Finally, she stopped editing after experiencing a lot of reverts toward the end of 2004 with a clear message declaring her retirement on the user page in 2005. We also look into the talk pages of active editors and find that they had experienced less sensitive comments for their reverted edits and an example scenario is depicted in the second row of the Table 2.

Quality	Missing $(\mu,\sigma)$	Active $(\mu,\sigma)$
Avg. revert count per page	(0.031, 0.041)	(0.026, 0.031)
Admin score	(734.38, 165.38)	(823.86, 174.93)

Table 1: Mean (

\mu

) and std. deviation (

\sigma

) of the quality features for missing and active editors.

Admin score: Wikipedia maintains a pool of editors, known as admins, who are responsible to perform various administrative tasks and the tasks are pivotal in monitoring various quality issues of Wikipedia. These set of people act voluntarily and every wikiproject tries to find the admins from among the prolific and experienced editors. XTools provides the admin score¹²¹²12https://xtools.readthedocs.io/en/3.1.6/tools/adminscore.html, intended to find how suitable an user is for serving as an admin. The more the admin-score, the higher is the chance of the editor to become an admin. We observe in Table 1 that active editors on average have a higher admin score and therefore higher chances of being an admin as compared to the missing editors.

comments received by	comments
a missing editor	if that happens I will, with considerable regret, withdraw from Wikipedia altogether. An encyclopedia that can’t or won’t defend itself against Stalinist and LaRouche wreckers will never succeed and doesn’t desrve to. — A****, 01:05, 16 Nov 2004 (UTC)
an active editor	Greetings. I noticed you had undid my edits on H** C and J K, specifically the ones….I’ve since reverted them. But I’m opening discussion here, in case there’s disagreement. Looking forward to your thoughts. — Ga**, 14:06, 23 Dec 2015 (UTC)

Table 2: Example comments on editors’ talk page showing comments received with reverted edits. The comments received by an active editor is less harsh as compared to a missing editor.

4.3 Linguistic features

As we have discussed earlier a number of editors have shared their basic information, Wikipedia-related activities, awards and badges earned etc. and sometimes their grievances¹³¹³13https://en.wikipedia.org/wiki/User:Bcrowell about the platform on their user pages that finally hold them back in continuing further contributions. We assume that these pages to be the profile of the editors and leverage the user generated text in characterizing the two sets of editors. We extract the text from the HTML version of the latest editor page and perform pre-processing to remove various links and HTML tags. We next extract various features from this pre-processed text.

POS tags: We compute the POS tags of the text in the editor pages using the NLTK parser [28]. Prior to tagging, we tokenized the sentences and words common to both the groups of editors (missing and active) are removed from the corpus. We also remove the stop words and the non-English words. We observe several frequent POS tag categories such as “JJ”, “NN”, “NNS”, “VBD”, “VBP” that exhibit significant differences between the missing and the active editors.

Empath categories: Empath is a text analysis tool [16] which can identify psycho-linguistic signs hidden in a text in the form of pre-validated lexical categories. For our purpose, we compute the normalised value for each predefined category for each sentence. We then average out this value for all the sentences present in the profile of an editor. We observe that a number of lexical categories (21 in number), i.e, “Internet”, “Noise”, “Trust”, “Reading”, “Violence”, “Negative Emotion”, “Positive Emotion” etc. show the difference in distributions for two classes of editors is statistically significant as per the Mann-Whitney U test. However the absolute differences between the two classes in terms of the values of the various lexical categories is negligibly small. However the absolute differences between the two classes in terms of the values of the various lexical categories is negligibly small.

Sentence vector: We generate the sentence vector using the Universal Sentence Encoder [13] which outputs a 512 dimensional vector representation of the text. We compute the sentence vector for every editor which can be thought of as a summary of the language usage of the users on the platform. To visualise how different the sentence vectors from the two classes are, we plot them using t-SNE [29]. We consider the first two principal components of the vectors for visualisation. Figure 4(a) shows that there is no clear separation between the sentence vector drawn from the two classes. However, there are at least two small but very distinct pockets of vectors from a particular class clustering together expressing opposing sentiments toward the platform. We handpick a few examples from the talk pages of editors belonging to each of these two clusters. These are shown in Figure 3. The difference in sentiments toward the platform is apparent in the selected sentences from the two clusters. While missing editors seem to express grudges against the platform, the active editors mostly narrate their positive experiences with the platform.

4.4 Feature correlation

To examine the relationship among the different features described above, we compute the Pearson’s correlation coefficients [9] between the all-pair features. Figure 4(b) shows the correlation coefficient matrix for the 16 features (except the linguistic features). We observe that some features are positively correlated and some features are negatively correlated (e.g., major edits and minor edits) with each other. However almost all the coefficients are in the range -0.05 to 0.05 (i.e., $\sim 0$ correlation) except for the pairs (addition in major edits, deletion in major edits) and (addition in minor edits, deletion in minor edits), which belong to the same category. We do not include the linguistic features in this study as they are characteristically very different from these features.

5 Classification of editors

In this section we use the mentioned features to automatically classify the missing editors from the active ones. However, in order to make the model outcomes interpretable we incrementally include the features to better understand their effect on the overall performance. We name the feature groups as follows-

•

Activity features as G1
•

POS tags and Empath as G2
•

Sentence vector as G3
•

Admin score as G4
•

Count of reverts as G5

Classification model: Corresponding to each editor in our dataset, we construct a normalised feature vector. We experiment with multiple classifiers such as Random Forest, Logistic Regression SVMs, XGBoost, AdaBoost. In this task, a binary classifier is built to predict if the given editor is a missing or active one. The train-test spilt for our experiments is set to 80:20. We use accuracy along with weighted precision, recall and F1-score as the evaluation measures.

Results: Among the different features, we observe that the 14 activity features, reverts and admin score seem to be performing much better than the linguistic features (see Table 3). The G1 feature group, i.e., the activity features seem to be the strongest individual discriminator resulting in an accuracy of 75% (F1-score 74%). Together with G4, this accuracy goes to 78% (F1-score 79%). Among the linguistic features, the sentence vector seems to be the most effective in conjunction with activity and the admin score features. Together this feature group G1, G3 and G4 attains an accuracy of 82% (F1-score 81%). The feature group G1, G4 and G5 also attains a similar accuracy of 82% (F1-score 81%). Larger groups of features do not seem to improve performance. These results further corroborate that missing editors hardly leave any special linguistic cues before they abdicate the platform.

Features	Classifier	Precision	Recall	F-score	Accuracy
G1	XGBoost	0.74	0.74	0.74	0.75
G2	XGBoost	0.63	0.68	0.64	0.68
G3	AdaBoost	0.63	0.67	0.63	0.67
G1 $\oplus$ G2	Random Forest	0.77	0.78	0.76	0.78
G1 $\oplus$ G3	XGBoost	0.75	0.76	0.75	0.76
G1 $\oplus$ G4	AdaBoost	0.78	0.79	0.79	0.78
G1 $\oplus$ G5	AdaBoost	0.78	0.79	0.79	0.77
G1 $\oplus$ G2 $\oplus$ G4	XGBoost	0.77	0.78	0.77	0.78
G1 $\oplus$ G3 $\oplus$ G4	XGBoost	0.81	0.82	0.81	0.82
G1 $\oplus$ G4 $\oplus$ G5	AdaBoost	0.81	0.82	0.81	0.82
G1 $\oplus$ G3 $\oplus$ G5	XGBoost	0.78	0.79	0.78	0.79
G1 $\oplus$ G2 $\oplus$ G4 $\oplus$ G5	Random Forest	0.82	0.82	0.82	0.81
G1 $\oplus$ G3 $\oplus$ G4 $\oplus$ G5	XGBoost	0.80	0.81	0.80	0.81

Table 3: Prediction outcomes: combination of different features. The green rows indicate the best result using the respective feature combination.

Feature importance analysis: In this section we investigate the importance of the individual features in this feature group that consists of only activity and quality features (16 in all). To this purpose, we use three different ways to compute the importance.

Feature importance function: We use the Gini impurity bases feature importance which is a standard technique in tree based ensemble methods [34] to compute the feature importance values. We observe the features that dominate the top of the rank list are - reverts, ORES score, admin score, deletion in major edits, edits in namespace 0 and span.

Permutation importance function: We compute feature importance using permutation importance [7] and observe almost similar importance order as that of the previous function. The features that come at the top 6 places of the importance list are - reverts, ORES score, admin score, span, edits in namespace 0 and edits in namespace 4.

LIME: LIME [38] is an explainability tool meant to generate explanations regarding the workings of a ML model. We use all the instances in the test data for generating the explanations. We note the top five features returned by LIME that are responsible for the prediction outcomes for each instance. We observe that across all the instances the set of features $\{$ reverts, admin score, edits in namespace 0, edits in namespace 1, ORES damaging true, edits in namespace 4 $\}$ present the best explanations.

Overall, we observe that the quality features are the most discrimiantory among all. This points to the fact that they constitute the best signals for identifying whether an editor is about to leave the platform. Platform moderators can deploy simple monitoring schemes to understand these signals early on and take preventive steps to minimise the loss of such prolific contributors.

6 Discussion

The primary objective of this work has been to bring forth the issue of the growing depletion of editors, especially the experienced editors in Wikipedia. One of the most important findings of our work is that missing editors can be differentiated from the existing active ones by a set of simple and interpretable characteristics of editors extracted from their interaction with the platform.

Dissatisfaction of prolific editors: Further to test the accuracy of our model, we have observed a number of specific cases where the classifiers are confident in predicting the missing editors. We revisited their talk pages and found that in many such cases the editors had to face bullying¹⁴¹⁴14https://en.wikipedia.org/wiki/Wikipedia:WikiBullying from other editors. Those bullying instances are usually mentioned as a part of the article talk pages. Sometimes the user has also been found to express his/her grievances on their personal user page. Figure 5 shows few example cases of toxic arguments and explicit reasons for leaving the platform that the editor had to face before they abdicated. Although a number of editors have stopped their contribution silently, the external causes of discontent with the platform that trigger the churn of prolific editors need to be considered carefully.

Enabling platform moderation: We collected a total of 8 false positive cases (mentioned as active but predicted as missing) in which our model is highly confident (confidence probability $>0.8$ ). We found that out of these sample cases, the last edit for 4 editors were done latest in $2017$ . We believe that all these editors should be potentially included in the list of missing editors but somehow got overlooked by the Wikipedia community. In fact, one out of these 4 cases have been actually identified as a missing editor in the most recent edition of the missing editor list, i.e., June 2021. From an application point of view, our system can monitor the current editor pool and flag any missing editor which the community may overlook. Suitable policies may be adopted to retain/reinstate such editors.

Design implication: Our idea was to keep the model as much interpretable as possible so that it is easy to understand whether the editing activity and the language usage across the platform are useful for classification. Further, our dataset is unbalanced and the total number of data points is limited to only $3715$ . Hence, in our work, we have followed simple feature based classification model. Moreover, we proposed a novel task in which the feature set design is such that they can be calculated in all category of Wikipedia articles. The existing literature tried to divide editors either with respect to a certain number of Wikiprojects or based on a set of Wikipedia articles. Therefore suitable baselines are scarce. Thus here we consider feature ablations as baselines as is normative in many NLP applications [32] where suitable baselines are not available .

7 Conclusion

In this paper we take a deep dive into the emerging issue of editor depletion in Wikipedia. Our investigation shows that longitudinal activity traces as well as linguistic clues in their profile, i.e., user talk pages provide necessary signals to ascertain whether a prolific contributor will be quitting the Wikipedia platform in near future. We identify a number of important features that summarise the level of activity and behaviour of an editor and can be used to investigate the casual connection for leaving the platform. For instance, if two missing editors have almost equivalent qualities and extremely similar editing activity patterns and have retired almost at the same time, then it might well be the case that both of them left the platform for very similar reasons. In future, we plan to use these features to identify the causes of a group of editor leaving Wikipedia. This would allow for a better streamlining of the governance measures based on the nature of the cause of disengagement. Upon acceptance, we plan to place all our data and code in the public domain to facilitate further research in this area.

References

[1] The future of wikipedia: Wikipeaks? https://www.economist.com/international/2014/03/04/wikipeaks, accessed: 2014-03-04
[2] Wikipedia: Editor activation. https://tinyurl.com/2njfds4y
[3] Wikipedia: Surviving new editor. https://tinyurl.com/4cjdyfs4
[4] Wikipedia: Teahouse. https://tinyurl.com/jzf346
[5] Wikipedia: Wikiproject editor retention. https://tinyurl.com/acmyrp8s
[6] Aaltonen, A., Lanzara, G.F.: Building governance capability in online social production: Insights from wikipedia. Organization Studies 36(12), 1649–1673 (2015)
[7] Altmann, A., Tolosi, L., Sander, O., Lengauer, T.: Permutation importance: a corrected feature importance measure. Bioinformatics 26 10, 1340–7 (2010)
[8] Balali, S., Steinmacher, I., Annamalai, U., Sarma, A., Gerosa, M.A.: Newcomers’ barriers… is that all? an analysis of mentors’ and newcomers’ barriers in oss projects. Computer Supported Cooperative Work (CSCW) 27(3), 679–714 (2018)
[9] Benesty, J., Chen, J., Huang, Y., Cohen, I.: Noise reduction in speech processing, vol. 2. Springer Science & Business Media (2009)
[10] Chen, X., Iwaihara, M.: Weakly-supervised neural categorization of wikipedia articles. In: International Conference on Asian Digital Libraries. pp. 16–22. Springer (2019)
[11] Choi, B., Alexander, K., Kraut, R.E., Levine, J.M.: Socialization tactics in wikipedia and their effects. In: Proceedings of the 2010 ACM conference on Computer supported cooperative work. pp. 107–116 (2010)
[12] Ciampaglia, G.L., Taraborelli, D.: Moodbar: Increasing new user retention in wikipedia through lightweight socialization. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. pp. 734–742 (2015)
[13] Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364 (2017)
[14] Danescu-Niculescu-Mizil, C., West, R., Jurafsky, D., Leskovec, J., Potts, C.: No country for old members: User lifecycle and linguistic change in online communities. In: Proceedings of the 22nd international conference on World Wide Web. pp. 307–318 (2013)
[15] Dror, G., Pelleg, D., Rokhlenko, O., Szpektor, I.: Churn prediction in new users of yahoo! answers. In: Proceedings of the 21st International Conference on World Wide Web. pp. 829–834 (2012)
[16] Fast, E., Chen, B., Bernstein, M.S.: Empath: Understanding topic signals in large-scale text. In: Proceedings of the 2016 CHI conference on human factors in computing systems. pp. 4647–4657 (2016)
[17] Gallus, J.: Fostering public good contributions with symbolic awards: A large-scale natural field experiment at wikipedia. Manag. Sci. 63, 3999–4015 (2017)
[18] Gorwa, R.: What is platform governance? Information, Communication & Society 22(6), 854–871 (2019)
[19] Green, T., Spezzano, F.: Spam users identification in wikipedia via editing behavior. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 11 (2017)
[20] Halfaker, A., Geiger, R.S., Morgan, J.T., Riedl, J.: The rise and decline of an open collaboration system: How wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist 57(5), 664–688 (2013)
[21] Halfaker, A., Kittur, A., Riedl, J.: Don’t bite the newbies: how reverts affect the quantity and quality of wikipedia work. In: Proceedings of the 7th international symposium on wikis and open collaboration. pp. 163–172 (2011)
[22] Kairam, S.R., Wang, D.J., Leskovec, J.: The life and death of online groups: Predicting group growth and longevity. In: Proceedings of the fifth ACM international conference on Web search and data mining. pp. 673–682 (2012)
[23] Kiesel, J., Potthast, M., Hagen, M., Stein, B.: Spatio-temporal analysis of reverted wikipedia edits. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 11 (2017)
[24] Kittur, A., Kraut, R.E.: Harnessing the wisdom of crowds in wikipedia: quality through coordination. In: Proceedings of the 2008 ACM conference on Computer supported cooperative work. pp. 37–46 (2008)
[25] Konieczny, P.: Volunteer retention, burnout and dropout in online voluntary organizations: stress, conflict and retirement of wikipedians. In: Research in Social Movements, Conflicts and Change. Emerald Publishing Limited (2018)
[26] Lemmerich, F., Sáez-Trumper, D., West, R., Zia, L.: Why the world reads wikipedia: Beyond english speakers. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. pp. 618–626 (2019)
[27] Li, A., Yao, Z., Yang, D., Kulkarni, C., Farzan, R., Kraut, R.E.: Successful online socialization: Lessons from the wikipedia education program. Proceedings of the ACM on Human-Computer Interaction 4(CSCW1), 1–24 (2020)
[28] Loper, E., Bird, S.: Nltk: The natural language toolkit. In: ACL 2006 (2006)
[29] Maaten, L.V.D., Hinton, G.E.: Visualizing data using t-sne. Journal of Machine Learning Research 9, 2579–2605 (2008)
[30] Maki, K., Yoder, M., Jo, Y., Rosé, C.: Roles and success in wikipedia talk pages: Identifying latent patterns of behavior. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1026–1035 (2017)
[31] Morgan, J.T., Halfaker, A.: Evaluating the impact of the wikipedia teahouse on newcomer socialization and retention. In: Proceedings of the 14th International Symposium on Open Collaboration. pp. 1–7 (2018)
[32] Mowery, D., Bryan, C., Conway, M.: Feature studies to inform the classification of depressive symptoms from twitter data for population health. arXiv preprint arXiv:1701.08229 (2017)
[33] Murić, G., Abeliuk, A., Lerman, K., Ferrara, E.: Collaboration drives individual productivity. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–24 (2019)
[34] Nembrini, S., König, I., Wright, M.N.: The revival of the gini importance? Bioinformatics 34, 3711 – 3718 (2018)
[35] Panciera, K.A., Halfaker, A., Terveen, L.: Wikipedians are born, not made: a study of power editors on wikipedia. In: GROUP ’09 (2009)
[36] Pinto, J.M.G., Kiehne, N., Balke, W.T.: Towards semantic quality enhancement of user generated content. In: International Conference on Asian Digital Libraries. pp. 28–40. Springer (2018)
[37] Proffitt, M.: Leveraging Wikipedia: Connecting Communities of Knowledge. American Library Association (2018)
[38] Ribeiro, M.T., Singh, S., Guestrin, C.: ”why should i trust you?”: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016)
[39] Samoilenko, A., Lemmerich, F., Zens, M., Jadidi, M., Génois, M., Strohmaier, M.: (don’t) mention the war: A comparison of wikipedia and britannica articles on national histories. In: Proceedings of the 2018 world wide web conference. pp. 843–852 (2018)
[40] Sarasua, C., Checco, A., Demartini, G., Difallah, D., Feldman, M., Pintscher, L.: The evolution of power and standard wikidata editors: comparing editing behavior over time to predict lifespan and volume of edits. Computer Supported Cooperative Work (CSCW) 28(5), 843–882 (2019)
[41] Suzuki, Y., Yoshikawa, M.: Mutual evaluation of editors and texts for assessing quality of wikipedia articles. In: Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration. pp. 1–10 (2012)
[42] Yang, D., Halfaker, A., Kraut, R., Hovy, E.: Who did what: Editor role identification in wikipedia. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 10 (2016)
[43] Yazdanian, R., Zia, L., Morgan, J., Mansurov, B., West, R.: Eliciting new wikipedia users’ interests via automatically mined questionnaires: For a warm welcome, not a cold start. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 13, pp. 537–547 (2019)
[44] Zhang, D., Prior, K., Levene, M., Mao, R., van Liere, D.: Leave or stay: The departure dynamics of wikipedia editors. In: International Conference on Advanced Data Mining and Applications. pp. 1–14. Springer (2012)
[45] Zhang, Y., Sun, A., Datta, A., Chang, K., Lim, E.P.: Do wikipedians follow domain experts?: a domain-specific study on wikipedia knowledge building. In: JCDL ’10 (2010)