The lead article effect
Lead articles in academic journals tend to receive morecitations than other articles. But does this mean they are anybetter? Research by Professor Victor Ginsburgh (Professor ofEconomics of Art and Culture, ECARES, Université Libre deBruxelles) on
“There exists a lively debate among scientists about evaluationmethods. Some prefer peer review-based research assessments, whileothers think that bibliometric citation-based methods should beused as a verifiable mechanism for promotion and distribution ofpublic research funds.”
Quotes equals quality?
“Like peer reviews, but for other reasons, citations suffer fromseveral problems. One of them is that they are related to the orderin which editors arrange the sequence of papers in each issue of ajournal. Research by Smart and Waldfogel (1996), Ayres and Vars(2000), Pinkowitz (2000), and Hudson (2007) finds that leadingarticles – those at the front of the journal – get more cites thanothers. This is tested by running regressions of thenumber of cites on the order in which the paper is placed and onsome control variables.”
“Readers thus seem to believe that the editors of journals aresmart enough to pick the ‘best’ paper ready for the coming issueand choose it as a leading paper. They also believe that the papereditors find to be the best is actually the best.”
Editorial quirk
“In recent work with co-authors (Coupé et al. 2010), I run ananalysis that compares the number of cites conditional on ordering,in two types of publication strategies: random versus selectivelyordered ranking of papers. The European EconomicReview (EER) provides a natural experiment due to an editorialquirk.”
“Between 1975 and 1997, the initial of the first author’ssurname was used to order papers in some issues; in others it wasnot so. As long as we are ready to accept that thealphabetical order is random, in the sense that on average itcannot help separate good and bad papers, this can be considered anatural experiment. This allows us to untangle whether leadingpapers are more cited because they lead or because they are ofhigher quality.”
First is better
“If in alphabetically ordered issues, leading papers also getmore cites than others, then one can wonder whether editors reallyhave a good guess at quality when they use their judgment inordering. If this were the case, leading papers are more citedbecause they are leading (and readers expect them to be better) andnot because they are of better quality.”
“To check for consistency, we also compare this with cites topapers in American Economic Review (AER), where, exceptby chance, the order is never alphabetical.”
“Our results show:
- Leading papers get marginally more cites in all three types ofjournals (EER alphabeticallyordered, EER non-alphabetically ordered,and AER).
- As expected, the effect for AER is much larger thanfor EER.”
“But the difference in the mean number of citesbetween AER and EER papers is not very large (5vs 2 cites). Moreover, for EER the difference in themarginal effect on citations of the first paper is not verydifferent for alphabetical and non-alphabetical issues (1.9 v.2.8), though a likelihood ratio test shows that the difference isstatistically significantly different from zero.”
Longer also better
“This suggests that the lead article when editors exercisediscretion is of better quality, but citation numbers overstate howmuch better it is. Based on the estimates, two thirds of the effect(1.9/2.8) is the result of going first, while one third only can beattributed to better quality. Note that while there is nodifference between first and second paper in AER, forEER cites decrease after the first paper.”
“Long papers are more cited than short ones, and notes areusually less cited (for AER the difference is quitelarge). The sequence of annual dummies that represent the year ofpublication, and thus the age of the paper in 2000, pick upcoefficients that are declining in the case of AER: recentpapers get less (cumulated) cites. The coefficients show noparticular trend for EER. One possible reason may be that thenatural decrease of cites for more recent papers is compensated bymore cites due to increasing average quality of EER overtime.”
More self-citations
“The ordering by the initial of the name may not be entirelyrandom, since, in economics, names usually appear in alphabeticalorder. It is thus possible that lead papers in alphabeticallyordered issues are more likely to be co-authored. To the extentthat such papers get more cites, either because they are of betterquality or because of more self-citations, the lead article effectmay simply capture the influence of a larger number of co-authors.This was controlled by including the number of authors as avariable. Its effect is positive and highly significant in the caseof discretionary ordering, both in EER and AER, butinsignificant in alphabetical issues. More importantly, however,the inclusion of this variable, even when highly significant, didnot change the sign and significance of the main variable ofinterest. This thus suggests that the estimated effect is purely a’lead article effect’.”
“Objective methods such as those based on cites are not perfectand should therefore be used with care, or corrected sinceapproximately two-thirds of the additional cites that leadingpapers get seem to be due to the effect of going first, while onlyone-third can be considered a genuine quality effect of theeditors’ discretionary choice. Hence, given that most editors rankarticles on the basis of their personal quality assessment, evenobjective citation counts have an important subjectivecomponent.”
Academic conservatism
“In addition, the fact that leading articles get more cites justbecause they are leading may be costly for young scientists, sincewell-established (and highly cited) scientists may get more citesthan what they truly deserve: their reputation makes it more likelyfor their articles to be lead articles. This practice may result inintensifying the emergence of ‘superstars’, help conservatism andeven crowd out some good articles by younger scientists who do notget properly cited.
The appearance of new electronic journals, as well as the factthat old-time paper journals become electronic, may induce changesin these patterns. Scientists are now becoming used to downloadingindividual papers and have, in general, no access to the issues ofa journal (though the journal still exists, even if virtually, andpapers are ordered in each issue). But the fact that paper copiesdo not lie on the desk of a scientist will certainly have aninfluence on citations in the future.”
(Author’s note: Our paper was published as a leading paper inissue 61(1) of Oxford Economic Papers. Did the editor want tomake a joke? Or did he think it was better than the other paperspublished in the same issue? If I were you, I wouldn’t believehim.)
