Abstract
This study explores the extent to which bibliometric indicators based on counts of highly-cited documents could be affected by the choice of data source. The initial hypothesis is that databases that rely on journal selection criteria for their document coverage may not necessarily provide an accurate representation of highly-cited documents across all subject areas, while inclusive databases, which give each document the chance to stand on its own merits, might be better suited to identify highly-cited documents. To test this hypothesis, an analysis of 2515 highly-cited documents published in 2006 that Google Scholar displays in its Classic Papers product is carried out at the level of broad subject categories, checking whether these documents are also covered in Web of Science and Scopus, and whether the citation counts offered by the different sources are similar. The results show that a large fraction of highly-cited documents in the Social Sciences and Humanities (8.6–28.2%) are invisible to Web of Science and Scopus. In the Natural, Life, and Health Sciences the proportion of missing highly-cited documents in Web of Science and Scopus is much lower. Furthermore, in all areas, Spearman correlation coefficients of citation counts in Google Scholar, as compared to Web of Science and Scopus citation counts, are remarkably strong (.83–.99). The main conclusion is that the data about highly-cited documents available in the inclusive database Google Scholar does indeed reveal significant coverage deficiencies in Web of Science and Scopus in several areas of research. Therefore, using these selective databases to compute bibliometric indicators based on counts of highly-cited documents might produce biased assessments in poorly covered areas.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Supplementary material to book chapter containing summary tables already available at: https://doi.org/10.17605/OSF.IO/PQR53.
References
Acharya, A. (2015). What happens when your library is worldwide and all articles are easy to find? Retrieved June 19, 2018, from https://youtu.be/S-f9MjQjLsk?t=7m9s.
Acharya, A., Verstak, A., Suzuki, H., Henderson, S., Iakhiaev, M., Lin, C. C. Y., & Shetty, N. (2014). Rise of the rest: The growing impact of non-elite journals. Retrieved June 19, 2018, from http://arxiv.org/abs/1410.2217.
Archambault, É., Vignola-Gagné, É., Côté, G., Larivière, V., & Gingrasb, Y. (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases. Scientometrics, 68(3), 329–342. https://doi.org/10.1007/s11192-006-0115-z.
Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. https://doi.org/10.1007/s11192-008-0216-y.
Bar-Ilan, J. (2010). Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar. Scientometrics, 82(3), 495–506. https://doi.org/10.1007/s11192-010-0185-9.
Bornmann, L., & Leydesdorff, L. (2018). Count highly-cited papers instead of papers with h citations: Use normalized citation counts and compare “like with like”! Scientometrics, 115(2), 1119–1123. https://doi.org/10.1007/s11192-018-2682-1.
Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509. https://doi.org/10.1007/s11192-013-1161-y.
Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H.-D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. https://doi.org/10.1016/j.joi.2008.11.001.
Chavarro, D., Ràfols, I., & Tang, P. (2018). To what extent is inclusion in the Web of Science an indicator of journal ‘quality’? Research Evaluation, 27(2), 106–118. https://doi.org/10.1093/reseval/rvy001.
Clarivate Analytics. (2017). Emerging Sources Citation Index Backfile (2005–2014). Retrieved June 19, 2018, from https://clarivate.com/wp-content/uploads/2017/10/M255-Crv_SAR_ESCI-infographic-FA.pdf.
De Solla Price, D. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306. https://doi.org/10.1002/asi.4630270505.
de Winter, J. C. F., Zadpoor, A. A., & Dodou, D. (2013). The expansion of Google Scholar versus Web of Science: A longitudinal study. Scientometrics, 98(2), 1547–1565. https://doi.org/10.1007/s11192-013-1089-2.
Delgado López-Cózar, E., Martín-Martín, A., & Orduna-Malea, E. (2017). Classic papers: déjà vu, a step further in the bibliometric exploitation of Google Scholar (EC3’s Working Papers No. 24). Retrieved June 19, 2018, from https://arxiv.org/abs/1706.09258.
Delgado López-Cózar, E., Orduna-Malea, E., & Martín-Martín, A. (2018). Google Scholar as a data source for research assessment. In W. Glaenzel, H. Moed, U. Schmoch & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Berlin: Springer.
Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446–454. https://doi.org/10.1002/asi.23056.
Else, H. (2018). How I scraped data from Google Scholar. Nature. https://doi.org/10.1038/d41586-018-04190-5.
Frandsen, T. F., & Nicolaisen, J. (2008). Intradisciplinary differences in database coverage and the consequences for bibliometric research. Journal of the American Society for Information Science and Technology, 59(10), 1570–1581. https://doi.org/10.1002/asi.20817.
García-Pérez, M. A. (2010). Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of h indices in Psychology. Journal of the American Society for Information Science and Technology, 61(10), 2070–2085. https://doi.org/10.1002/asi.21372.
Halevi, G., Moed, H., & Bar-Ilan, J. (2017). Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature. Journal of Informetrics, 11(3), 823–834. https://doi.org/10.1016/J.JOI.2017.06.005.
Harzing, A.-W. (2013). A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel prize winners. Scientometrics, 94(3), 1057–1075. https://doi.org/10.1007/s11192-012-0777-7.
Harzing, A.-W. (2016). Sacrifice a little accuracy for a lot more comprehensive coverage. Retrieved June 19, 2018, from https://harzing.com/blog/2016/08/sacrifice-a-little-accuracy-for-a-lot-more-comprehensive-coverage.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a.
Kousha, K., & Thelwall, M. (2008). Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines. Scientometrics, 74(2), 273–294. https://doi.org/10.1007/s11192-008-0217-x.
Levine-Clark, M., & Gil, E. L. (2008). A comparative citation analysis of Web of Science, Scopus, and Google Scholar. Journal of Business & Finance Librarianship, 14(1), 32–46. https://doi.org/10.1080/08963560802176348.
Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables on citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381. https://doi.org/10.1002/asi.21534.
Martín-Martín, A., Costas, R., van Leeuwen, T., & Delgado López-Cózar, E. (2018). Evidence of Open Access of scientific publications in Google Scholar: A large-scale analysis. https://doi.org/10.17605/osf.io/k54uv.
Martín-Martín, A., Orduna-Malea, E., Ayllón, J. M., & Delgado López-Cózar, E. (2016). A two-sided academic landscape: snapshot of highly-cited documents in Google Scholar (1950–2013). Revista Española de Documentacion Cientifica, 39(4), e149. https://doi.org/10.3989/redc.2016.4.1405.
Martín-Martín, A., Orduña-Malea, E., Ayllón, J. M., & Delgado-López-Cózar, E. (2014). Does Google Scholar contain all highly cited documents (1950–2013)? (EC3 Working Papers No. 19). Retrieved June 19, 2018, from http://arxiv.org/abs/1410.8464.
Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018). Data and code for: Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: A multidisciplinary comparison. https://doi.org/10.17605/OSF.IO/DNQZK.
Martin-Martin, A., Orduna-Malea, E., Harzing, A.-W., & Delgado López-Cózar, E. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163. https://doi.org/10.1016/j.joi.2016.11.008.
Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125. https://doi.org/10.1002/asi.20677.
Mingers, J., & Lipitakis, E. A. E. C. G. (2010). Counting the citations: A comparison of Web of Science and Google Scholar in the field of business and management. Scientometrics, 85(2), 613–625. https://doi.org/10.1007/s11192-010-0270-0.
Moed, H. F., Bar-Ilan, J., & Halevi, G. (2016). A new methodology for comparing Google Scholar and Scopus. Journal of Informetrics, 10(2), 533–551. https://doi.org/10.1016/j.joi.2016.04.017.
Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. https://doi.org/10.1007/s11192-015-1765-5.
Orduna-Malea, E., Ayllón, J. M., Martín-Martín, A., & Delgado López-Cózar, E. (2015). Methods for estimating the size of Google Scholar. Scientometrics, 104(3), 931–949. https://doi.org/10.1007/s11192-015-1614-6.
Orduña-Malea, E., Martín-Martín, A., Ayllón, J. M., & Delgado López-Cózar, E. (2016). La revolución Google Scholar: Destapando la caja de Pandora académica. Granada: Universidad de Granada.
Orduna-Malea, E., Martín-Martín, A., & Delgado López-Cózar, E. (2017). Google Scholar as a source for scholarly evaluation: A bibliographic review of database errors. Revista Española de Documentación Científica, 40(4), e185. https://doi.org/10.3989/redc.2017.4.1500.
Rahimi, S., & Chandrakumar, V. (2014). A comparison of citation coverage of traditional and web citation databases in medical science. Malaysian Journal of Library and Information Science, 19(3), 1–11. Retrieved June 19, 2018, from http://jice.um.edu.my/index.php/MJLIS/article/view/1779.
Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2.
Thelwall, M. (2017). Three practical field normalised alternative indicator formulae for research evaluation. Journal of Informetrics, 11(1), 128–151. https://doi.org/10.1016/j.joi.2016.12.002.
Thelwall, M., & Fairclough, R. (2017). The accuracy of confidence intervals for field normalised indicators. Journal of Informetrics, 11(2), 530–540. https://doi.org/10.1016/j.joi.2017.03.004.
van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346. https://doi.org/10.1023/A:1010549719484.
Vine, R. (2006). Google Scholar. Journal of the Medical Library Association, 94(1), 97. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1324783/.
Acknowledgements
Alberto Martín-Martín enjoys a four-year doctoral fellowship (FPU2013/05863) granted by the Ministerio de Educación, Cultura, y Deportes (Spain).
Author information
Authors and Affiliations
Corresponding author
Appendix: top 5 most common subject categories assigned by Web of Science and Scopus to highly-cited documents in Google Scholar, by Google Scholar broad subject categories
Appendix: top 5 most common subject categories assigned by Web of Science and Scopus to highly-cited documents in Google Scholar, by Google Scholar broad subject categories
Google Scholar category: Humanities, Literature & Arts | Google Scholar category: Social Sciences | ||
|---|---|---|---|
Web of Science categories (176 docs.) | Scopus categories (203 docs.) | Web of Science categories (421 docs.) | Scopus categories (466 docs.) |
Area Studies (24) Linguistics (21) Psychology (18) Literature (17) Social Sciences—Other Topics (16) | Arts and Humanities (138) Social Sciences (127) Psychology (17) Economics, Econometrics and Finance (11) Medicine (7) | Psychology (58) Education & Educational Research (57) Business & Economics (56) Government & Law (48) Social Sciences—Other Topics (32) | Social Sciences (285) Arts and Humanities (97) Medicine (76) Psychology (69) Economics, Econometrics and Finance (49) |
Google Scholar category: Business, Economics & Management | Google Scholar category: Engineering & Computer Science | ||
|---|---|---|---|
Web of Science categories (141 docs.) | Scopus categories (146 docs.) | Web of Science categories (504 docs.) | Scopus categories (556 docs.) |
Business & Economics (113) Social Sciences—Other Topics (20) Public Administration (12) Environmental Sciences & Ecology (9) Science & Technology—Other Topics (6) | Business, Management and Accounting (87) Economics, Econometrics and Finance (70) Social Sciences (36) Arts and Humanities (12) Decision Sciences (10) | Engineering (217) Computer Science (145) Materials Science (56) Chemistry (52) Science & Technology—Other Topics (44) | Engineering (223) Computer Science (158) Materials Science (72) Chemical Engineering (65) Social Sciences (61) |
Google Scholar category: Physics & Mathematics | Google Scholar category: Health & Medical Sciences | ||
|---|---|---|---|
Web of Science categories (225 docs.) | Scopus categories (226 docs.) | Web of Science categories (661 docs.) | Scopus categories (678 docs.) |
Physics (74) Mathematics (73) Science & Technology—Other Topics (31) Engineering (21) Mechanics (17) | Physics and Astronomy (97) Mathematics (89) General (31) Engineering (27) Computer Science (25) | General & Internal Medicine (170) Science & Technology—Other Topics (80) Surgery (53) Neurosciences & Neurology (36) Psychology (24) | Medicine (482) General (80) Biochemistry, Genetics and Molecular Biology (73) Social Sciences (32) Nursing (32) |
Google Scholar category: Life Sciences & Earth Sciences | Google Scholar category: Chemical & Material Sciences | ||
|---|---|---|---|
Web of Science categories (378 docs.) | Scopus categories (378 docs.) | Web of Science categories (170 docs.) | Scopus categories (170 docs.) |
Science & Technology—Other Topics (122) Environmental Sciences & Ecology (51) Biochemistry & Molecular Biology (48) Agriculture (37) Cell Biology (27) | Agricultural and Biological Sciences (122) General (118) Biochemistry, Genetics and Molecular Biology (89) Environmental Science (61) Medicine (40) | Chemistry (75) Science & Technology—Other Topics (34) Materials Science (31) Biochemistry & Molecular Biology (19) Physics (18) | Chemistry (85) Biochemistry, Genetics and Molecular Biology (53) Chemical Engineering (48) Materials Science (40) General (29) |
Rights and permissions
About this article
Cite this article
Martín-Martín, A., Orduna-Malea, E. & Delgado López-Cózar, E. Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison. Scientometrics 116, 2175–2188 (2018). https://doi.org/10.1007/s11192-018-2820-9
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-018-2820-9