Synthesizing academic research about innovation, science, and creativity.
Do Academic Citations Measure the Impact of New Ideas?
A huge quantity of academic research that seeks to understand how science works relies on citation counts to measure the value of knowledge created by scientists. This measure of scientific impact is so deeply embedded in the literature that it's absolutely crucial to know if it’s reliable. So today I want to look at a few recent articles that look into this foundational question: are citation counts a good measure of the value of scientific contributions?
This podcast is an audio read through of the (initial draft of the) post Do Academic Citations Measure the Impact of New Ideas?, originally published on New Things Under the Sun.
Teplitsky, Misha, Eamon Duede, Michael Menietti, and Karim R. Lakhani. 2022. How Status of Research Papers Affects the Way They are Read and Cited. Research Policy 51(4): 104484. https://doi.org/10.1016/j.respol.2022.104484
Gerrish, Sean M., and David M. Blei. 2010. A Language-based Approach to Measuring Scholarly Impact. Proceedings of the 26th International Conference on Machine Learning: 375-382. http://www.cs.columbia.edu/~blei/papers/GerrishBlei2010.pdf
Gerow, Aaron, Yuenig Hu, Jordan Boyd-Graber, and James Evans. 2018. Measuring Discursive Influence Across Scholarship. Proceedings of the National Academy of Science 115(13): 3308-3313. https://doi.org/10.1073/pnas.1719792115
Poege, Felix, Dietmar Harhoff, Fabian Guesser, and Stefano Baruffaldi. 2019. Science Quality and the Value of Inventions. Science Advances 5(12). https://doi.org/10.1126/sciadv.aay7323
Yin, Yian, Yuxiao Dong, Kuansan Wang, Dashun Wang, and Benjamin Jones. 2021. Science as a Public Good: Public Use and Funding of Science. NBER Working Paper 28748. https://doi.org/10.3386/w28748
Card, David, and Stefano DellaVigna. 2020. What do Editors Maximize? Evidence from Four Economics Journals. The Review of Economics and Statistics 102(1): 195-217. https://doi.org/10.1162/rest_a_00839
Tahamtan, Iman, and Lutz Bornmann. 2019. What do Citation Counts Measure? An Updated Review of Studies on Citations in Scientific Documents Published Between 2006 and 2018. Scientometrics 121: 1635-1684. https://doi.org/10.1007/s11192-019-03243-4
Kousha, Kayvan, and Mike Thelwell. 2016. Are Wikipedia Citations Important Evidence of the Impact of Scholarly Articles and Books? Journal of the Association for Information Science and Technology 68(3): 762-779. https://doi.org/10.1002/asi.23694
How common is independent discovery?
An old divide in the study of innovation is whether ideas come primarily from individual/group creativity, or whether they are “in the air”, so that anyone with the right set of background knowledge will be able to see them. In this episode, I look at how much redundancy there is in innovation: if the discoverer of some idea had failed to find it, would someone else have figured it out later?
This podcast is an audio read through of the (initial draft of the) post How common is independent discovery?, originally published on New Things Under the Sun.
Ogburn, William F., and Dorothy Thomas. 1922. Are Inventions Inevitable? A Note on Social Evolution. Political Science Quarterly 37(1): 83-98. https://www.jstor.org/stable/2142320
Haagstrom, Warren O. 1974. Competition in Science. American Sociological Review 39(1): 1-18. https://doi.org/10.2307/2094272
Hill, Ryan, and Carolyn Stein. 2020. Scooped! Estimating Rewards for Priority in Science. Working Paper.
Painter, Deryc T., Frank van der Wouden, Manfred D. Laubichler, and Hyejin Youn. 2020. Quantifying simultaneous innovations in evolutionary medicine. Theory in Biosciences 139: 319-335. https://doi.org/10.1007/s12064-020-00333-3
Bikard, Michaël. 2020. Idea Twins: Simultaneous discoveries as a research tool. Strategic Management Journal 41(8): 1528-1543. https://doi.org/10.1002/smj.3162
Ganguli, Ina, Jeffrey Lin, and Nicholas Reynolds. 2020. The Paper Trail of Knowledge Spillovers: Evidence from Patent Interferences. American Economic Journal: Applied Economics 12(2): 278-302. https://doi.org/10.1257/app.20180017
Lück, Sonja, Benjamin Balmier, Florian Seliger, and Lee Fleming. 2020. Early Disclosure of Invention and Reduced Duplication: An Empirical Test. Management Science 66(6): 2677-2685. https://doi.org/10.1287/mnsc.2019.3521
Iaria, Alessandro, Carlo Schwarz, and Fabian Waldinger. 2018. Frontier Knowledge and Scientific Production: Evidence from the Collapse of International Science. Quarterly Journal of Economics: 927-991. https://doi.org/10.1093/qje/qjx046
Borjas, George J., and Kirk B. Doran. 2012. The Collapse of the Soviet Union and the Productivity of American Mathematicians. The Quarterly Journal of Economics 127(3): 1143-1203. https://doi.org/10.1093/qje/qjs015
Hill, Ryan, and Carolyn Stein. 2021. Race to the bottom: competition and quality in science. Working paper.
Cotropia, Christopher Anthony, and David L. Schwartz. 2018. Patents Used in Patent Office Rejections as Indicators of Value. SSRN Working Paper https://dx.doi.org/10.2139/ssrn.3274995
Science is getting harder
A basket of indicators all seem to document a similar trend. Even as the number of scientists and publications rises substantially, we do not appear to be seeing a concomitant rise in new discoveries that supplant older ones. Science is getting harder.
This podcast is an audio read through of the (initial draft of the) post Science is getting harder, published on New Things Under the Sun.
Bloom, Nicholas, Charles I. Jones, John Van Reenen, and Michael Webb. 2020. Are Ideas Getting Harder to Find? American Economics Review 110(4): 1104-1144. https://doi.org/10.1257/aer.20180338
Wang, Dashun and Albert-László Barabási. 2021. The Science of Science. Cambridge: Cambridge University Press. https://doi.org/10.1017/9781108610834
Li, Jichao, Yian Yin, Santo Fortunato, and Dashun Wang. 2019. A dataset of publication records for Nobel Laureates. Scientific Data 6: 33. https://doi.org/10.1038/s41597-019-0033-6
Collison, Patrick and Michael Nielsen. 2018. Science is Getting Less Bang for Its Buck. The Atlantic.
Chu, Johan S.G. and James A. Evans. 2021. Slowed canonical progress in large fields of science. PNAS 118(41): e2021636118. https://doi.org/10.1073/pnas.2021636118
Milojević, Staša. 2015. Quantifying the cognitive extent of science. Journal of Informetrics 9(4): 962-973. https://doi.org/10.1016/j.joi.2015.10.005
Carayol, Nicolas, Agenor Lahatte, and Oscar Llopis. 2019. The Right Job and the Job Right: Novelty, Impact and Journal Stratification in Science. SSRN working paper. http://dx.doi.org/10.2139/ssrn.3347326
Larivière, Vincent, Éric Archambault, & Yves Gingras. 2007. Long-term patterns in the aging of the scientific literature, 1900–2004. Proceedings of ISSI 2007, ed. Daniel Torres-Salinas and Henk F. Moed. https://www.issi-society.org/publications/issi-conference-proceedings/proceedings-of-issi-2007/
Cui, Haochuan, Lingfei Wu, and James A. Evans. 2022. Aging scientists and slowed advance. arXiv 2202.04044. https://doi.org/10.48550/arXiv.2202.04044
Marx, Matt, and Aaron Fuegi. Reliance on Science: Worldwide Front-Page Patent Citations to Scientific Articles. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3331686
When Extreme Necessity is the Mother of Invention
We all know the proverb “Necessity is the mother of invention.” This proverb is overly simplistic, but it gets at something true. One place you can see this really clearly is in global crises, which vividly illustrate the linkage between need and innovation, without the need for any fancy statistical techniques.
Let’s look at three examples.
This is an audio read through of the (initial version of) When Extreme Necessity is the Mother of Invention, published on New Things Under the Sun.
Agarwal, Richer, and Patrick Gaule. 2022. What Drives Innovation? Lessons from COVID-19 R&D. Journal of Health Economics 82: 102591. https://doi.org/10.1016/j.jhealeco.2022.102591
Bloom, Nicholas, Steven J. Davis, and Yulia Zhestkova. 2021. COVID-19 Shifted Patent Applications towards Technologies That Support Working from Home. AEA Papers and Proceedings 111: 263-266. https://doi.org/10.1257/pandp.20211057
Hassler, John, Per Krusell, and Conny Olovsson. 2021. Directed Technical Change as a Response to Natural Resource Scarcity. Journal of Political Economy 129(11): 3039-3072. https://doi.org/10.1086/715849
Ilzetzki, Ethan. 2022. Learning by Necessity: Government Demand, Capacity Constraints, and Productivity Growth. Working paper.
Gross, Daniel P., and Bhaven N. Sampat. 2020. Organizing Crisis Innovation: Lessons from World War II. NBER Working Paper 27909. http://doi.org/10.3386/w27909
Steering Science with Prizes
New scientific research topics can sometimes face a chicken-and-egg problem. Professional success requires a critical mass of scholars to be active in a field, so that they can serve as open-minded peer reviewers and can validate (or at least cite!) new discoveries. Without that critical mass,undefined working on a new topic topic might be professionally risky. But if everyone thinks this way, then how do new research topics emerge; how do groups of people pick which topic to focus on?
One way is via coordinating mechanisms; a small number of universally recognized markers of promising research topics. This podcast looks at some evidence about how well prizes and other honors work at helping steer researchers towards specific research topics.
This is an audio read through of the (initial version of) "Steering Science with Prizes", published on New Things Under the Sun.
Azoulay, Pierre, Toby Stuart, and Yanbo Wang. 2014. Matthew: Effect or Fable? Management Science 60(1): 92-109. https://doi.org/10.1287/mnsc.2013.1755
Reschke, Brian P., Pierre Azoulay, and Toby E. Stuart. 2018. Status Spillovers: The Effect of Status-conferring Prizes on the Allocation of Attention. Administrative Science Quarterly 63(4): 819-847. https://doi.org/10.1177/0001839217731997
Jin, Ching, Yifang Ma and Brian Uzzi. 2021. Scientific prizes and the extraordinary growth of scientific topics. Nature Communications 12: 5619. https://doi.org/10.1038/s41467-021-25712-2
Azoulay, Pierre J., Michael Wahlen, and Ezra W. Zuckerman Sivan. 2019. Death of the Salesman but Not the Sales Force: How Interested Promotion Skews Scientific Valuation. American Journal of Sociology 125(3): 786-845. https://doi.org/10.1086/706800
Azoulay, Pierre, Christian Fons-Rosen, and Joshua S. Graff Zivin. 2019. Does Science Advance One Funeral at a Time? American Economic Review 109(8): 2889-2920. https://doi.org/10.1257/aer.20161574
Progress in Programming as Evolution
Evolution via natural selection is a really good explanation for how we gradually got successively more complex biological organisms. Perhaps unsurprisingly, there have long been efforts to apply the same general mechanism to the development of ever more complex technologies. One domain where this has been studied a bit is in computer programming. Let’s take a look at that literature to see how well the framework of biological evolution maps to (one form of) technological progress.
This podcast is an audio read through of the (initial version of the) article "Progress in Programming as Evolution", published on New Things Under the Sun.
Arthur, W. Brian, and Wolfgang Polak. 2006. The evolution of technology within a simple computer model. Complexity11(5): 23-31. https://doi.org/10.1002/cplx.20130
Miu, Elena, Ned Gulley, Kevin N. Laland, and Luke Rendell. 2018. Innovation and cumulative culture through tweaks and leaps in online programming contests. Nature Communications 9: 2321. https://doi.org/10.1038/s41467-018-04494-0
Miu, Elena, Ned Gulley, Kevin N. Laland, and Luke Rendell. 2020. Flexible learning, rather than inveterate innovation or copying, drives cumulative knowledge gain. Science Advances 6(23): eaaz0286. DOI: 10.1126/sciadv.aaz0286
Valverde, Sergi and Ricard V. Solé. 2015. Punctuated equilibrium in the large-scale evolution of programming languages. Journal of the Royal Society Interface 12: 20150249. http://doi.org/10.1098/rsif.2015.0249