Monday, May 15, 2017

Vast literatures as mud moats

I don't know why academic literatures are so often referred to as "vast" (the phrase goes back well over a century), but it seems like no matter what topic you talk about, someone is always popping up to inform you that there is a "vast literature" on the topic already. This often serves to shut down debate, because it amounts to a demand that before you talk about something, you need to go read voluminous amounts of what others have already written about it. Since vast literatures take many, many hours to read, this represents a significant demand of time and effort. If the vast literature comprises 40 papers, each of which takes an hour to read, that's one week of full-time work equivalent that people are demanding as a cost of entry just to participate in a debate! So the question is: Is it worth it?

Often, reading the literature seems like an eminently reasonable demand. Suppose I were to think about minimum wage for the first time, knowing nothing about all the research economists have done on the topic. I might very confidently say some very silly things. I would be unaware of the relevant empirical evidence. There would probably be theoretical considerations I hadn't yet considered. Reading the vast literature would make me aware of many of these. In fact, I think the minimum wage debate does suffer from a lack of knowledge of the literature.

But the demand to "go read the vast literature" could also be eminently unreasonable. Just because a lot of papers have been written about something doesn't mean that anyone knows anything about it. There's no law of the Universe stating that a PDF with an abstract and a standard LaTeX font contains any new knowledge, any unique knowledge, or, in fact, any knowledge whatsoever. So the same is true of 100 such PDFs, or 1000. 

There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.

But astrology generally isn't worth talking or thinking about, either. The real question is whether there are interesting, worthwhile topics where reading the vast literature would be counterproductive - in other words, where the vast literature actually contains more misinformation than information.

There are areas where I suspect this might be the case. Let's take the obvious example that everyone loves: Business cycles. Business cycles are obviously something worth talking about and worth knowing about. But suppose you were to go read all the stuff that economists had written about business cycles in the 1960s. A huge amount of it would be subject to the Lucas Critique. Everyone agrees now that a lot of that old stuff, probably most of it, has major flaws. It probably contains some real knowledge, but it contains so much wrong stuff that if you were to read it thinking "This vast literature contains a lot of useful information that I should know," you'd probably come out less informed than you went in. 

Of course, many would say the exact same about the business cycle theory literature that emerged in response to the Lucas Critique and continues to this day. But if so, that just makes my point stronger. The point is, a bunch of smart people can get very big things wrong for a very long period of time, and that period of time may include the present.

I have personally encountered situations where I felt that reading the vast literature didn't improve my knowledge of the real thing that the literature was about. For example, I read a lot of the macro models that came out in the years following the 2008 financial crisis. Obviously, the financial sector is very important for the macroeconomy (as more people should have realized before 2008, but which almost everyone realizes now). But the ways that macro papers have modeled financial frictions are pretty unsatisfying. They are hard to estimate, the mechanisms are often implausible, and I bet that most or all will have glaring inconsistencies with micro data. I could be wrong about this, of course, but I felt like reading this vast literature was setting me on the wrong track. I'm not the only one who feels this way, either.

The next question is: Can a misinformative vast literature be used intentionally as a tactic to win political debates? It seems to me that in principle it could. Suppose you and your friends wanted to push a weak argument for political purposes. You could all write a bunch of papers about it, with abstracts and numbered sections and bibliographies and everything. You could cite each other's papers. If you wanted to, you could even create a journal, and have a peer review system where you give positive reviews to each other's B.S. papers. Voila - a peer-reviewed literature chock full of misinformation.

In practice, I doubt anyone ever does this intentionally. It takes too much coordination and long-term planning. But I wonder if this sometimes happens by accident, due to the evolutionary pressures of the political, intellectual, and academic worlds. The academic world gives people an incentive to write lots of papers. The political world gives people an incentive to use papers to push their arguments. So if there's a fundamentally bad argument that many people embrace for political reasons, there's an incentive for academics (or would-be academics) to contribute to a vast literature that is used to push that bad argument.

And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows.

If you and your buddies have a political argument, a vast literature can help you defend your argument even if it's filled with vague theory, sloppy bad empirics, arguments from authority, and other crap. If someone smart comes along and tries to tell you you're wrong about something, just demand huffily that she go read the vast literature before she presumes to get involved in the debate. Chances are she'll just quit the argument and go home, unwilling to pay the effort cost of wading through dozens of crappy papers. And if she persists in the argument without reading the vast literature, you can just denounce her as uninformed and willfully ignorant. Even if she does decide to pay the cost and read the crappy vast literature, you have extra time to make your arguments while she's so occupied. And you can also bog her down in arguments over the minute details of this or that crappy paper while you continue to advance your overall thesis to the masses.

So when I want to talk and think and argue about an issue, and someone says "How about you go read the vast literature on this topic first?", I'm presented with a dilemma. On one hand, reading the vast literature might in fact improve my knowledge. On the other hand, it might be a waste of time. And even worse, it might be a trap - I might be charging headlong into a rhetoritician's mud moat. But choosing not to read the vast literature keeps me vulnerable to charges of ignorance. And I'll never really be able to dismiss those charges.

My solution to this problem is what I call the Two Paper Rule. If you want me to read the vast literature, cite me two papers that are exemplars and paragons of that literature. Foundational papers, key recent innovations - whatever you like (but no review papers or summaries). Just two. I will read them. 

If these two papers are full of mistakes and bad reasoning, I will feel free to skip the rest of the vast literature. Because if that's the best you can do, I've seen enough.

If these two papers contain little or no original work, and merely link to other papers, I will also feel free to skip the rest of the vast literature. Because you could have just referred me to the papers cited, instead of making me go through an extra layer, I will assume your vast literature is likely to be a mud moat.

And if you can't cite two papers that serve as paragons or exemplars of the vast literature, it means that the knowledge contained in that vast literature must be very diffuse and sparse. Which means it has a high likelihood of being a mud moat.

The Two Paper Rule is therefore an effective counter to the mud moat defense. Castle defenders will of course protest "But he only read two papers, and now he thinks he knows everything!". But that protest will ring hollow, because if you can show bystanders why the two exemplar papers are bad, few bystanders will expect you to read further.

If it proves to be as effective as I think, the Two Paper Rule, if widely implemented, could make for much more productive public debate. The mud moat defense would be almost entirely neutralized, dramatically reducing the incentive for the production of vast low-quality literatures for political ends. It could allow educated outsiders and smart laypeople access to debates previously dominated by vested insiders. In other words, it could shine the light of reason on a lot of dark, unexplored corners of the intellectual universe.


Some people seem to misunderstand the purpose of the Two Paper Rule. The Two Paper Rule is not about summarizing the literature's findings - for that, you'd want a survey paper or meta-analysis. It's about evaluating the quality of the literature's methodology.

Sometimes a lit review will reveal pervasive methodological weakness - for example, if a literature is mostly just a bunch of correlation studies with no attention to causal effects. But often, it won't. For example, if the literature has a lot of mathematical theory in it, a lit review will generally contain at most one stripped-down partial model. But that doesn't give you nearly as much info about the quality of the fully specified models as you'll get from looking at one or two flagship theory papers. Or suppose a literature consists mostly of literary theorizing; the quality of the best papers will depend on the clarity of the writing, which a lit review is unlikely to be able to reproduce. Sometimes, lit reviews simply report results, without paying attention to what turn out to be glaring methodological flaws.

In other words, if you suspect that a literature functions mainly as a mud moat, what you need to assess quickly is not what the literature claims to find, but whether those claims are generally credible. And that is why you need to see the best examples the literature has to offer. Hence the Two Paper Rule.

Meanwhile, Paul Krugman endorses the Two Paper Rule. Tyler Cowen is more skeptical of its universality.  


  1. I would like to see which two papers would be chosen for MMT. It would be extremely helpful for those of us not in the know.

    1. Anonymous11:00 AM

      Since there don't exist two papers that aren't absolute crap in that literature (or even one), you're SOL.

    2. Papers schmapers! Economics isn't a science. Not two papers -
      one video. Stephanie Kelton's Angry Birds. It's on youtube.



    4. Good pics. This paper might deserve a mention also:


  2. One important note about the "vast literature" is that for many people who aren't academics or aren't in college you often have to purchase academic articles online for hefty fees from websites like Jstor or other more specific institutions. So, if they don't have access to a University's database(this is something I'm very grateful for) or aren't someone with money from a research grant trying to even get the actual papers is going to exclude many people from the get go.

    1. Those who appeal to the vast literature can download and post 2 papers (fair use). It can be the responsibility of he (or she) who invokes a literature to call it from the vasty deep (and make it come by paying if necessary).

    2. This isn't a very solid argument. It's very easy these days to get almost any paper just y posting some request on facebook, some forum, etc. Anyone with basic internet literacy can find out how to do this.

  3. If from the "vast literature" the demand is made to read two books (because your torturer is wise to your two paper trick), demand back a single review of each book, blurbs not allowed.

    1. Books aren't papers ;-)

    2. Anonymous9:29 PM

      Well, that depends a lot on the discipline. In (and around) the humanities, papers can often be more derivative than books. Plus, if it's not just literature on a topic, but a theory (or a field containing several), and you're not familiar with any of it, you need the founding books of the theory/theories to follow the argument in later papers. And no, reviews of a book don't give you a taste of the writing, and rarely of the method.

  4. Regarding your tweets wondering why people take self-esteem from military success of their group: you should read Jack Donovan, author of The Way of Men, on this.

    Suppose you were the youngest in a big family in a rough neighborhood 100 years ago--a place where people fought often and being tough mattered. When you came of age you might find that your father and older brothers had made a reputation: don't mess with the Smith boys. That's of great value to you. You have to fight less because of the success of those who came before. But if you show yourself weak or cowardly, you could undo that reputation, making things harder not only for yourself but for the rest of your family.

    Why do you think street-gang members identify with the West Side or whatever? They want to send the message, Don't mess with people from the West Side. A strong reputation is money, but like inherited wealth imposes a duty not to squander it. Ditto with a tribe--how great for you and your posterity if your tribe--be it Apache or Scotsman--is known as badasses. But what shame for you if you don't hold up the reputation.

    This is known as Honor.

  5. That is the reason for review papers, with references for anyone that wants to probe specifics deeper, preferably of relatively recent vintage. 'Vast literature' just means they know it has been researched, but don't know anything about the results.

    1. Review papers don't tell you if they methods are good though. For that, you need original-work papers.

    2. well OK, but a good survey paper will quickly bring you up to speed, what sort of evidence has been put forward etc. This is especially useful for broad topics with many aspects to consider, where a couple of 'single issue' papers wouldn't get across. If you were interested in foreign aid, for example, I'd say, start here:

      A great survey paper can say something useful about the quality of the methods too.

    3. responding to your update. Fine, if evaluating methodology is your intent, surveys often less useful. But when you're sounding off on a topic, and making a hash of well-worn ideas or neglecting important things, and you get told to "go read the literature" that's not an instruction to evaluate the methodology

    4. The approach I would take is to read a significant invited review paper in a major journal. Identify the most interesting, surprising, or controversial findings. Look at the most significant references given for them. If you want to double check their importance look up their citation numbers and read the most cited ones. It will give you a feel for the subject as well as how it is done.

  6. Intersectionality...

  7. Anonymous8:18 PM

    reminds me of the calls to "engage with" the "extensive" or "vast" literature of trans/ POC critical race theorists in the whole Hypatia thing. But I don't suppose that had anything to do with this :)

  8. New project: a list of the 2 first articles you should read, for each topic.

  9. One option is to look at the RePEc Biblio. Still very incomplete, but volunteers are welcome. The literature is summarize to 10-20 articles, though. And if possible, there are links to gated articles and corresponding accessible working papers.

  10. "The next question is: Can a misinformative vast literature be used intentionally as a tactic to win political debates? It seems to me that in principle it could. Suppose you and your friends wanted to push a weak argument for political purposes. You could all write a bunch of papers about it, with abstracts and numbered sections and bibliographies and everything. You could cite each other's papers. If you wanted to, you could even create a journal, and have a peer review system where you give positive reviews to each other's B.S. papers. Voila - a peer-reviewed literature chock full of misinformation."

    We call it the Mt, Perlin Society.

  11. Unfortunately I have to disagree with your claim that no one creates a body of work composed entirely of misinformation on purpose. As proof, I cite the existence of

    1. How cute, and yet, ridiculous of you to smear

      The only thing the author omits in this excellent essay is to emphatically embrace his weakly presented hypothetical as being unambiguous -- of course there are huge incentives, created by and through government and its entanglements, for academia to become court historians and propagandists.

      Look, 95% of it fails the market test -- it's work that exists because taxpayers coercively fund the crap. This essay speaks perfectly to that phenomenon.

  12. This is a nice addition to the young field of General Bullshit Detection (cf. the course at the U of Washington).

  13. There might be a slight problem using the two paper rule inciting some of the works of Issac Newton given his extensive and interrelated writings on both physics, math and alchemy. It's worth reading Keynes essay on Newton, delivered posthumously by his brother. (Keynes had acquired many of Newton's papers starting in 1936, a fascinating story in its own right

    I would make the point that the two-paper rule, while obviously useful in debate, might well lack greater utility. The value of a paper is its ability to further understanding.Newton's physics an mathematics are of huge value, while his works on alchemy are a historical dead end, conclusively proved wrong, now only of interest in historical terms. Maybe there is a lesson for economics here

  14. This strategy is most definitely used to defend the value of certain academic areas like continental philosophy. If one challenges the academic value/coherence of this area one is repeatedly told one can't evaluate it at all until one fully immerses oneself in it (in particular one is often told that one gets some kind of gestalt value out of reading whole treatises and merely finding the article length pieces one reads to be incoherent gibberish isn't evidence). This creates the unfortunate situation where the only people who are judged to be competent to comment on the academic merit/contribution are those who have invested huge amounts of time and effort in the area and thus have strong incentives to claim it is important.

    1. Agree 100% on this about continental philosophy (although I live on the continent and have a degree in philosophy ;-)

  15. Agree 100% on this.

    Related: it takes months, and sometimes even years to properly engage with an academic literature. It's therefore easy to tell the difference between someone who has assimilated it and can make their own arguments drawing from it vs someone who is just using it as a moat. Unfortunately most people on the internet seem to fit into the latter camp.

    1. The problem is that vast literatures serve as moats also for those who aren't *just* using them as a moats. I think Noah's problem is with people who have sincerely dedicated themselves to adding a bit to vast but worthless literatures. Their objection to critics who have not spent months assimilating the literature is entirely honest and not strategic. But it effectively defends invalid claims.

      In any case, Noah's rule works both with sincere and purely rhetorical appeals to vast literatures.

  16. (I am having trouble posting this: apologies if it ends up with multiple posts).

    Minimum wage
    Book length summary What Does the Minimum Wage Do? (2014)

    2-paper rule: Probably the AER (12/2000) exchange between Neumark & Wascher, and Card & Krueger. At the time, these 2 pairs were the most prominent researchers in this field, and in this exchange they engaged very closely with each other. Since then there have been improvements in technique (e.g, ways to calculate more accurate estimated standard errors and t-statistics, better ways to implement a quasi-experiment) but no substantial breakthroughs. We will probably have to wait 5-10 years for those, when data from the current round of increases up to $12-$15 has become available for analysis. I am hopeful that we will then be able to make statements about the effects with much more confidence than is the case right now.

  17. Anonymous3:01 PM

    This applies equally to molecular biology, genetics, biochemistry
    Usually, a knowledgeable person [ie, has worked in the field for a few years] will tell you, here are the 3 key papers, or just read the papers from person X and Y to get up to speed, or here is the review that actually summarizes it for you

  18. The very real problem you wrestle with here, I think is due to a lack of good methodological standards in economics, and in the social sciences. The rule of thumb here is better than nothing, but a poor substitute for good methodology—for by what standard to you judge that the good papers are good? My take is that what is missing is the demand that both theory and experiment push to find "crucial evidence"—evidence that tentatively confirms one theory while refuting a rival:

  19. The survey paper matter should not be an alternative but an addition. Read the two (or maybe three) key big papers and then look at a survey paper to see how they fit into the "vast literature." That does better than one or the other exclusively.

  20. Anonymous3:03 AM

    What if the buddy comes up with the "humans are irrational" argument in response to that two papers? Hard to imagine the dialogue can peacefully end after this.

  21. I very much like this essay. I entirely agree that vast literatures often act as mud moats and that people take advantage of this (and also that this isn't the result of a conspiracy).

    However, I do not like your example. I do not think that the 1960s macro literature was any more vulnerable to the Lucas critique than the post 1973 literature. I think it is well worth reading (based on having read very few papers in the vast literature). For one thing, it includes the Lucas Critique which is, in fact, due to Marshak.

    two papers (for your rule).

    Samuelson, P.A., and R.M. Solow. 1960. “Analytical Aspects of Anti-Inflation Policy”
    American Economic Review Papers and Proceedings 50(2), pp. 177-94.

    Contains useful thoughts including that the Phillips curve with shift due to shifts in expectations so the short term and long term curves are different. Also that cyclical unemployment can become structural (hysteresis). Not vulnerable to the Lucas critique which it anticipates.

    Tobin Cowles foundation discussion paper 315

    (in which he notes that Keynes described the Lucas Supply Function in The General Theory ..." That Lucas didn't cite Keynes was, from then on, academic misconduct.

    Krugman says Tobin's AEA Presidential address is better, but I haven't read it.

    I do not think you will be inclined to dismiss the vast 1960s macro literature after reading these two papers. OK one from 71 but that's close.

    Notice especially how the Lucas critique was absolutely nothing new. Lucas's innovation was to assert that the solution was to assume rational expectations and optimization of problems which weren't the ones actual people solved. This critique of Lucas is devastating

    I note again I am not the only one who thinks that we have a whole lot to learn from Samuelson, Solow and Tobin

    Note 1960, 1971 and 1972 were all before the rational expectations revolution.

    I think your example of a worthless literature is deeply misguided. Macro wasn't always the mud moat that it has become.

    1. I completely agree, Robert. I happen to think the two paper rule is too narrow, especially for an actually vast lit like "1960s macro." This is why looking at a few more papers might be wise, including especially a survey paper. There are plenty of lits where two papers will simply miss stuff, and that the two papers selected have methodological problems does not mean that the rest do also, as you accurately note regarding this matter of the Lucas Critique, which was indeed first posed by Marschak.

      That said, I do agree with the basic point of Noah's that when someone tries to dismiss an argument by simply referring to a "vast literature" it is completely legitimate and useful to call them on it and at least provide a basic summary of what are the most important points of that vast literature, with or without leading papers or summary/review articles of whatever number.

  22. I know you said 2 papers, but I would like to quote a paragraph from a third "Inflation does give a stimulus, but the stimulus is greatest when the inflation
    starts … If the inflation continues, people get adjusted to it. But when people
    are adjusted to it, when they expect rising prices, the mere occurrence of what
    has been expected is no longer simulating. Nor can the fade-out be prevented
    by accelerating the inflation; for acceleration of inflation can be expected too'"

    This is a valid point.

    Who wrote it ? When was it written ?

    1. Damn, I should have been able to guess that from the style, which is pretty distinctive. But doesn't Old Keynesian economics present a problem for Noah's 2-papers approach? Nearly any macro paper worth reading from around 1950 to 1970 refers to the General Theory, which everyone of that generation was assumed to have read.

    2. I sure wouldn't recognize it from the style. In fact, I hadn't noticed a distinctive style of [the economist whose name I won't type since guessing it is the game].

      I think that good papers can explain the concepts. I think that one doesn't have to have read The General Theory to understand paleo Keynesian papers (I'm not sure since I read many of them only after reading the General theory, but I think it works).

      However, there is indeed a problem. There is a vast literature on quantum field theory. If I suspect it is crap & have read nothing on quantum mechanics, what two papers on quantum field theory should I read.

      It aint paleo Keynesian economics.

      There are fields where people are clearly on to something, but such that you have to study for years to understand current research.

      I will try to modify Noah's proposal so it can fit with quantum field theory. First, one could say the literature is quantum mechanics, note a few reports of elementary experiments and prove that the quantum mechanics have a point. So the 2 papers are chosen from a literature which the critic hasn't read with the choice limited only by the critics ignorance.

    3. But aren't we all supposed to have read the GT? At least be able to quote chapter 12 ans nod knowingly?

      Or, to change the metaphor slightly, professional investment may be likened to those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole; so that each competitor has to pick, not those faces which he himself finds prettiest, but those which he thinks likeliest to catch the fancy of the other competitors, all of whom are looking at the problem from the same point of view. It is not a case of choosing those which, to the best of one’s judgment, are really the prettiest, nor even those which average opinion genuinely thinks the prettiest. We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be. And there are some, I believe, who practise the fourth, fifth and higher degrees.

  23. Anonymous11:59 PM

    "You will recognise yourself from my description. (Or you will recognise others who fit this description).

    "You are probably very smart. You are probably very well-educated -- either formally, or self-educated, and probably both. You spend a lot of time on the internet reading economics blogs and commenting on those blogs. You maybe even have a blog of your own, where you write about economics topics. You are probably politically engaged. You are probably a lefty, but may be a righty, or someone who is not easily categorised on that political spectrum. You probably think of yourself as a critic of economics, or a critic of what you see as orthodox economics. You are probably sympathetic to what you see as heterodox economics.

    "But you have never once read a first year economics textbook.

    "You have probably many times told me, or people like me, that I really really should read something you want me to read.

    "Well now it's my turn.

    "I think you really really should read a first year economics textbook."

    Prof. Nick Rowe wrote that about five years ago (I've kept his italics and bold fonts).[*]

    Note that he is not asking an unnamed online critic to read 40 papers, but to read a full first year economic textbook. You know, those requiring at least an entirel semester of full-time study (at the end of which, a considerable part of the book has not been treated).

    I suspect -- if you explained it to him -- he would quickly understand that he is asking the unnamed reader to pay an entry cost. A considerable one, at that, given an slightly facetious comment he added ("You could do it [read the book, that is] in one day. Maybe two, if you go really slowly and carefully.")

    He didn't think that an excessive or unreasonable request, I suppose. Nor did his commentators:

    "1. I completely agree
    "2. I would love to know if there was a specific comment that inspired this post."

    You don't see many qualifications in Prof. Rowe's post or in his commentators' contributions. Quite to the contrary, he seemed to believe the benefits of the exercise far exceed whatever costs (so do I, to be honest).

    Moreover, I don't remember anyone asking, as you did above: "Can a misinformative vast literature be used intentionally as a tactic to win political debates?"

    I know he didn't have me in mind when he wrote that (I'm neither smart nor well-educated; I do have a good memory, though). If you think it appropriate, I could go there tell him, on your authority, what he must nominate two papers, instead.

    Or maybe you prefer to do it yourself?


    B.L. Zebub

  24. I think a better demand is to read the first 30-50 pages of a textbook on the subject.

    a) it means it's worth having a textbook on, and it's established enough that there's a body of work around it
    b) there should be some kind of "motivation for the field" at the start of it
    c) There's enough pith to the topic that understanding the 30-50 pages of good stuff naturally leads to a rich body of work (in the other 150 pages)
    d) You've got consistent notation, and don't have to mess about flicking from topic to topic

  25. Well, I tried to read a first year economics text a couple of times. But the fairy stories started on page 2 and got wilder as it went, so I gave it up for the real stuff.

  26. "The next question is: Can a misinformative vast literature be used intentionally as a tactic to win political debates? ... In practice, I doubt anyone ever does this intentionally."

    J. Edgar Mihelic and The Contentious Otter beat me to the punch with the real-world examples of Mount Perlin Society and Others include the Kochtopus, the Wedge Strategy and a huge amount of the drug and alternative medicine literature.

    I think that this is a standard public relations methodology. They probably have a term for it.

    1. What an amazing comment. Didn't realize I came upon a mainstream Econ website. Yet the essay wonderfully speaks to everything wrong with mainstream Econ.

      Mises, etc and its ilk at least are funded by honest money. And clearly it's ridiculous to joke that they exist for some kind of political point, because no one "respectable" cites them in discourse. They are much more the gadfly who seeks truth and therefore annoys the soi-disant intelligentsia.

  27. One possible failing of the Two Paper Rule would be due to long tails. It may well be that 99% of the literature is crap, but two papers might be good.

    Another possible failing is if the good papers are only negative or critical. For example, some of the best papers in philosophy are refutations of other philosophers. But relatively few papers in philosophy tend to have real merits outside of showing errors of others. IMHO.

    And I strongly agree that a survey ought to be read as well as the two papers with methodologies. When reading the two papers, we should remember Hume:

    "If we take into our hand any volume; of quotations or ancient personality, for instance; let us ask, Does it contain any abstract reasoning concerning quantity or number? No. Does it contain any experimental reasoning concerning matter of fact and existence? No. Consign it then to the flames: For it can contain nothing but sophistry and illusion."

  28. Noah,

    I just saw your update, which looks directed at my advocating looking at review essays, again, not as a substitute for looking at some top papers, but in addition to. Let me note another reason why, and that is that unless one is dealing with a very narrow literature, looking at just two papers, however selected, may not cover the methodologies used in that literature. You may see two papers that are highly cited but methodologically flawed, while some other papers not so highly cited may well be methodologically superior. Even the examples from old macro that Robert above notes are good for that, with, e.g. Marschak's 1950 paper that introduced what would later be called the Lucas Critique, but in a superior way, not at all making the top couple of papers from that literature in terms of citations or whatever.

    Anyway, good review essays do not when there are different methodologies being used in different papers, although not all do.

    Barkley Rosser

  29. A good example outside of economics is Lysenkoism. It was a Lamarckian theory of evolution that nearly destroyed Russian agriculture, but it produced a vast literature. Marxism, and particularly Soviet communism, argued that new times required new men, and that new men could be made. They did not need to be bred. (Breeding reeked of hereditary aristocracy anyway.) Larmarckian theories that claimed that exposure and training could not only modify organisms, but that those changes could be inherited, were naturally attractive. Odds are you've already run into the Lamarckian stretch yoga theory of giraffe evolution.

    Lysenko championed this theory and started his rise in the Soviet scientific hierarchy in the 1920s. By the 1940s he was in charge of big chunks of Soviet science including all of biology. Like spitting on the sidewalk or being late for work, Darwinian arguments could get you sent to the gulag.

    There is a vast literature full of papers documenting Lamarckian evolution in action. The point might be that exposing plants to more and more extreme temperatures could produce new temperature resistant strains. There was no need for breeding, mutation and selection. Needless to say, this made it much harder for someone trying to actually breed a temperature resistant strain. It could be done, but one had to be politically careful lest the necessary Darwinian theory became visible. Of course, being able to breed a resistant strain wasn't all that important if one was suitably politically correct and one's contribution to the vast literature praised the party and prevailing ideology effusively enough.

    Lynsenko was finally removed in 1953 when Stalin died. I'm not sure of what they did with the vast literature. Presumably it is lurking in some back room somewhere or perhaps has been burned for heat showing that even the most worthless vast literature might still have some value.

    I like your two paper rule. It has a biblical appeal, much like Abraham asking God to spare Sodom if he could find even ten righteous man living there. A field or theory which lacks even two papers that are methodologically solid, is not much of a field.

    1. Kaleberg,

      Lysenko was not removed from power when Stalin died, although his power was weakened. Khrushchev liked him, and it took the removal of NK in 1964 to bring about his removal and public disgrace.

      There are problems with the fixation on two papers. These can be badly selected. They may not represent the literature. As it is, it seems that Noah wants the rule to humiliate or at least expose pompous people spouting off about vast literatures, and I am all for exposing such people. But, indeed, if part of the issue here is actually figuring out whether the literatures are methodologically or otherwise fundamentally flawed, one needs to check a bit further one way or another than some arbitrarily selected supposedly leading two papers, which may in fact misrepresent the literature.

  30. Can a misinformative vast literature be used intentionally as a tactic to win political debates? See the Intelligent Design papers attempting to counter Darwinism.

  31. OK a challenge. If there is a mud moat problem, Smith must present 2 exmples. He has one --astrology. I have demonstrated that his claim about 60s era macro is false by citing two excellent papers (S&S 1960 and one from 1967 whose author hasn't been named plus 1971 and 1972 which are close to the 60s).

    Honest application of the 2 paper rule would force Noah to read those two papers and report whether he is convinced of his mud moat view of 60s macro. The challenge "show me 2 papers" only works if one then reads the 2 papers.

    I challenge Noah Smith to retract his claim or critique one (1) of the 4 papers I cited (if 2 I will have to reveal Prof. 1967).

  32. I am particularly interested in yor comment "The next question is: Can a misinformative vast literature be used intentionally as a tactic to win political debates" because it's being done, adroitly, in the larger "popular" media. Google "FDR" and "depression", and you will find most of the citations claiming that FDR *expanded* the Great Depression. Why so many? They all bounce back to a single study done by a couple of people in California, which paper has been endlessly bounced back and forth (with appropriate meta discussion) among "think tanks" and "news sites" and "discussion boards", all garnering precious Google links - propelling this arcane theory into mainstream discussion. From which we learn, of course, that "government involvement of any sort in the economy, no matter how dire, is always bad." And now I have "vast literature" to back it up. Pfooey!

  33. "In practice, I doubt anyone ever does this intentionally. It takes too much coordination and long-term planning." ...

    Americans for Tax Reform
    Koch family foundations


  34. Am I the only one whose first association here was to the Rebecca Tuvel controversy?

  35. Have just used same tactic with climate "skeptic". After knocking down post after post, finally said, "Give me one post that, if it's wrong, you will accept that global warming is true."

  36. Getting out of the economics swamp
    Comment on Noah Smith on ‘Vast literatures as mud moats’

    You say: “The point is, a bunch of smart people can get very big things wrong for a very long period of time, and that period of time may include the present.”

    True, this happened with Geo-centrism for roughly 1500 years starting with the smart mathematician, astronomer and geographer Ptolemy around 100 AD. It happens again with economics, starting 200+ years ago with the not so smart Adam Smith. As as result it holds for economics in general “the vast literature actually contains more misinformation than information.”

    The situation in economics is special insofar as there are TWO economixes: political economics and theoretical economics. The main differences are: (i) The goal of political economics is to successfully push an agenda, the goal of theoretical economics is to successfully explain how the actual economy works. (ii) In political economics anything goes; in theoretical economics scientific standards are observed.

    Political economics as a whole is scientifically worthless. The proper place for this “vast literature” is the waste basket. Political economics is easily recognizable by its swampiness. Swampiness is what what Popper called an immunizing stratagem because: “Another thing I must point out is that you cannot prove a vague theory wrong.” (Feynman) Therefore, the political economist is mainly occupied with producing a methodological smoke screen that consists of vague concepts, anything goes, pluralism of false theories, alleged complexity, ontological uncertainty, unreliable data, the faux humility of ‘I know that I know nothing’, and the post-modern replacement of true theory by an emotional narrative.#1

    The beauty of the swamp ‘where nothing is clear and everything is possible’ (Keynes) is that logical and empirical failure is inconsequential.#2 What you call the mud moat has indeed saved the life of the representative economist since 200+ years.

    The other part of the “crappy vast literature” consists of methodological blunder, i.e. stuff that does not satisfy the scientific criteria of material and formal consistency. This holds for everything that comes under the banner of Walrasianism, Keynesianism, Marxianism, Austrianism and Pluralism. These approaches are mutually contradictory, axiomatically false, materially/formally inconsistent and ALL got the pivotal economic concept profit wrong.

    There are the hard rocks of true or false and the swamp between them. The swamp is the natural habitat of agenda pushers, confused confusers, incompetent scientists, political economists, status-quo inertionalist, commonsensers, and anti-scientists. The very characteristic of science is to relentlessly drive the question under discussion to the point of a clear-cut decision between true or false, in other words, to get out of the swamp: “We are lost in a swamp, the morass of our ignorance. … We have to find the roots and get ourselves out! … Braids or bootstraps are necessary for two purposes: to pull ourselves out of the swamp and, afterwards, to keep our bits an pieces together in an orderly fashion.” (Schmiechen)

    See part 2

  37. Part 2

    The braids or bootstraps of science are material and formal consistency. Formal consistency is established by the axiomatic-deductive method, material consistency by state-of-art testing.

    The very characteristic of economics is that the “vast literature” consists of inconsistent and inconclusive blather or politics dressed up as science. So, perhaps 90 percent of the content of peer-reviewed quality journals consists of models/theories that are axiomatically false. Axiomatically false means based on false Walrasian microfoundations or false Keynesian macrofoundations or ad hoc plucked out of thin air. Therefore, the one and only interesting question in the given situation is how to achieve the paradigm shift from defective micro- and macrofoundations to materially and formally consistent macrofoundations.#3

    Accordingly, the straightforward criterion for disposing the “crappy vast literature” speedily into the waste basket is: If it isn’t macro-axiomatized, it isn’t economics.

    Egmont Kakarot-Handtke

    #1 See also ‘What is so great about cargo cult science? or, How economists learned to stop worrying about failure’

    #2 The political economist and incompetent scientist Keynes was one of the loudest defenders of conceptual vagueness: “Another danger is that you may ‘precise everything away’ and be left with only a comparative poverty of meaning. ... Such a problem was avoided, said Keynes, by Marshall who used loose definitions but allowed the reader to infer his meaning from ‘the richness of context’.” (Coates)

    #3 See ‘True macrofoundations: the reset of economics’

  38. "Human pheromones": On a smaller scale I encountered a self-reinforcing scientific literature that has grown up about the molecules claimed by a corporation to be human pheromones. I don't think it was done as a conspiracy at all. However, the literature continues to grow.

    It's likely to be built up by false positives as no evidence that the molecules, androstadienone and estratetraenol, are pheromones has been published. Previous debunking had concentrated on the details of individual papers in the literature. I went back to the original claims and showed them to be baseless - making the detail of the later experiments unimportant to the claim these were pheromone effects.

    Wyatt, T. D. (2015) 'The search for human pheromones: the lost decades and the necessity of returning to first principles', Proc Roy Soc B: 282(1804), pp. 2014.2994. [open access]

  39. Why the "vast literature"? Because one can't survive in academia without churning out countless publications. It is not unusual these days to expect graduating PhDs in neuroscience, for example, to have 40 papers to their credit. Junior professors who try to carefully replicate their own work before submitting for publication to ensure that the results are not a fluke are just asking to be turned down for tenure. Tenured professors who want to hang on to their funding know they must keep cranking out pubs or wave goodbye to their research labs. We've pushed the bar into the stratosphere for measuring what it means to be a successful scholar or scientist.

    Now we're paying the price in terms of literatures of overwhelming size, fields in which new journals proliferate like breeding rabbits, and entire bodies of research results that can't be replicated because they weren't true results to begin with.

  40. "If these two papers are full of mistakes and bad reasoning,.." what are the criteria to decide the "mistakes" and "bad reasoning"? Every paper starting with market clearing and/or market equilibrium, for example, sounds like "bad reasoning" to me....