A Field Guide to Millennials and Gen-Xers in Social Data Analytics

Background: This was vaguely solicited advice to a funding agency which, exercising the usual discretion characteristic of this site, shall remain anonymous. [1] Hence the organization into ten points rather than the usual seven.

1. They are digital natives: You cannot manage them unless you speak that language, and fairly fluently at that. They will instantly detect posers in this domain.

2. They are very social and travel in non-exclusive herds or, as they prefer, tribes. They are innately collaborative and remarkably adept at self-organization, including long-distance collaboration.

3. Consistent with their social ethos, they share and expect others to share. In the data analytic world, if you aren’t on GitHub, you might as well not exist.

4. Observational evidence suggests that they can survive a total of between three to ten hours of exposure to the 200 yellow-on-green, 8-point-type PowerPoint slide presentations that characterize monthly program reviews. Ethical constraints have precluded establishing this number precisely, though they will usually respond to such treatments by fleeing the project—see Point #10—rather than clawing their own eyes out.

5. They are fearless adopters, assessors and modifiers of new technology. Contrary to stereotypes, when properly motivated they have a remarkable capacity for work: A millennial working on the early stages of one project I was involved with ended up in the emergency room due to dehydration after a night of data-wrangling. He survived, and now teaches at Princeton.

6. They like feedback, intermediate rewards and have a possibly overly acute—though generally accurate—sense of injustice when assessing organizational management.

7. Unlike the notoriously sexist first generation of political methodologists (and the hopelessly sexist game theorists before them [2]) they are fully open to participation of, and leadership by, women. In fact they find exclusively male environments rather odd and alienating.

8. They prefer to approach their work with a sense of humor: at the recent New Directions in Text as Data conference, the first slide of the first presentation was a drawing of a squirrel with a martini glass [3]. The presenter, a woman, said “Every presentation at this [predominantly Millennial and GenX-er] conference needs to have a picture of an animal.” This had not been announced in advance, but indeed every subsequent presentation contained a picture of an animal.[4]

9. They are skeptics, and if someone tells them something is impossible when they know it has already been done—the forte of physicists and engineers talking about social science research—you will lose them immediately. They are also skeptical of each other: the recently established Millennial and Gen-X journal Research and Politics has the strongest replication norms in political science, and quite likely anywhere in the social sciences..

10. Every reasonably-sized data analytics company in the world has at least half a dozen openings: a Millennial’s alternative to working for your project is to work for Google, Apple, Amazon, Facebook or Microsoft. Not Starbucks.

A Gen-Xer social data analytics researcher I know seeking to escape an academic institution in a rather remote village interviewed with each of these five, and within weeks had offers from every one; she went with Microsoft Research in New York City. Another, who had taken several graduate courses from her institution’s statistics department, concluded that the methods required for publication in a certain social science were completely useless, so she quit and after a few months at a data start-up, ended up at Apple, in a city which is probably at the outer limits of the possibility curve on the dimensions “coolness” and “affordable.” A final example—though I have more—involved a student with considerable experience in conflict data analytics who took a summer internship at an insurance company where he wrote a little model to predict the location of pirate attacks, which the company then used to secure a contract with the world’s largest shipping company. You will be shocked, shocked to learn that he has an offer for non-academic employment as well. Also in what has been considered one of the world’s coolest cities. During the time of the Roman Empire. Before that unpleasantness with the Saxons. I digress.


1. Which is to say, blindingly obvious to almost everyone likely to read this. It is not Penn State.

2. Where various personality disorders at the clinical level also seemed to correlate with professional success, and I’m not just thinking of John Nash. Though apparently John von Neumann was a pretty nice guy.

3. It made sense in context…well, sort of… The martini had an acorn rather than an olive. I do not know whether the martini was shaken or stirred: it wasn’t that kind of squirrel.

4. My contribution (taken at a workshop I attended in South Africa):blackswan


Posted in Uncategorized | 2 Comments

Is Trump pulling a Colbert on the Republican Party?

As a Will Rogers Democrat [1], I’ve been watching with delight as Donald Trump states, re-states, and then doubles-down on statements that seem almost perfectly designed to offend the median voter and put other Republican presidential hopefuls—now a not insignificant percentage of the entire party—in uncomfortable positions. The whole thing is almost too good to be true.

Then in the fading wakefulness of last evening, as the brain starts making random connections that are both crazy and creative, it hit me in a flash: what if it is too good to be true?! [2]

That is, what if Trump is pursuing the Stephen Colbert strategy of setting himself up as a parody of the looney right, and simply seeing how far he can go? Consider the following seven characteristics that point in this direction.

1. As has been pointed out in numerous forums [3], for most of his life Trump has been not only a Democrat, but as Dana Milbank has pointed out, a fairly progressive Democrat at that.

2. His “born again Republican” schtick does not have any obvious motivation—nothing equivalent to Ronald Reagan’s battles with Communist-influenced unions in Hollywood—and he flipped almost immediately into an extreme position.

3. His positions, first as a “birther” and now as a combination of an anti-immigrant know-nothing, Putinesque Great Dictator and Green Lantern seem almost optimally designed to embarrass the GOP by focusing attention on the looniest of the loony ideas floating therein. To date he is completely ignoring Republican leadership appeals to stop the madness.

4. What, exactly, is “Trumpism”?: it seems to change by the day but in addition to overt racism, includes such howlers as secret plans to impose his will on a country holding $1.2-trillion in US debt. In contrast, the libertarianism—if that’s what it still is—of Rand Paul may seem a bit fringe, but at least it is a coherent  ideology with a long political and intellectual history.

5. No one gives Trump even a remote chance of winning a general election, however successful he might be in primary elections in small states.[5] But he could certainly thoroughly disrupt those already problematic Republican primary debates.

6. Trump may or may not be threatening to run a Ross-Perot-like third-party candidacy if—which is to say, when—he doesn’t get the GOP nomination. This would almost certainly throw the electoral votes of Michigan, Ohio, Pennsylvania, Florida, Virginia and possibly North Carolina to the Democrats, at which point we could save some time and effort by skipping the election altogether and just appointing Hillary.

7. Like Colbert, Trump is a television entertainer: let us not forget that.

Trump, bluster notwithstanding, seems to be paying at least some price for these antics,[4] though of course if he is worth any significant fraction of what he claims, he is nowhere near the poverty level, and in the meantime it appears that he will be providing lucrative employment for legions of contract lawyers.

But whether or not it is an act, this is going to be a tough tiger to dismount. And if Trump is, in fact, doing this—with thus far devastating effectiveness—as a favor to Hillary Clinton and the Democratic Party, probably the best comparison is not to Colbert, but to the undercover but nonetheless heroic and self-sacrificing Severus Snape. Hey, they already appear to share the same hair stylist!


1. “I am not a member of any organized political party. I am a Democrat.”
Read more at http://www.brainyquote.com/quotes/authors/w/will_rogers_2.html#6z1ffTDwfSkujHaa.99

2. The conservative Washington Times—I’m not a regular reader, and found this only as the result of Google—is apparently also having some of these same doubts.

3. Google “donald trump democratic donations” for a long list of essays to this effect.

4. To say nothing of the price his employees must be paying: the thoroughly loathed “Trump Winery” is just a few miles from here and I’m certain few if any of the people working there expected to be signing on for this sort of thing.

5. Though this is quite a recent development, as Chris Cillizza pointed out just three weeks ago.

Posted in Politics | 1 Comment

Seven reasons I probably can’t help you get my open source software running on your computer

The EL:DIABLO/PETRARCH system is beginning to get some traction, and with this, we are starting to get increasing numbers of emails asking for assistance. In a few instances, these can be resolved and in a very, very few instances, they have alerted us to important bugs. However—there’s probably a rule of thumb here—if an issue can’t be resolved in two or three email exchanges, it usually cannot be resolved at all.

This, I am sure, ends up being very frustrating to the person who originally sought the assistance, and it almost certainly leads to some variant on the following generalization about open source in general and programmers in particular: Open source is too good to be true, and programmers are a bunch of introverted overpaid arrogant bastards with inadequate personal hygiene who deliberately obfuscate their work to make it seem more difficult than it really is even as they pretend to provide it to you for free. Working on a script?: Have a programmer eviscerated by a velociraptor, crushed in a trash compactor, or reduced to Valyrian dragon chow.[1] All of these are sure crowd-pleasers.

Well, it’s actually a little more complicated than that, and hence I am devoting a bit of time—also provided for free—to explaining why.

1. No two research computers are identical

For purposes of this discussion, we will stipulate that the software you are trying to use works when correctly installed. In the case of EL:DI/PETR, we’ve been running the system 24/7 for about six months and have encountered a single untrapped bug (which we corrected, or rather, trapped). The major open source projects—R, LaTeX, perl, Python, NumPy—have accumulated millions of hours of use on tens of thousands of installations and thus, you can reasonably assume, also work.[2]

So why won’t it work on your computer?

In the absence of access to your machine—and please, don’t ever, ever grant access to your machine to someone you just know via the internet—the basic answer is “we haven’t a clue.” In situations where one is experimenting with open source software—as opposed, for example, to a standardized telemarketing operation selling the rights, smirk, smirk, to gold bars to protect you from the massive hyperinflation of 2009-2011 that destroyed the value of the US dollar[3]—it is safe to assume that on the software side, no two computers are identical. I don’t know what else you have on your machine, or how it is installed: In a complex system like Python, or even EL:DI/PETR, there are multiple ways of doing this, and they all work, but they work differently. I also don’t know your directory structures, and probably nine times out of ten, that’s where the problem is.

It is even worse when you are in an institutional setting since your machine may be deliberately configured, for good reasons or bad, so that it is impossible to run the software. I’ve known institutional IT specialists I would trust to do almost anything correctly, and I’ve known some who would have difficulty operating a vending machine.[3] Again, I have essentially no information on how your machine is configured, nor any ethical way of obtaining that information. This is a problem.

A Story: Last summer I spent the better part of two weeks trying, via email, to help a graduate student in Europe get TABARI running on a Linux installation. I’ve run TABARI on dozens of Linux systems, nothing new there, but hadn’t seen this issue, and could not debug it remotely: The effort was unsuccessful.

A few weeks later I upgraded my software to the latest version of the GNU compiler suite, and suddenly TABARI would not compile for me either. In fifteen minutes, I’d traced the issue to a simple problem in the ‘make’ file, and moving a single file name from the beginning of a list to the end solved everything. I’m sure changing the ‘make’ utility made sense to someone somewhere, and probably even makes the whole GNU suite more consistent, but it broke this particular piece of code. The person I was trying to help happened to be just a bit ahead of the curve in getting an up-to-date installation.

Email exchanges, by the way, are particularly inefficient because of the lag time: I solved the problem in fifteen minutes because I could run however many experiments—dozens, typically—each taking a few seconds (try this, didn’t work; try that, didn’t work, rinse and repeat). Email: we’re talking weeks.

2. You shouldn’t be using open source software unless you accept the core elements of the social contract of open source.

Open source software is only “free as in puppy.” Entire books have been written about why open source works, and let me say that for most programmers, particularly those working independently who have to provide their own tools, it has been an utter and complete game changer, about as far from “too good to be true” as you can imagine. Others have written more extensively on the implicit rules of open source software, but for our purposes I’d emphasize:

  1. We have provided you with every single line of code so there is no possible issue with the software that you cannot, given sufficient knowledge and effort, figure out.
  2. In addition to whatever intrinsic rewards we may obtain [4], we are posting this code because we want it to be improved: steel sharpens steel.
  3. As with every complex social system involving coordination for a common good [6] working effectively in the open source world involves internalizing and following a set of cultural norms.

Or rather “sets”, as there are multiple cultures, for example those of Python, javascript and R. The most obvious difference is in the skill set the user is presumed to have already mastered but there are others: for example I think generally web-based programming forums such as those for javascript and CSS are noticeably more polite and social than Linux or C forums.

A Story: Ever tried playing in a bluegrass, Irish or old-time “jam”, those informal gatherings of musicians of multiple skill levels? These typically start out really easy, with slow, familiar tunes, lots of open chords, and pretty much everyone can contribute. But then a point comes when the banjo player is channeling Earl Scruggs and the mandolin player Bill Monroe and the fiddler must, indeed, have made a deal directly with Satan (Satanic fiddlers, I repeat myself). At that point, unless you have had a heck of a lot of practice, there is no way you can possibly keep up and the best thing to do is just put down your instrument and enjoy the show.[7]

A really good musician can work in multiple genres, but a medium level bluegrass player is not going to immediately become a medium-level Irish player, or vice-versa. The fact that you are really skilled in R doesn’t mean you’ve got a comparable level of skill in Python, or vice versa, though you are probably well on the way to getting there. But there is a still a learning curve and at first you are going to get hung up on some really simple things.

Open source is like that. There are levels where everyone can play: certainly you should be able to do “Hello, World!” in pretty much any language, and then as you work your way up things get harder—say to the Python statistical data management system Pandas, which appears to be channeling R, and quite possibly Satan—at some point the software will get beyond your skill level. For example despite almost 50 years of programming experience [8] , I would not even begin to think I could touch the Linux kernel code. EL:DI/PETR, with three or four complex parts, and multiple authors, is heading towards the upper range: it isn’t the Linux kernel, but getting it fully deployed is a lot complicated than deploying its predecessor, TABARI. [9]

3. Wizardry is hard.

One of the many things I like about J.K. Rowling’s Harry Potter series is that learning to be a wizard is hard work: potions can take days to concoct, and still fail, and there are spells that take years to master and some people never really figure them out.

Yep, sounds a lot like programming to me, which is probably why most programmers really like Harry Potter.

Not everything is hard, and actually, you never know for sure just what will be hard. Sometimes things work right away; sometimes they can take hours to get it working even when the final correction is just a couple lines—or characters—of code. That was certainly my experience with installing one of the major Python packages—I believe it was NumPy—which after several hours I finally traced to having incorrectly installed something earlier. If you are willing to put in that sort of work, open source provides an astonishing set of opportunities. If you aren’t, at some point it isn’t going to work for you.

A Story: You’ve all heard the ca. 1955 three rules for making it through life, right?

  • never eat at a place named Mom’s
  • never play poker with a man named Doc
  • never get into bed with someone crazier than you are [10]

Here’s the analogue for finding a good programmer:

  • Programming is 10% book learning and 90% experience
  • The programmers who you might hire will vary by at least a factor of ten in terms of the amount of time it will take them to complete a task; the very best could have an edge of one-hundred, and a remarkable number of people who say they can do the job will not be able to finish it at all
  • Like competent craftspeople in every place and age, they are in short supply: there are never enough good programmers for the tasks requiring a good programmer [11]

This is not to discourage people from learning programming, starting with the 50% of the population who were commonly programmers when I first learned FORTRAN [12] but currently are discouraged from entering the field. But it will take quite an investment of time, particularly with the complexity of current environments, and, in my experience, at least some aptitude for the task.[13] However, the resources now available are vast and very accessible and in the end you will find that most skillful programmers are not merely introverted arrogant bastards best suited for dragon chow. A few aren’t even introverted! The pay and independence isn’t bad either.

4. Critical thinking is great, but problem solving skills are even better.

Some instructional programs do this really well, and others do not. More generally, to the massive frustration of those who will not rest until they’ve increased income inequality to the level of the late Roman Empire [14] programming remains a craft, not a simple set of routine tasks: You learn a set of tools but then it can take years to figure out how to best apply them.

As for assuming that learning programming can’t be all that hard and anyone can do it—and for some tasks, it isn’t that hard, and if that’s your situation, ignore the following—let’s go back to the musical analogy:[15] You’re putting together a bluegrass band, and you need a banjo player. Which strategy do you think is going to have a better outcome?: call your worthless nephew Bob and say “Bob, stop smoking that joint and go learn to play the banjo.” Or find someone who already knows how to play the banjo—ideally, bluegrass banjo—who you can persuade to join the band.

A Story: A while back I was involved on the edges of a small construction project with a guy who has been doing carpentry for about as long as I’ve been doing programming. There were no specific plans, just a building site, a load of lumber, and—definitely—a nail gun. As I watched the guy at work, I realized that the difference between how he did the project and how I would have done the same thing is that he was always thinking three or four steps ahead, and, out of long experience, doing things that would prevent problems further down the line. That’s a pretty good definition of an expert, and much of the reason an expert can do things faster and more effectively than someone just watching Home Depot videos on YouTube. In a few hours that pile of lumber was a woodshed, and I suspect it will be standing longer than I will.

5. Software is neither static nor, particularly across systems, logically consistent.

Software, particularly open source, is organic and evolves, but for that same reason, it exists in a series of punctuated equilibria. Once something works it will probably be maintained, until it isn’t. To function in this environment, you need the twenty-first century equivalent of horse sense, which works because while half the software one is using probably didn’t exist five years ago, other parts existed forty years ago.

A Story. About three weeks ago we noticed that the big black plastic wireless device in our guest bedroom that exists primarily to consume extortionately-priced ink cartridges was now simply an inert big black plastic brick providing the final resting place for stink bugs. We had changed nothing: it just stopped responding. Per the haiku that I believe was the motto of the Windows development team during the “Blue Screen of Death” decades:

Your computer was fine
But now it will do nothing
Sad, so very sad

I went onto the web and found we were definitely not the only people on the planet who were unexpectedly having this experience, and tried various of the suggested solutions to no avail. In the end, I simply unplugged both the printer and our wireless router, plugged them in again, and everything worked.

Computers are like that.

6. Simple is really hard, and really expensive. So is documentation.

Though if the software comes with documentation, you should at least read it. Yes, yes, we know that you didn’t need to read the documentation when you bought a system from one of those companies that has more money in their [off-shore, tax-sheltered] bank account than is held by the US Treasury, but a lot of open-source software was not written by such companies.[16] And virtually all open source software was written to solve sets of problems which may well not be precisely the same set of problems you are trying to solve.

To take our proximate case, EL:DI/PETR is already at a level of complexity where I don’t know the purpose of every line of code. We still don’t have a large developer community, but the system involves very substantial development by not only me, but by John Beieler.[17] Do I understand all of Beieler’s code?: no, I just know that it works. Neither Beieler nor I even begin to understand all of the code in Stanford’s CoreNLP suite. Which also works. Usually.

I actually did understand every line of code in TABARI, but that was a ca. 2000 program with only limited open source involvement. And even with TABARI, there is a rule of thumb that any code you wrote more than six months ago might as well have been written by another person. It takes a couple of hours for me to get back into the code in any significant part of PETR, the last time I was doing serious programming on it was four months ago, and in the intervening period I’ve worked on three or four other projects of comparable complexity.

A Story: I will not provide a story here: go read the documentation. Including the comments in the code. And if you find something wrong with the documentation, at least tell us, but better yet, fix it.

7. I’m not working for you

You are probably being rewarded—whether with hourly pay, a stipend, class credit, the prospect of an M.A. or Ph.D. thesis, or whatever—for getting the open source software to run. I am not, and I’ve already gone the extra step of making the code available with at least some level of documentation.

So, I know what you are thinking: it’s that old open-source bait-and-switch! I’m only pretending to provide the software but in fact in order to actually use it, you have to pay me. Arrogant bastard programmers! Dragon chow! Dragon chow!

Again, it’s a little more complicated than that.

It is true that, as with every contract programmer—and I’d extend this to the many small development groups with only a single level of management, though not to those working for corporations with bank accounts, or VC funding, exceeding the currency reserves of most nation-states—I only get to eat what I kill. So to speak. And contrary to the popular mythology about open source, most is actually written by professional programmers.[18]

But that doesn’t mean I want to work for you. For starters, wrapping this entire essay back to Point #1 [19], I probably can’t help you unless I can put my hands on your machine.[20] Which is not terribly practical if you are on the other side of the country. Or the planet.

Once I’ve got that access, assuming we’ve eliminated the problems that could be resolved by reading the documentation, getting the software to work would probably take me anywhere from fifteen minutes to a couple hours, but, returning again to Point #1, I have absolutely no way of predicting what amount of time will be involved. For that amount of work, it still makes little sense for you to hire me.[21]

In the meantime, the time I’m spending doing what is really your job is taking away from the time I have available for projects where I have an on-going relationship with the client and I’d really like to deliver them a quality product more or less when I said I would. Responding to a probably impossible email request does not contribute to that end.

So yes, if you can’t figure out the software, you probably need to hire a programmer, but a local programmer, and someone with administrative access to your equipment, and probably someone you will be asking to do work on more than one occasion. Because they are unfamiliar with EL:DI/PETR, they probably won’t be able to solve the problem in fifteen minutes, though they probably will be able to solve it. If no one with that skill set exists in your vicinity, I’d say you’ve just found a business opportunity. But not one that I can solve. I have promises to keep and, indeed, miles to go before I sleep.

A Story: My wife took on the task of getting our ComCast account moved from State College to Charlottesville, a process involving roughly the same degree of complexity and effort as the Iranian nuclear program negotiations. After one particularly extended—if futile—conversation with a ComCast rep, she got him to say where he was located. “I’m in the Philippines. The rainy season is coming, and my bosses are all really crabby.” We eventually got the system working, then squirrels ate through the coaxial cable.[22]


1. This occurs in Season Six: George R. R. Martin is going kinda slow writing the books so the script writers are going to need to get kinda creative.

2. One of the reasons I’ve switched to working almost entirely with open source software is I was finding more bugs in the propreitary software I was working with than in the open source.

3. You remember the dramatic dollar hyper-inflation of 2009-2011, right? When liberals who had foolishly ignored the copious warnings on Fox and conservative talk radio were reduced to using rolled-up Obama posters to kill rats for food?

4. What is printed on the bottom of cans of RedBull sold at Microsoft [substitute Apple or Google per your preference]? “Open other end.”

Though this is nothing compared to the IT people employed at the lower levels in academia, where salaries are typically not competitive with the private sector and, as the saying goes, they usually don’t get the sharpest crayons in box. In university research centers, this is less of a problem, since the attractiveness of working on a variety of problems and [sometimes] with state-of-the-art equipment and software compensates. But if the job is routine and doesn’t pay competitively, you’re probably looking at a pretty much continuous mess’o’trouble.

5. The ego boosts are real: I still remember the first time, this when open source was still fairly new, when I saw that a rather esoteric program—it implemented the algorithm for fitting hidden Markov models—that I’d converted from someone else’s open source code in C to Pascal (or maybe it was the other way around) was being used by some electrical engineers in a publication. And about the same time, some undergraduate at Stanford cleaned up the TABARI code so it no longer generated warnings when compiled on Linux.

6. Which is to say, every human system except those postulated by “rational choice” economists and political scientists, who like Alice in Wonderland’s Red Queen can only thrive in their tenured sinecures by learning to believe six impossible things before breakfast. I digress. Though it isn’t just six.

7. In the EL:DI/PETR world: just use the data.

8. I exaggerate: it’s really only been 48 years.

9. The code in TABARI, written in C/C++, is substantially more complicated, which is also why TABARI is about fifteen times faster than PETRARCH, but the full system in EL:DI/PETR is more complex. And of course does a lot more.

10. There’s a fourth: if you can’t identify the sucker at a poker table, it’s you.

11. Consequently, if someone with an active research program suggests you hire one of their students…

12. “Ada Lovelace wrote the first loop. I will never forget that. None of us ever will.” Admiral Grace Hopper.

13. Luthier Wayne Henderson, channelling a Victorian-era joke, says that all you need to do to make a guitar is take a bunch of wood and a pocket knife and carve away everything that doesn’t look like a guitar. There are times I feel like something along those lines can happen, in reverse, with a program.

That is, sometimes I’ll sketch a design, but particularly on smaller projects where I’m unlikely to re-use the program, I’ll just sit down and start writing code, not necessarily consciously knowing why, but confident that at some point I’m going to need that code because I’ve written similar programs many times in the past. So I suppose, following Henderson, a lot of what I do is just take an empty text file and keep adding chunks of code until it works like a program.

But don’t try this at home.

14. Actually, it’s already at that level, but that’s for another discussion.

15. Yes, I see another blog entry emerging here…

16. Though quite a bit is.

17. On GitHub, you can see exactly who contributed what.

18. Who is a professional programmer? If that’s in your job description, it’s an easy determination. But if, like me, you’ve done contract work on a variety of tasks—my degrees are in mathematics and political science, not computer science, which wasn’t even a major when I was an undergraduate—it comes along more gradually. In such a situation, I’d say being a “professional” means is getting hired, on multiple occasions, to write software someone else is going to use for their work. Quite a few years ago I started adding up how much I’d been paid over time for writing such software and realized it came to multiple hundreds of thousands of dollars, at which point I figured I was a “professional.”

Once again, there is a good analogy with musicians. You’re an orchestra or studio musician with a union card: sure. But you start out playing for fun, then for beer money, and then at weddings, then you record a CD and it sells: when does the transition occur?

19. Or, in musical theory, resolving the essay to the opening key. A pretty good technique in blogs, it turns out.

20. Not in the sense of Oral Roberts, though at times that appears to help as well.

21. Which is to recall a particularly notorious tax form that Pennsylvania requires of LLCs which is so bad—it runs some ten pages, and I swear probably contains a provision absolving any corporation which provides free natural gas to kitchen faucets  in northwest Pennsylvania of all tax liabilities—that accountants refuse to do it. I asked. I did my best, and concluded I owed about $250 only to receive a notice some eight months later that I’d done the calculations incorrectly—certainly no surprise there, as the 38 pages of instructions appear to be a bad translation from proto-Hittite [23]—and actually owed only $21. No refund check, however: that’s probably a twenty-page form. Pennsylvania: the state where the only task the government performs competently is incentivizing citizens to purchase wine in New Jersey, Maryland and Ohio.

22. This story, you will notice, has nothing to do with the rest of the entry, but mouseCorp—slave-drivers—pays me by the word. Blogs are like that.

23. There really are 38 pages of instructions. But they weren’t translated from proto-Hittite but rather they written in something far worse: fluently crafted lobbyistian. Which is to say, most of that form exists to provide massive tax breaks to a tiny number of individuals, but this is quite deliberately written in a fashion that makes it nearly impossible to figure out who that beneficiary is. And as part of the game, the beneficiary will, of course, be constantly complaining about how unfair the tax system is.

And, well, it’s pretty unlikely that going to change: here’s a nice article on why you are never going to see the IRS provide any user-friendly software and—I’m shocked, shocked—lobbying money is involved. And it co-occurs with an article on the privatization of security—that old Weberian “sovereign monopoly on the legitimate use of force” was sooo twentieth-century—and presumably in due order we will see the slaughtering of unarmed black men out-sourced, with the help of Erik Prince, to Chinese “security” guards operating with impunity. And not just in Africa. Suggesting, once again, that we’ve moved into a system that looks a heck of a lot more like Rome under the Borgias than Wisconsin under the La Follettes. A topic for a later set of entries.

Posted in Uncategorized | Leave a comment

Seven observations on the newly released ICEWS data

Before we get to the topic of the post, the usual set of apologies about the absence of recent postings—starting with that Duke Nukem Forever style “Feral+well, whatever.” Isn’t that I’ve dropped out, it is rather that I’ve been too busy with other projects. And by the way, I haven’t retired,[1] unless logging 2,200 hours of work last calendar year is “retired.” Someday, things will slow down, I’ll get to the backlog. But enough about me…that’s not what we’re up to today.

Instead, the point of today’s posting is to comment upon the long, long awaited release of a public version of the Integrated Conflict Early Warning System (ICEWS) dataset, which appeared without fanfare on Dataverse late in the afternoon on Friday, 27 March, with Jay Ulfelder probably responsible for first spotting it.

This is a massive resource: The investment in the ICEWS project, albeit not all of the funding going into the data, is probably roughly equalled the whole of NSF spending on all international relations and comparative politics research during the time it was active. As with any large data set it is going to take a while to figure out all of its quirks. The ever-resourceful David Masad already has some excellent instructions and initial analyses and visualizations here, and I’m sure more will be forthcoming in coming weeks—and years—but I wanted to use the occasion to alert my [ever declining] readership to this, and provide some initial observations.

1. It exists!

Long overdue, to be sure, given that at the ICEWS “Kick-off Meeting” in 2007 we were assured that everything in the project would be open, a concept almost immediately quashed, and then some, by the prime contractors. I’m pretty sure we have the persistent and unrelenting efforts of Mike Ward and Philippe Loustaunau to thank for the release. I’ve also got a pretty good idea of who is responsible for the delays but, well, let’s just focus on positive things right now.[2]

Here’s the link to the data: http://thedata.harvard.edu/dvn/dv/icews.  There are actually four “studies” involved

  • 28075: This is the main data set: 26 files, most of these are around 30 Mb each. Took me a couple tries to get some of them, though that may have been due to a lousy wireless connection on my end. It’s Dataverse; it will work.
  • 28117: Aggregated data: this may the quickest way to get into the data, assuming what you are interested in is covered by one of the very large number of aggregations that have already been computed. I’ve not really looked at these yet, and much of the documentation appears oriented to a proprietary dashboard which is not provided, but particularly for people not comfortable working with very large disaggregated data sets, it could be very useful.
  •  28118:These are the dictionaries, more on this below.
  • 27119: This was the big disappointment: we had been told the release would include a set of “gold standard cases”, which we assumed would be the much-needed gold standard cases needed to validate event coding systems, but these, alas, are just some sort of esoteric records associated with the much-disputed ICEWS “events of interest.” Hard to imagine it will be much use for anyone, but I’ve been wrong before.

2. Dictionaries!

As we’ve [3] been arguing for decades, the primary advantage of automated coding is the ability to maintain consistent coding across data sets being coded across a number of years and, through the dictionaries, to have a high level of transparency.[4] For that you need the dictionaries as well as the coder, and the KEDS project [5] has been consistently providing those as part of the data. To its great credit, ICEWS has followed this norm, and what dictionaries they are!: a primary actor dictionary with over 100,000 political actors. The format is derivative of that used in TABARI and PETRARCH—I’m guessing it will take about fifteen minutes to write a converter.

The agent dictionary—also provided, along with a somewhat cryptic “sectors” dictionary— on the other hand, is definitely a work in progress, though probably fine for the major sub-national actors, which for the most part are still those established by Doug and Joe Bond’s work on PANDA and IDEA back in the 1990s, and subsequently incorporated into CAMEO. The quirk that really sticks out for me is the treatment of religion: for ICEWS, it seems “Christian” and “Catholic” are separate primary categories [6]—granted, the late Ian Paisley [7] would agree—and the Great Schism of 1054 apparently was no big deal. The whole of Judaism gets only two entries, rather an oversimplification for the neck of the woods I’m usually studying. There is an extraordinarily eclectic set of ethnic groups—with a distinct oversampling in India—and, well, overall, the agent dictionary is sort of like rummaging through some old trunk in your grandparent’s attic [8], and I’m pretty sure we’re well ahead of this at the Open Event Data Alliance.

ICEWS did not provide event code dictionaries, which are presumably tightly linked with the proprietary BBN ACCENT coder. This is a bit of an issue, since ACCENT does not actually code CAMEO, but their own variant which they have documented in a very extensive manual. Not ideal but no worse than the situation with any human-coded data.

3. You’ll need to convert it for statistical analysis but I’ve got a program for that.

The public-release ICEWS uses a very quirky format that is apparently designed to be read rather than analyzed, as the underlying codes are presented in verbose, English-language equivalents. Unless you are ready to settle into a few quiet evenings reading through the 5-million records, you’ll probably want to use the data in statistical analyses, which means you’ll want to get shorter codes. It just so happens, I’ve got an open-source program for that at https://github.com/philip-schrodt/text_to_CAMEO and I’ve even provided you’all with COW codes as well as ISO-3166-alpha3 codes. I didn’t fully convert the sector dictionaries, but this will at least give you a good start.

4. Massive use of local sources

That old criticism that event data are nothing but the world as viewed from the point of Western imperialists? This will be hard to sustain with ICEWS, which uses hundreds of local sources, and each event contains information on the source. I’ve only looked in detail at 2013, and here these follow more or less a rank-size distribution, with some of the major international sources (Xinhua, BBC) being major contributors, but the tail of that distribution is extremely long.

5. The distribution is flat

While the internet, and new social media more generally, are revolutionizing our ability to inexpensively generate large-scale datasets relevant to the study of political behavior, a serious problem has been dealing with the exogenous effects of the rapid expansion of internet-based sources that began in the mid-2000s. Any “dumpster diving the web” approach leads to an exponential increase starting about this time, which for any statistical analysis is a bug, not a feature.

ICEWS avoids this: they seem to be using a relatively fixed set of sources, and the total density is largely flat. As Masad’s visualizations and some others I’ve seen show, there appears to be a bit of variation—1995 and 1996 seem undersampled—and more will probably appear as further research is done, since there have been major changes in the international journalism environment beyond just the increase in the availability of reports, but these variations are not exponential, and can probably be accommodated with relatively simple statistical adjustments.

6. 80% precision, but no assessment of the accuracy

The release is accompanied by an extensive analysis showing that the “accuracy” of the ACCENT coder is around 80%. Which would be very nice, except that the study actually assesses not accuracy, but precision, which, while interesting, gives us no information whatsoever on the measure most people are interested in: the probability of correctly coding a randomly chosen sentence (accuracy), rather than the probability that a sentence that was coded was coded correctly (precision). Echoing the exchange between Col. Harry Summers and one of his Vietnamese counterparts over the unbroken string of US battlefield successes, the assessed precision “May be so, but it is also irrelevant.”

The arguments here are a bit technical, though involve nothing more than simple algebra, so I’ve relegated this to an appendix to this post. The upshot, to paraphrase Ray Stevens, “Yo selected on the dependent variable, and I can hear yo’ mama sayin’, “You in a heap o’ trouble son, now just look what you’ve done””

7. It should splice with Phoenix

The current dataset has a one-year embargo, though the word on the street is that the embargo will remain at just one year, more or less . That is, the data will be periodically updated, ideally monthly, perhaps quarterly. [Addendum: in a very promising sign, the March 2014 data were indeed made available on 1 April 2015.] This will be adequate for most retrospective studies, but still won’t help with the real-time forecasting that event data are increasingly used for.

Here the recently-released OEDA Phoenix data set comes to the rescue, or will once we’ve got another four or five months of ICEWS data, as Phoenix gets going around the beginning of July 2014. Provided ICEWS is updated regularly, within a fairly short period of time one should be able to use the ICEWS 1995-2014 data for calibration, and then use Phoenix to cover the end of ICEWS to the present (Phoenix is updated daily).

Assuming, of course, that the data are sufficiently similar that they can be spliced, possibly with some adjustments. The major distinction between the data sets is likely to be the sources, with ICEWS using Open Source Center feeds and proprietary data services, and Phoenix a white-list of Web-based sources. This is likely to make a big difference in some areas—in the very limited exploration I’ve done, ICEWS seems to disproportionately focus on India, for example, and for statutory reasons, contains no internal data on the US—and less in others. Actor dictionaries will not be an issue as the ICEWS dictionaries could be used to code the Phoenix sources, though this may not be necessary.

The different coding engines may or may not make a difference: in the absence of a confirmable set of gold standard cases for events, and verb dictionaries, we will need a significant period when the two sets overlap to find out whether the two systems perform significantly differently. My guess is that they won’t differ all that much, particularly if common actor dictionaries are used, since that both coders are based on full parsing, and the differing sources will be the bigger issue. Both Phoenix and ICEWS provide information on the publications where the coded text came from, so these could be filtered to get similar source sets.

In the absence of a public version of ICEWS overlapping with the [still relatively brief] Phoenix data, we can only do indirect measures of the likely similarities, but some quick analyses I’ve done comparing marginals of the first six months of Phoenix with the last six months of ICEWS indicate two promising points of convergence: the density of data (events per day) was quite similar and—even more telling—the marginal densities of the event types were very similar (actors less so but again, that’s easily corrected since the ICEWS actor dictionaries are public).  Again, we won’t be able to do the more crucial test—the correlation of dyad-level event counts—until there is a substantial overlap in the public data, but initial indications are promising.

What needs to be done (all open)

Call me a greedy anti-intellectual knuckle-dragging Neanderthal—and you will—but when I read a recent article in Science about the construction of an esoteric scientific instrument whose construction cost was $300-million and annual operating costs are $30-million, and then compared that with the pittance that is being allocated—when we can avoid our programs being shut down altogether [9]—for event data which could contribute significantly to at the very least to increasing the ability of NGOs to accurately anticipate situations where “bad things might happen” [10], or even to a reality-based foreign policy, I get a tad irritated. Consider these aspects of the instrument in question:

  • it may not work—its also-costly predecessor did not—and half of the project is situated in a place in Louisiana that makes it less likely to work, suggesting it is largely a mindless boondoogle. A boondoogle located in Louisiana, I’m shocked, shocked.
  • if it does work, it merely further confirms a century-old theory which we’ve got complete confidence in already, and as the Science article points out, is confirmed billions of times each day, for example as a smart phone displays inappropriate content having determined that you are a male walking within fifty meters of a Victoria’s Secret outlet store. [14]
  • and the predictions of the theory at issue were already confirmed by other observational evidence four decades ago, for which the discoverers got a nice trip to Stockholm.

Which is to say, this is just the natural sciences equivalent of a performance art project [13], but at a rather higher price tag. And unlike space telescopes and Mars rovers, we don’t even get nice pictures from it.

So, like, if we can spend what will probably eventually total some half-billion dollars before this thing winds down, presumably with the yawn-inducing equivalent of the umteenth iteration of  “Hey, ya’know, Mars once had water on it!!” how about spending 1%—just a lousy 1%—of the that amount (which is probably also about 10% of the cost of ICEWS) on enhancing event data? And this time with social scientists in charge, not folks whose prime competence is raiding the public purse under the guise of protecting our national interests against opponents who disappeared decades ago. Oh, and every single line and file of the project open source. I’m just asking for 1%!  A guy can dream, right?

So, say we’ve got $5-million. Here’s my list

1. Open gold standard cases. Do it right: the baseline will be the openly available Linguistic Data Consortium GigaWord news files, use a realistically large set of coders with documented training protocols and inter-coder performance evaluation, do accuracy assessments, not just precision assessments. Sustained human coder performance is typically about 6 events per hour—probably faster on true negatives—and we will need at least 10,000 gold standard cases, double-coded, which comes to a nice even $50K for coders at $15/hour, double this amount for management, training and indirects, and we’re still at only $100K.

2. Solve—or at least improve upon—the open source geocoding issue. This is going to be the most expensive piece, and could easily absorb half the funds available. But the payoffs would be huge and apply in a wide number of domains, not just event data. I’d put $2M into this.

3. Extend CAMEO and standard sub-state actor codes, using open collaboration among assorted stakeholders with input from various coding groups working in related domains. We know, for example, that one of the main things missing in CAMEO are routine democratic processes such as elections, parliamentary coalition formation, and legislative debate, and there are people who know how to do this better than us bombs-and-bullets types. On sub-state actor coding, religious and ethnic groups are particularly important. I’m guessing one could usefully spend $250K here. Also call it something other than CAMEO.

4. Automated verb phrase recognition and extraction, which will be needed for extending the CAMEO successor ontology. I actually think we’re pretty close to solving this already, and we could get some really good software for $50K. [11]  If that software works as well as I hope it will, then spend another $250K getting verb-phrase dictionaries for the new comprehensive system.

5. Event-specific coding modules, for example for coding protests and electoral demonstrations. Open-ended, but one could get a couple templates for $100K.

6. Systematic assessment of the native language versus machine translation issue. That is, do we need coding systems (coders and dictionaries) specific to languages other than English, particularly French, Spanish, Arabic and Chinese [12], or is machine-translation now sufficient—remember, we’re just coding events, not analyzing poetry or political manifestos—so given finite resources, we would be better off continuing the software development in English (perhaps with source-language-specific enhancement for the quirks of machine translation). Hard to price this one but it is really important so I’d allocate $500K to it

7. Insert your favorite additional requirements here: we’ve still got $1.75M remaining in our budget, which also allows a fair amount of slack for excessively optimistic estimates on the other parts of the project. Or if no one has better ideas, next on my list would be systematically exploring splicing and other multiple-data-set methods such as multiple systems estimation. And persuade Lockheed to dust off the unjustly maligned JABARI—or make the code open source if they have no further use for it—and give us another alternative sequence based on that program.

All this for only 1% of the cost of a single natural science performance art project! Come on, someone out there with access to the public trough—or even some New Gilded Age gadzillionaire—let’s go for it! Pretty please?


1. Yeah, I can just imagine the conversations at ISA in New Orleans (I was on Maui. Just on vacation. Really.)

“Hey, Schrodt really disappeared once he left Penn State. Figured that would happen…”

“Really, it’s bad: I heard that he was last seen on the side of the exit ramp off I-99 to Tyrone, looking really gaunt and holding a cardboard sign that said ‘Will analyze mass atrocities for food’.”

“Yes, that’s right: so sad. So keep that in mind if you are thinking about leaving academia, or even imaging the possibility of asking any senior faculty to get their fat Boomer butts out of the way.”

Well, no, that’s not really accurate. But we’ll save that for another blog entry. Meanwhile, you can follow me on GitHub. And I’ll be at EPSA in Vienna.

2. And keep our faith in the wheel of karma.

3. I’m not exactly sure who “we” is—I’m neither royalty nor, to my knowledge, have a tapeworm—but I’m trying to represent the views of a loose amalgam of people who have been working with machine-coded event data for a good quarter-century now.

4. Total transparency when the coding software is available, which is not the case here, but even without the software these dictionaries are a huge improvement over the transparency in most human coding projects, where too many decisions rest on an undocumented and ever-shifting lore known only to the coders.

5. Or whatever it should be called: it will always be KEDS—Kansas Event Data System—to me.

6. Two Protestant denominations get designations at the same level as “Herdswoman” and “Pirate Party”—Episcopal (but not Anglican) and Methodist—and there is an entry for “Maronite.” That’s it: no Lutherans, no Baptists, no Pentecostals, no Mormons, not even the ever-afflicted Jehovah’s Witnesses. In fact in the ICEWS agent ontology, the only religions worthy of subcategories are Christian, Catholic, Buddhist, Hindu and Moslem, though the latter has not been affected by that unpleasantness at Karbala in 680 CE.  The ontology developers, however, appear to have spent a bit too much time watching re-runs of the Kung-Fu television series—or more ambiguously, Batman Begins—as only Buddhism produces “warriors.”

7. To say nothing of the late Fred Phelps.

8. Yeah, yeah, they moved to a condo in Arizona two decades ago and the old place was torn down and replaced with a MacMansion, but it still makes for a nice metaphor.

9. Though I did notice that Senator Jeff Flake was one of the few Republicans not to throw his lot in with the GOP efforts to provide free policy advice to the Islamic Republic of Iran, so perhaps his M.A. in Political Science did some good.

10.I think we are now at a point where these things can make a serious difference: The absence of major electoral violence in the 2013 Kenyan elections and—fingers crossed—the 2015 Burundi elections may eventually be seen as breakthroughs on this issue.

11. But meanwhile, don’t get me started on the vast amounts that is wasted on hiring programmers who never finish the job. Really, people, the $75 to $150 an hour to hire someone with a professional track record who will actually write the programs you need is a better deal than spending $25,000+ a semester—stipend, tuition and indirects, and this is actually a low estimate for many private institutions—for one or more GRAs are supposed to be learning programming but who, in fact, stand a pretty good chance of getting absolutely nowhere because writing sophisticated research software does not, in many instance, provide a good pedagogical platform. No matter what your Office for the Suppression of Research says.

12. Hindu/Urdu is also important in terms of the number of speakers, but, for better or worse, the media elites in the region use English extensively.

13. If you aren’t familiar with this concept, Google “bad performance art.” NSFW.

14. To clarify, not the precise example used by Science. Continue reading

Posted in Methodology | 4 Comments

The Mouse Goes Into Business [2]

Okay, let me be straight with you here: This one is going to be really boring and probably shouldn’t be a blog post at all. But the guys over at MouseCorp—slave drivers!—were really unhappy about the drop-off in blog productivity over the summer, and are absolutely hounding my butt to make quota before the end of the year, so like I gotta do something, right? Boring, not really worth reading, jokes even lamer than usual, nothing to see here, move along.

Well, got that off my chest. So…attitude adjustment…


and back to blogging.

Some of the challenges in setting up an independent business were discussed a couple of months ago, here and here. But for the most part, the process  was a series of fascinating little victories. A process interesting in an inevitable geeky way to me but I realized unless you are planning to do pretty much exactly what I’ve done, probably not all that interesting, nor all that original. But one or more elements of my experience might someday be useful somewhere to someone—more than can be said for most of things I’ve published in refereed journals—so here we go:


Out of inclination—and the advice of many others—part of the original feral plan had been to get an office outside my residence. As it happened, I inadvertently did an experiment in this regard, as the first option I thought I had lined up—a sublet at the end of a long dimly-lit hallway in a grim office complex—fell through (the occupant failed to check whether it was okay to sublet), and I worked out of my home office for about a month. It was okay, but the temptations everyone talks about—grab something from the kitchen, weed the garden, don’t take a shower until noon—were definitely there. I then lucked into—well, not luck, but my wife’s extensive social networks—an absolutely lovely office in an elegant house now occupied by several small businesses.  It was an easy walk from home, the rent was $450 per month, and “on a handshake”, a distinct contrast to one place I had checked into where the receptionist glared at me and said “We prefer only to provide space for people willing to sign a five year lease.” [1]

None of the other businesses had anything to do with mine—and I suppose in an ideal  situation I’d be sharing space with some tech firms—but they provided enough random human activity that I didn’t feel isolated. Also—yes, I’ve spent my entire life in large organizations—the fact that people could simply drive into our parking lot, which I could see from my window, was a pleasant and unexpected bonus.  The arrangement included a small kitchen, a shared coffee maker, a copy machine and a very large shared candy dish.

My office situation in CVille is not quite as good—in particular there is no common space—but in a quirky historical building [3], much the same distance from home, with parking, and at an even lower price. And two blocks from the CVille pedestrian mall and its seemingly endless array of coffee shops and restaurants: the one downside to the State College location was the sole food establishments in the immediate vicinity were a massive beer distributor—hey, it’s State College—and a friendly if rather downscale Quik-Trip and a walk of several blocks got you only to a Starbucks, Subway and Dunkin Donuts. Access to the CVille mall: priceless. Downside of the current venue is the absence of a chatty little community, no shared coffee maker, and no candy bowl. So as my lease runs out next summer—one year lease, that’s okay—I’ll probably be looking for an alternative.


My landlord in State College had accumulated a very eclectic collection of furniture in the garage of the building—sort of Antiques Roadshow meets Hoarders—and told me just to take my pick. I found a beautiful old solid wood desk with a couple missing drawers and a utilitarian table;  some guys with no necks hauled these upstairs for me and I assembled them with power tools. In CVille, the office was partially furnished and I completed that shopping at the local SPCA rummage store, finding an office chair for $7 and a nice little table for $10. In both offices I built custom tables from plywood and 2x4s—every time I make one of these I get better at it—to put the larger tables into the “L” configuration I prefer. No one is going to mistake this for the executive offices of a Fortune 100 company, though the furniture in State College was actually a lot nicer than what I’d had in Kansas.[4]


Lots of office spaces now include internet, but neither of mine did. Getting a hardwired connection was quite expensive, particularly given that I didn’t know how long I’d be staying, and the solution I’ve used is a Verizon wireless “JetPack”: basically a phone pretending to be a wireless hotspot.  The data plan I’ve got is limited to 5 Gb a month, but this is sufficient for my day-to-day work, and for the occasional high-bandwidth demands—notably teleconferences and updating software—I use our hardwired connection at home.[5] And the JetPack, about the size of a hockey puck, comes along when I travel and provides reliable internet access in the likes of airports, interstate highways and, Thor-forbid, conference hotels.


I already had a high-capacity MacBook Pro I’d used in Norway, and subsequently purchased a couple more machines from Penn State surplus for a few hundred dollars each which I used until about a month ago, when I was finally sufficiently settled that I got a large-screen iMac pimped out with a lot of RAM and a 3Tb hard drive. And with all of that equipment, no brain-addled bureaucrat telling me what I can and can’t install on it, in particular…


Everything I use [6] either came with the machine, is free-as-in-beer, and/or is free-as-in-open-source. And increasingly, everything I use for analytics is Python.

Color printer/scanner/copier:[11]

There are numerous machines in the $80 range that combine these functions, connect via your wireless network, and work right out of the box. Sure, the manufacturers expect to make up for that low cost selling ink cartridges, but I’ve got an older black-and-white laser printer I use for the very occasional high-volume print job—mostly I’m just reading things as PDF files [8]—or put the files on a USB drive and go to Kinkos or Staples.


There this fantastic thing called the World Wide Web, ya know? It’s absolutely full of information.

Establishing a legal entity as a small business:

Discussed in greater detail here, and while aggravating, and leading to the unexpected side effect of longing for Richard Nixon, not particularly onerous. Discovering that mortgage lenders find new small business owners as attractive as an Ebola carrier in a mosh pit was somewhat more problematic, though had a happy ending, as discussed here.

Business cards:

$10 or less from VistaPrint. Which can also provide every other imaginable business swag you may or may not need. Had some really cool EL:DIABLO-logoed coffee mugs made for my collaborators.

Health insurance

This is costing maybe 20% more than I was paying earlier, but minimal issues finding it in either Pennsylvania or Virginia. Thankfully I was able to get this under the Affordable Care Act, and not that dreaded Obamacare![7] But sorry, if you are using access to health insurance as an excuse not to get your Boomer butt out of the way, that one no longer works.


1. And State College wonders why it is not a popular venue for start-ups?

2. This isn’t really a footnote. Though I will use it to note that I’m not “monetizing” the various links to commercial enterprises you see here—nor has that been done in any of my entries. This blog is provided as a public service [yeah, right…] and maybe even the grist for an eventual book [yeah, right…]

3. Thomas Jefferson and James Monroe allegedly used to hang out at the site when doing business in the colonial-era courthouse across the street. Before the place burned down sometime in the second half of the nineteenth century, then was rebuilt in brick. The floors squeak.

4. At Penn State, in contrast, I’d inherited the custom wood furniture from the previous department head, which I then supplemented with some additional matching wood furniture from my own start-up budget. This was inherited by an assistant professor, who probably has the nicest office furniture of any assistant professor in the country. He also now has what is probably the longest job title of any assistant professor in the country.

5. Except during the times it is not working because squirrels have eaten through the coaxial cable.

6. Okay, in fact I use a single purchased program, BBEdit, though mostly because I’ve been supporting the company for some twenty years now and they long ago registered the trademark “It doesn’t suck”: how can you not like that? And the software does not, in fact, suck. But there are now open source alternatives even here.

7. Joke…

8. Please, please do not send docx!—yes, I can read those files through GoogleDocs [9] or LibreOffice but that is sooo 1990s.

A positive externality of paying for printing is that you don’t fall into the habit that I’ve seen so often in large organizations where one prints, say, a 10-page memo, or 75 page conference paper (or 400-page dissertation) because, well, because that is how they did things in the days of Gutenberg. PDFs on screen are, I would suggest, a little easier on the environment. And if you avoid printing on paper that was purchased from a Koch Industry subsidiary—and a lot of office paper is—then you are benefiting the environment twice over!

9. At which point if your document mentions, say, Boko Haram suicide bombers, and documents sent to me tend to, by posting these on Google means I’m effectively posting them to all sorts other places [10] but what would really get me into trouble is if your docx which I post to the Google Panopticon contains the phrase “Bargains on tropical timeshare resorts!”

10. Which were intercepting my email anyway…okay…whatever…tin hat and masking tape over the laptop camera…

11. It does fax as well, though I believe the fax is used with about the same frequency as carrier pidgeons, and far outpaced by bicycle couriers.

Posted in Ramblings | Leave a comment

Seven Theses on Theil and Drezner [1]

The Yule season is upon us, and as I sit watching sun progress across the windswept prairies of central Kansas, my Twitter-lurking attention was drawn to Dan Drezner’s recent challenge to Peter Theil’s prediction of the impending collapse of the “education bubble.”

With all due respect to Messrs. Theil and Drezner, I think they are oversimplifying the issue [2], as the pressures for change in higher education are quite multi-faceted. I’ve written—though of course, never bothered to publish—are rather extended  essay on this issue  which you should consider reading as an alternative to listening to <severe tangential riff alert!!!> the annual rendition of the cross-country trip your grandparents imposed upon your parents: coast-to-coast on a partially-completed Interstate system in a 1962 Ford station wagon with the emission controls, gas mileage and suspension of a Soviet T-34, your grandfather chain smoking unfiltered Camels while your grandmother confined herself to the more ladylike Salem Menthols, the recirculated air of the vehicle rapidly accumulating the same mix of carcinogens as the Love Canal Superfund site, though arguably this was less of a health threat than the food at Howard Johnsons and Nickerson Farms, to say nothing of the “Reptile Gardens” run by that family where people had six fingers and no teeth, and on this journey their children—your parents—counted cows [4], particularly while crossing Kansas, from the rear of the vehicle, unencumbered by seatbelts, much less the child safety seats you were strapped into even for those half-mile drives to Whole Foods when your parents went to pick up organic free-range tofu, elaborate contraptions which would have given the crews of the Challenger and Columbia shuttles at least a 50% chance of survival, and…well,  even this far into the story, except that you’ve now heard it at least nine times, you are wondering how any DNA was ever passed on in your family…I digress…<back to Drezner and Theil, at least sort of>…there are multiple changes going on.  Consequently, while I would not take Drezner up on his wager—which is to say, he is probably correct, and Theil is under-estimating the ability of academic institutions to resist change—there is more to be said.

So, in the spirit of the words variously ascribed to Stravinsky, Picasso, and Steven Jobs—”good artists copy, great artists steal”—and in the immortal—for as we are told, he is now immortal [5]—words of Stephen Colbert [6], the answers to life’s great questions are themselves questions, what follows are [of course] seven more specific questions, each with a Ehrlich/Simon style question following Drezner’s model, then a second Good Judgment Project (GJP) style question for just the next year. But unlike Drezner, I don’t plan to put money on any of these: I have squirrels to feed.

1. Public funding of higher education

Ehrlich/Simon: By 2020, the average state-funded contribution to the public universities which are members of the American Association of Universities (AAU) [10] will be at least 20% less than it was in 2014.

GJP: By 1 January 2016, a bill to privatize a public university which is a member of the AAU will have passed in at least one legislative chamber in some state.

Comments: Ehrlich/Simon version simply continues existing trends which show no sign of reversing. The logical extension of this is the GJP version, and many more major institutions than most people realize are already well below the 10% public funding point. An increasing number of state legislatures have increasingly tax-averse Republican majorities and such privatizations could generate substantial revenue streams: large universities are multi-billion-dollar corporations with tens of billions of dollars in capital plant and many could readily generate funding buy themselves out were the circumstances right. Relations between conservative state legislatures and universities were generally antagonistic even in the best of times, and we’re certainly no longer in the best of times. This one is nearly inevitable in the relatively near future.

2. Decline of tenure

Ehrlich/Simon: By 2020, the percentage of student credit hours taught by tenured professors in universities which are members of the AAU will be at least 20% lower than it was in 2014.

GJP: By 1 January 2016, at least one member of the AAU will announce that tenure is being phased out in the liberal arts: all subsequent hires will have fixed-term appointments.

Comments: Both of these are academic manifestations of that most notorious of the Boomer [9] Mandates: “Raise the drawbridge, boys, I’m safe inside!” Slam dunk on the Ehrlich/Simon version, and just a matter of time before someone gets the nerve to pull the GJP variant: I’m pretty sure—though too lazy to look up—this has been done in some AAU professional schools, but not the liberal arts, and also has been done in the liberal arts outside the AAU.

3. Grade inflation

Ehrlich/Simon [reversed]: By 2020, the percentage of students receiving a grade of ‘C’ will increase by more than 20% in any college or university ranked 50 or higher by US News and World Report (USNWR).

GJP: By 1 January 2016, at least ten institutions ranked in the top 100 USNWR list of either colleges or universities will be shown to have submitted false information.

Comments: Allowing reversal of the polarity, this question is also a slam-dunk, as the motto of the modern educational system is the “Dodo’s Verdict” in Alices’s Adventures in Wonderland:EVERYBODY has won, and all must have prizes.” [12]  Even those students spending their entire six undergraduate years majoring in substance abuse with a minor in sexual assault. How much of this due to USNWR rankings and how much to the “self-esteem” madness—supported by zero empirical research [16]—is unclear.

As for the mouse, I’ll look at the courses you took, but I’ll pay no attention to your grades: I want to see the code you’ve contributed to Github. [7]

The GJP version: Lots of this going on already, of course. The fun will really begin when the current oath of omertá breaks down and institutions start actively ratting each other out.

4. Degree-granting sport franchises

Ehrlich/Simon: By 2020, the average salary of the football and basketball coach in the 65 major university-based athletic franchises [8] will be at least three times the average salary of a full professor in the liberal arts, and at least 50% higher than the average salary of the most highly-paid administrator.

GJP: By 1 January 2016, inaction by the NCAA in responding to the University of North Carolina athletic grade scandal will lead to the restoration of Joe Paterno’s football record. Bonus prediction: through continued litigation and legal discovery, Penn State will recover a significant portion of the $60-million protection money paid to the NCAA.

Update 16 January 2015: Nailed it! That certainly didn’t take long. Issue of protection money is still open.

Comments: The Ehrlich-Simon merely projects existing trends. A more interesting question, I suppose, would be projecting the number of institutions which will follow the University of Alabama/Birmingham model of dropping out of this competition once it is clear they will not be part of the “The Sixty-Five.” Per Joe Nocera’s recent column, UAB’s decision is being denounced at the present, but given the millions in financial costs involved, and the sheer hopelessness of any secondary school competing in the new system, methinks we are witnessing a serious case of “Denial is not just a river in Egypt” here.

As for Penn State, if Joe Paterno’s record was vacated because he may or may not have known he had a serial pedophile on his staff, should not UNC have every single victory vacated during the eighteen-year period they were running an academic eligibility scam? Yes, I know you are laughing so hard you are choking on your egg-nog: of course the NCAA isn’t going to penalize UNC, so they will reverse the decision on Paterno. If Penn State’s lawyers can ever get some time free from fighting Penn State’s “We have to destroy the university in order to save it” elected members of their own Board of Trustees, they’ve already discovered enough incriminating emails from the NC-“You say `protection racket’ like that’s a bad thing…”-AA that the refund is probably all but in the bank. Curiously, almost to the dollar the amount reportedly paid in the settlements with Sandusky’s victims.

5. Growth of distance-learning alternatives

Ehrlich/Simon: By 2020, members of the AAU will have increased their revenue from distance-learning enterprises, including MOOCs, by at least 100% compared to 2014.

GJP: By 1 January 2016, at least one member of the AAU will have formalized a system where all large introductory courses in the social sciences, humanities and mathematics can be satisfied with a non-classroom-based alternative and will be actively encouraging students to use this option.

Comments: Any university which is not pursing distance-learning revenue streams needs a new administration. But almost all of them are, often with very substantial investments, and protestations by such institutions that MOOCs are a mere fad has, shall we say, more than a little of a David Copperfield aspect to it.

There is a very obvious model for established universities to monetize MOOCs—give away the instruction, validate (for a fee substantially less than the cost of residential tuition) the credentials based on successfully mastering the material—and it will be in place fairly soon. MOOC-based credentials have the further advantage over the traditional model by having open content, and presumably there will be fewer incentives to reward mere self-esteem.

This GPJ question for this topic is one of the hardest to call because unlike monetizing MOOCs, I don’t see how a major institution gets from the status quo to this vastly more efficient alternative. Theil is correct that these introductory courses are ripe for elimination: we do not need thousands of instructors providing, in person, nearly identical material to tens of thousands of students—at least a third of whom will show up for lectures [13]—using the old “sage on the stage” methodology which countless studies have shown is pedagogically one of the worst conceivable methods of instruction and has changed little, except with the addition of PowerPoint, since the times of Pierre Abelard.  These courses are, however, immensely profitable—at many institutions they provide about 50% of the tuition credit-hours—provided one has a supply of instructional cannon-fodder in the form of poorly-paid adjuncts and graduate students (most of whom—as noted above—have no chance of ever getting a tenured position).

Eliminating these courses would probably reduce total instructional labor requirements by at least a third [14], leading to a far more efficient model, but in the absence of a law-school-like collapse of student enrollments, particularly one driven by tuition concerns, it is hard to see how institutions could cross over to this model in the near term. Eliminating the fixed costs of tenured instructors, however, probably would be a necessary first step, but this will be gradual. [15]

6. Decline of law schools

Ehrlich/Simon: By 2020, enrollment in accredited law schools will be less than 50% of the level it was in 2010.

GJP: By 1 January 2016, at least one university which is a member of the AAU will begin the process of closing down its law school.

Comments: The extraordinarily rapid decline in law school enrollments is almost unique in the annals of academia, as U.S. law schools were caught in a nearly perfect storm where they faced simultaneously a technologically-induced decline in the demand for legal professionals, had evolved a system which, unlike medical schools, by most accounts does almost nothing to prepare one for either passing the bar exam or the actual practice of law, and had institutionalized tuition levels far above their marginal costs, leading to over-expansion. I would guess that Theil is anticipating that the liberal arts are similarly situated and thus poised for a comparable rapid fall. Consistent with Drezner, I do not think that is the case.

7. Problems with publications

Ehrlich/Simon: By 2020, the average lag between the first presentation of a research finding and its publication in a “top five” social science journal will be the same or longer than it was in 2014.

GJP: By 1 January 2016, one of the top five journals in political science, sociology or international relations will be shown to have published an article based on data which was largely fabricated.

Comments: The refereed publication system is horribly broken but I’ve seen little evidence that it will self-correct, even in the presence of blazingly dysfunctional incentive structures for institutions, which continue to require that their employees give away their intellectual property so that the institutions can buy it back at extortionate rates. All the while constantly complaining about the injustice of a system they have the power to change in a heartbeat. As I’ve noted before, were an individual to act in this manner, their financial affairs would be quickly turned over to a court-appointed guardian.

The likelihood of the discovery of fraud is not a criticism of those fields but a simple extrapolation from recent experience in the natural sciences, where such malfeasance is uncovered almost weekly, as well as with the opportunities for greater scrutiny that is developing with stricter replication norms.

Update 20 May 2015: Science isn’t exactly a social science journal but certainly counts as a top-five journal in general, so I think the retraction of the Green-LaCour article counts. The interesting thing to watch now is whether this has any cascade effects.

My predictions:

All of the Ehrlich/Simon questions are virtual certainties (allowing [3] as zero), except [7], where it is conceivable that on-line open-access alternatives would cause a change.

The short time frame on the GJP questions puts these at much lower probabilities, though that also makes them more interesting. GJP (and other prediction markets) work with probabilities—the eventual accuracy evaluated by a Breier score—and to show how rapidly I think the changes are trending, I’ll give those probabilities with deadlines of both 1 January 2016 and 1 January 2017

Question 1 Jan 2016 1 Jan 2017
Privatize public university 20% 40%
Abolish tenure in liberal arts 10% 30%
USNWR ranking fraud 70% 90%
Paterno decision reversed 70% 80%
NCAA shakedown refund 60% 80%
Distance learning intro courses 30% 50%
Law school demise 80% 100%
Publication fraud 40% 60%

Again, if you’ve got a lot of time on your hands, I have written way more extensively on these issues here.


1. The reference, as I’m sure you all immediately recognized, is to Karl Marx’s Eleven Theses on Feuerbach. My generation would smoke a lot of dope, score some LSD and then read Marx. Except for the losers, who read Ayn Rand. Drezner’s generation—though probably not Drezner—would smoke a lot of dope, score some cocaine, and read Derrida and Foucault. Except for the losers, who read Ayn Rand. The current generation, as best I can tell, have been taking powerful psychoactive prescription drugs since nursery school and can legally purchase THC-laced consumables in states with really nice mountains, if not in Federal districts deprived of home rule, so they just score six-packs of Red Bull and read Twitter. Except for the losers, who read Ayn Rand and occasionally found internet-based companies with capitalization in the billions of dollars.

2. Over-simplifying in a blog posting!: I’m shocked, shocked!

3. To say nothing of gullible parents living in constant fear that sending their loveable puppies anywhere but a USNWR top-ranked school—or, Thor-forbid, an affordable public university—will mean a waste of those years spent shuttling the little darlings to zither lessons starting at the age of three, astrophysics camp, investing in the horses and kit required to participate on polo and fox-hunting teams, all culminating in the child’s critically-acclaimed solo performance, as a high-school junior, of the whole of Schoenberg’s free atonal compositions, on the zither. Some fourteen pages of closely supervised accomplishments now being required for mere consideration for admission to schools in the USNWR top 50, or 20, or 2, or whatever.[11] And should that fail, and, Thor-forbid, should it fail so badly that your children receive degrees from public universities, they have nothing to look forward to but a lifetime of sleeping under bridges wrapped in discarded U-Haul packing blankets, standing on freeway median strips begging for quarters, and sniffing paint thinner stolen from highway construction sites. Yes, parents, that is the consequence of getting a degree from an institution that doesn’t leave you with $100,000 in tuition debt, yesiree Bob!: U.S. News and World Report and the college-applications industry tell me so!

Lux et veritas, my ass: numquam intermissum a stulto.

4. The count goes to zero when you pass a graveyard! This was before DVDs.

5. He says, pretending as always that he actually watched the show rather than the reading about it the next day.

6. Or was that Alex Trebek? Again pretending that I actually watch the shows.

7. Were I hiring. Which I’m not.

8. The emerging four professional-grade 16-member athletic conferences plus Notre Dame.

9. And, if anything more so, “The Greatest Generation.”

10. If you are not familiar with the AAU, these are the big dogs of the research university world, so if something changes here, it is a major deal. Almost everything on this list will probably happen at, or be done to, a non-AAU institution before it occurs in the AAU. So using the AAU as the reference group is setting the bar high.

11. And even all that will be for naught should accident of ancestry mean that admission would cause your elite target institutions to exceed their long-suspected quotas for Asians and Asian-Americans.

12. And Charles Dodgson, let us remember, taught at an elite academic institution: some things, one suspects, never change.

13. There are various ways one can force students to be physically present, though that merely provides them uninterrupted time for updating their Facebook—nowadays, Instagram—pages.

14. At present it is hard to say what this number will be in the long run, as we probably need another ten years of experience to determine the optimal mix of professional, peer, and machine interaction in a MOOC, and that mix will also, of course, vary by topic. At least two secondary financial effects also need to be sorted out: how much of the professional instruction will be institution-based rather than decentralized (which will affect pay rates), and how much can be saved by the elimination of “graduate programs” that currently exist almost solely to provide ill-paid instructional cannon fodder for the generally farcical “discussion sections” of the introductory courses.

15. Theil, I would imagine, sees the rise of start-ups following the entirely new models—how higher education would look were it being invented today rather than 11501810 or 1950—as the solution, and some of this is happening, perhaps most dramatically (and certainly coherently) in the Minerva Schools. There are, however, at least two serious impediments to this

  • It’s a bad neighborhood: The existing for-profit universities have had a very nasty tendency to be little more than quasi-criminal enterprises whose business model rests on extracting funds from the public purse under the protection of for-profit members of Congress while saddling their clients with unconscionable and crippling levels of debt. Which is to say, acting like financial institutions.
  • The instant that any of the big dogs with established reputations, several dating back centuries, jump into this game—the costs of entry are quite low, and these institutions will become involved only when they are certain their models will scale—any institutions without such reputations will be at a very serious disadvantage, and could be wiped out almost overnight. Not an enterprise for the faint of heart. Or, possibly, the strong of brain.

16. My favorite result being a study of high schools that found the students with the greatest self-esteem were consistently the drug dealers.

Posted in Higher Education | Leave a comment

Hey, who you calling a ‘Crusader’?!

Given I analyze political conflict for a living—yes, you can actually get paid for that sort of thing—I’ve spent relatively little time on this blog writing about such topics. Avoiding a busman’s holiday, I suppose, or  maybe, just maybe, I find it less depressing railing against degree-granting sports franchises, corporations whose business model is taking your intellectual property for free and then selling it back to you, or the institutionalized insanity that supports that sort of thing.

But as this past week was spent first coding the activities of ISIS [1]—and they are really, really not nice people at all—and then participating in a workshop on the coevolution of tactics of violent groups, it struck me that not only have we seen the ISIS pattern before, but the number of parallels are quite remarkable: ISIS looks like the First Crusade and the subsequent Latin Kingdoms of the twelfth century.[2]

There is, of course, exquisite irony in this given that one of the prime rhetorical motifs of ISIS, as well as various al-Qaeda franchises for the past thirty years, has been arguing that they are opposing “Crusaders”, often including Israel in that category, a bit of an anachronism given the persistent Crusader penchant for getting into the swing of things by first slaughtering Jews in the Rhine Valley. And I’m not trying to use the “I don’t have cooties, you have cooties”  rhetorical technique, which most of us learn quickly on school playgrounds, but eventually out-grow unless experiencing the misfortune of winding up in the U.S. Congress, where this has become virtually the only rhetorical technique. No, I mean this has a serious historical analogy with, of course, the usual caveat that “history doesn’t repeat itself, but it rhymes.” [3]

Factors First Crusade and the Latin Kingdom of Jerusalem 1096-1187 ISIS 2014
Opportunity provided by weak states Pressure on Byzantium Empire (particularly the catastrophic military loss at Manzikerk in 1071) and the Abbasid Caliphate by the expanding power of the Seljuk Turks Syrian civil war and Sunni alienation from Shia regime in Baghdad. Though ultimately the parallel to the disruptive role of the Seljuk Turks can be traced to George “Mas’ud” Bush and his merry band of think-tank inspired ghazis
Religious motivators Urban II, Peter the Hermit, etc
Internet not required.
Numerous self-styled jihadi leaders. Internet enhanced.
Religious reward Temporal remission of sins Some ambiguous heavenly reward, possibly involving access to virgins and/or white raisins
Ample supply of seasoned fighters Europe anxious to divert the attentions of poorly supervised young men with no prospects of gainful employment or inheritance but nonetheless wielding sharp objects with reckless enthusiasm Multiple generations of experienced jihadi fighters dating back to the Soviet-Afghan War, with almost uninterrupted opportunities in between
Multinational, decentralized leadership Particularly true of the First “Baron’s” Crusade, which was the only one that was militarily successful [9] See above, with additional recruiting in 2014 after establishing territorial control
Excessive use of violence Contrast massacres following Crusader capture of Jerusalem with Salah-ad-Din’s later treatment of the city. They couldn’t post videos of this, but based on the approving treatment of the slaughter by most contemporary chroniclers, they certainly would have were they able. Trademark. With videos.
Apparently come out of nowhere and rapidly establish extended territorial control Which is how it looked from the perspective of Baghdad, Damascus and Cairo Which is how it looks from the perspective of Baghdad, Damascus and Washington
Idealized model from distant history Some Christian variant on the kingdom of David and Solomon, apparently Caliphate


So far, so bad. But after a rather inauspicious start, the Crusaders settled down and expansion stopped completely—despite concerted efforts—with the Latin Kingdoms beginning to contract starting with the fall of Edessa in 1149. What accounted for the decline?

  • Inability to maintain a sufficient colonizing or military force: once the First Crusade finished, most surviving Crusaders went home to deal with whatever mischief had occurred in their absence.[11] The remaining military force was not even sufficient to maintain control over existing territories—and hence alliances of convenience were quickly made with local Moslem elites—much less expand. Attempts to attract new settlers were almost entirely unsuccessful, in large part due to immigration restrictions: The nobility across Europe had exactly zero interest in having their serfs move to Palestine. Or anywhere.
  • Local rulers “went native,” or at least turned down the level of violence, and generally ruled with at least modest levels of responsibility, particularly given the rather low standards of the day. Politically, ruthless rampaging and terror tend to have a rather limited shelf life, with the apparent exception of the Assyrians.
  • Surviving adjacent Moslem states got their military act together, notably Damascus under the Zangas, and the Crusaders no longer had the advantage of surprise: the attempt to take Damascus in the Second Crusade was an abysmal failure.
  • In a much slower process, the Muslims eventually achieved a two-front political consolidation under Salah-ad-Din.
  • Non-stop dynastic squabbles weakened the Latin leadership internally, even in the face of severe external threats. This was further complicated by external meddling from Europeans—political and ecclesiastical—with little or no knowledge of the local conditions. Assisted by the equally unenthusiastic Byzantines who did understand the local conditions.
  • A few out-of-control elements—famously, Raynald of Chatillon—wrecked efforts to maintain truces and other pragmatic compromises in the face of an increasingly weak position.

How many of these conditions might we expect to see affect ISIS in some modern form?: pretty much every single one, and every one of these points, translated into a modern era, becomes a potential leverage point, as David Ignatius pointed out last week in the Washington Post. Such policies probably have a reasonably good chance of success because, through the lens of history, we can “see around the corner.” Or at least see through a mirror, however dimly.

Lots of similarities, but two obvious differences:

Most conspicuously, due largely to changes in technology, the time frame has sped up enormously, probably by a factor of at least 10: I’d put the current ISIS situation as similar to that of the Latin Kingdom around perhaps 1105. Correspondingly, due to globalization the ability to modify resource flows to meet the threat once the political situation has stabilized is also much faster: I rather doubt ISIS is going to last 90 years, or even 90 months.

Second, there’s one conspicuous feature of the Latin Kingdoms we have not seen yet: the emergence of formally structured autonomous military movements independent of the state, as occurred with the two major military monastic orders, the Templars and the Hospitallers.[10] Unless, in effect, we’ve already seen that development in the form of al-Qaeda, but al-Qaeda generally used a fairly unstructured franchise model rather than the highly structured bureaucratic model of the monastic orders, which arguably were the most “modern” institutions of their time. Alternatively, the “swarm”/franchise model of al-Qaeda might itself be an effective adaptation in the current environment.

So, coincidence, policy-relevant, or merely yet another over-generalized rant by an aging amateur medievalist with way too much time on his hands and Philip Daileader’s Great Courses lectures on audio while stalled in traffic on I-66?[7] Well, clearly the last, but to the extent that analogical reasoning is a valid analytical tool in human behavior [8], the momentum on this one is pretty strong and, for example, ISIS’s weakness in numbers has long been noted.

Again, history does not repeat itself, it only rhymes, but as with the Latin Kingdoms, I see the future of ISIS as the dustbin of history, not a new caliphate. And that can’t happen too soon.


1. I will not indulge in the silly rhetorical game of calling them ‘ISIL’: they are holding territory in the Tigris-Euphrates Valley, which is not part of the Levant in most historical uses of that word. To say nothing of the political agenda underlying the campaign of pretending that they are in the Levant, which I’m even less fond of.

2. In the real Levant, not ISIL Levant.

3. But Phil, don’t you realize that “jihad” and “crusade” are exactly the same thing?—Crusaders simply copied the Islamic “jihad” model! Or was it the other way around?

Uh, well, no, it isn’t that simple. It isn’t anywhere close to that simple. The two movements clearly influenced each other, presumably initially with Christian envy of the Islamic military expansion in the seventh and eighth centuries, though this was in the quite distant past by the time Urban II preached the Crusade. And as we see with ISIS, and countless other Islamic military rivalist efforts in the past two or three centuries, Islamic invocation of the need to counter Christian “Crusader” colonial efforts is common, motivated in part by those efforts helpfully invoking Crusader imagery themselves, as I can attest having been raised with ample exposure to Protestant missionary appeals.

Once one gets into the specifics, however, comparisons begin to break down because both concepts have been thoroughly mutable across centuries and in my [less systematic than it should be] reading, those justifications—and allowing that at least some Crusaders/jihadis sincerely believed them—have generally been driven by make-it-up-as-you-go-along theological approaches that from the outset have been quite distinct in the two religions.

Specifically, the individual motivation for a Christian warrior to engage in a crusade was to gain a temporal indulgence for the remission of sin, concern which plays a far greater role in Christianity than in Islam (or pretty much any other religion). The collective motivation for jihad in Islam, in contrast, was to expand the territorial area within which Islam could be practiced—the Dar al-Islam, or Abode/Zone of Peace—but, contrary to common representations in the West, this is not a central part of Islam, and in particular it is not one of the Five Pillars. In most orthodox Islamic theology, “jihad” is a fairly minor and decidedly ambiguous concept and, Fox News notwithstanding, certainly not central.

Where things start getting complicated—and mutual influence could well be relevant here—is when these two motivations start intermixing. In particular, after the Latin Kingdoms fail in the [real] Levant, the Crusading concept is moved to Europe where it is, indeed, employed with the primary purpose of expanding the territorial reach of [Latin] Christianity, specifically against Moslems in Spain, pagans in the Baltics, Cathars in the south of France, and eventually against whatever Catholic dynasties the Papacy happened to be at odds with at any given moment.[13] The indulgences thing also got a tad out of hand, eventually leading to that unpleasantness in Wittenberg in 1517, which, man, went like totally viral.[4]

Contemporary jihadis, meanwhile, have taken to emphasizing the prospect of heavenly reward, which is beginning to look at least similar to the Christian concept of remission of sin, though theologically—and by the way, not only am I not a lawyer, I am not a theologian [5], so if you suffer eternal damnation following my advice, I take no responsibility: please consult on issues dealing with eternal damnation and perpetual hellfire with an experienced and properly trained authority from your chosen faith.[12] I digress…—as I was saying, theologically that is distinct given Islam’s greater emphasis on practice compared to Christianity’s emphasis on the primacy of sin (individual and collective) and the difficulties escaping the consequences thereof.

And all this before we get into such issues as the legitimacy of forced conversion: orthodox Islam pretty clearly rejects this but that point has been lost on numerous jihadis across the centuries; Christianity is more ambivalent due to the sacramental power of baptism. And the issue of tolerance for “peoples of the book”: orthodox Islam is quite unambiguous on the importance of this but again, that point has been lost on numerous jihadis; there is nothing comparable in Christianity, which instead saw the development of anti-Semitism from about the second century forward, and ironically those anti-Semitic arguments have been adopted pretty much totally by contemporary radical Islam.

But if you are still reading this—yes, both of you—you really should be looking into the serious scholarship on these issues, and there is a fair amount of it, though it takes a bit of digging —and not trying to get the information off a blog. Really.

And as for the practical manifestations of Crusade versus jihad for those on the ground: alas, pretty much total convergence here. You are sitting in your village minding your own business and along comes a heavily armed horde comprised of a few folks probably sincerely motivated by radical theological interpretations accompanied by a whole bunch of opportunistic hell-raisers bent on rape and pillage [6]. You will almost certainly not have a very nice day.

Which I will then code.

4. Without the internet. But moveable type helped. Though moveable type by most accounts got the indulgence thing totally out of control in the first place.

5. Though in recognition of the season, I repeat—as in link to—my appeal that we all return to respecting the spirit of Yule rather than obsessing on how much crap we can purchase at WalMart and Best Buy.

6. These days both ISIS and Boko Haram seem particularly into the “rape” part, possibly an unpleasant side effect of the dominant media available on the internet.

7. I-66/stalled in traffic: I repeat myself here, of course.

8. The geeks making billions of dollars predicting your likely behaviors based on your activities on the web also believe it is the case. Shortly after reading this you will probably start getting Amazon ads for swords, armor, and group excursions to the Holy Land. Or suggestions that you join the jihad from your friendly local DHS entrapment specialist.

9. Frederick II’s Sixth Crusade actually secured—through negotiation—access to Jerusalem for pilgrims and Christian control of Bethlehem and  Nazareth. Which only pissed off Pope Gregory IX, who had excommunicated Frederick. Anticipating Goldfinger by several centuries, His Holiness’s attitude was “I don’t expect you to secure Jerusalem, Mr. Hohenstaufen. I expect you to die!”

10. And eventually, implementing the Crusades against pagans in northeastern Europe, those beloved Teutonic Knights.

11. By the Third Crusade, note in particular the interactions between Lackland, John and Lion-hearted, Richard t.

12. I’m sticking with Odin, who appears particularly skilled at causing lightening-induced delays to connecting flights at Dulles which would otherwise have taken off without me. And that whole crow thing is really cool.

13. Kicking off this theme, the [Christian] Fourth Crusade famously besieged and then sacked [Christian] Constantinople, but the motivations for this were almost exclusively commercial rather than theological. The as-ever-subtle Pope Innocent III was furious with this outcome, noting among other points “[The Greek Christians will see] in the Latins [Crusaders] only an example of perdition and the works of darkness, and with reason, detest the Latins more than dogs.” I believe we can also infer from this that Innocent III did not exhibit any particular fondness for dogs.

Posted in Politics | Leave a comment