Seven reflections on work—mostly programming—in 2020

Reading time: Uh, I dunno: how fast do you read? [0]

Well, it’s been a while since any entries here, eh? Spent much of the spring of 2019 trying to get a couple projects going that didn’t work out, then most of the fall working intensely on one that did, and also made three trips to Europe this year: folks, that’s where the cutting edge has moved on instability forecasting. And on serious considerations of improving event data collection: please check out https://emw.ku.edu.tr/aespen-2020/: Marseille in early summer, 4 to 8 page papers, and an exclusive focus on political event data!

All distractions, and what has finally inspired me to write is an excellent blog entry—found via a retweet by Jay Ulfelder then re-retweeted by Erin Simpson, this being how I consume the Twittersphere—by Ethan Rosenthal on remote work in data science:

https://www.ethanrosenthal.com/2020/01/08/freelance-ds-consulting/

While Rosenthal’s experience—largely working with private sector start-ups, many retail—would seem quite distant from the sort of projects Jay and I have usually worked on (sometimes even as collaborators), Jay noted how closely most of the practical advice paralleled his own experience [1] and I found exactly the same, including in just the past couple of  months:

• Desperately trying to persuade someone that they shouldn’t hire me
• Doing a data audit for a proposed project to make sure machine-learning methods had a reasonable chance of producing something useful
• Pipelines, pipelines, pipelines
• The importance and difficulties of brutally honest estimates

and much more. Much of this is consistent with my own advice over almost seven years of remote contracting—see here, here, and here—but again, another view from a very different domain, and with a few key differences (e.g. Rosenthal works in Manhattan. New York, not Kansas).

And having decided to comment on a couple of core points from Rosenthal, I realized there were some other observations since the spring of 2019—yes, it has been that long—and came up with the requisite seven, and meanwhile my primary project is currently on hold due to issues beyond my control involving a rapacious publisher in an oligopoly position—things never change—so here we go…

Deep work is a limited and costly resource

Rosenthal has one of the best discussions of the nuances of contractor pricing that I’ve seen. Some of this covers the same ground I’ve written on earlier, specifically that people on salary in large organizations—and academia is probably the worst offender as they rarely deal with competitive pricing or any sort of accountability [2], but people whose life experience has been in government and corporations can be just as clueless—have no idea whatsoever of what their time actually costs and how much is totally wasted. Rosenthal echoes the point I’ve made several times that unless you carefully and completely honestly log your time—I’ve done so, for decades, at half-hour increments, though I still have difficulties with the “honestly” part, even for myself—you have no idea how much time you are actually working productively. People who claim to do intellectually demanding “work” for 80 hours a week are just engaging in an exercise of narcissistic self-deception, and if you estimate level-of-effort for a project in that mindset, you are doomed. 

Where Rosenthal’s discussion is exceptional—though consistent with a lot of buzz in the remote-work world of late—is distinguishing between “deep” and “shallow” work and arguing that while “deep” work should be billed at a high rate—the sort of rate that causes academics in particular to gasp in disbelief—you can’t do it for 40 hours a week (to say nothing of the mythical 80-hours-per-week), and you are probably lucky to sustain even 30 hours a week beyond occasional bursts.[3] So, ethically, you should only be charging your top rate when you are using those deep skills, and either not charge, or charge at a lower rate, when you are doing shallow work. My experience exactly. 

Deep work can be really exhausting! [6] Not always: in some instances, when one has a task that fits perfectly into the very narrow niche where the well-documented and much-loved “flow” experience occurs, it is exhilarating and time flies: you almost feel like you should be paying the client (no, I don’t…). But, bunkos, in real work on real problems with real data, most of your time is not spent in a “flow” state, and some of it can be incredibly tedious, while still requiring a high skill set that can’t really be delegated: after all, that’s why the client hired you. In those instances, you simply run out of energy and have to stop for a while. [7]

Rosenthal also argues for the rationality of pricing by the project, not by the hour, particularly when working on software that will eventually be transferred to the client. The interests of client and contractor are completely aligned here: the client knows the cost in advance, the contractor bears the risk of underestimating efforts, but also has greater knowledge about the likely level of effort required, and the contractor has incentives to invest in developments that make the task as efficient as possible, which will then eventually get transferred (or can be) to the client. There’s no downside! 

Yet it’s remarkably hard to get most clients—typically due to their own bureaucratic restrictions—to agree to this, due to most organizations still having a 19th century industrial mindset where output should be closely correlated with time spent working. [8] Also for some totally irrational reason—I suppose a variant on Tversky and Kahneman’s well-researched psychological “loss-aversion”—project managers seem to be far more concerned that the contractor will get the job done too quickly, thus “cheating” them on a mutually-agreed-upon amount, while ignoring the fact the otherwise they’ve given the contractor zero incentive to be efficient. [9] Go figure. 

Remote work is cool and catching on

I’ve worked remotely the entire time I’ve been an independent contractor, so it’s fascinating to watch this totally catching on now: The most common thing I now hear when talking with CTOs/VPs-of-Engineering in the Charlottesville area is either that their companies are already 100% remote, or they are heading in that direction, at least for jobs involving high-level programming and data analytics. The primary motivator is the impossibility of finding very idiosyncratic required skill sets locally, this being generally true except in three or four extraordinarily expensive urban areas, and often not even there.

But it is by no means just Charlottesville or just computing, as two recent surveys illustrate:

https://usefyi.com/remote-work-report/

https://buffer.com/state-of-remote-work-2019

While there are certainly tasks which don’t lend themselves to remote work, I’ll be curious to see how this finally settles out since we’re clearly early in the learning curve. [10]

Three observations regarding those surveys:

1. The level of satisfaction—noting, of course, that both are surveying people doing remote work, not the entire workforce—is absolutely stunning, in excess of 90%: it’s hard to think of any recent workplace innovation that has had such a positive reception. Certainly not open office plans!
2. I was surprised at the extent to which people work from home [10a], as I’ve argued vociferously for the importance of working in an office away from home. At least three things appear to account for this difference: First, flexibility in childcare is a major factor for many remote workers that is not relevant to me. Second, I’m doing remote work that pays quite well, and the monthly cost of my cozy little office is covered in my first three or four hours of deep work, which would not be true for, say, many editing or support jobs. Third, from the photos, a lot of people are in large suburban houses, probably with a so-called “bonus room” that can be configured as customized workspace, whereas my residence is in an older urban neighborhood of relatively small mid-20th-century houses.
3. People are appreciating that remote work can be done in areas with relatively low real estate prices and short commuting times: my 1-mile “commute” is about 20 minutes on foot and 5 minutes on a Vespa, with further savings in our family owning just one car. If remote work continues to expand, this may have discernible redistributive effects: as The Economist notes on a regular basis, the high professional salaries in urban areas are largely absorbed by [literal] rents, and since remote work is generally priced nationally, and sometimes globally, there is nothing like getting Silicon Valley or Northern Virginia wages while living outside those areas. [11] This is apparently leading to a revival of quite a few once-declining secondary urban centers, and in some instances even rural areas, where the money goes a really long way.

All this said, your typical 19th-century-oriented [typically male] manager does not feel comfortable with remote work! They want to be able to see butts on seats! And at least 9-to-5! This is frequently justified with some long-reimagined story where they assisted a confused programmer with a suggestion [12], saving Earth from a collision with an astroid or somesuch, ignoring that 99% of the time said programmer’s productivity was devastated by their interruptions. But managerial attitudes remain “If it was good enough for Scrooge and Marley, it’s good enough for me.” Still a lot of cultural adaptation to be done here.

The joy of withered technologies 

From David Epstein. 2019. Range: Why generalists triumph in a specialized world. NY: Riverhead Books, pg. 193-194, 197. [12a]

By “withered technology”,  [Nintendo developer Gunpie] Yokoi meant tech that was old enough to be extremely well understood and easily available so it didn’t require a specialist’s knowledge. The heart of his “lateral thinking with withered technology” philosophy was putting cheap, simple technology to use in ways no one else considered. If he could not think more deeply about new technologies, he decided, he would think more broadly about old ones. He intentionally retreated from the cutting edge, and set to monozukuri [“thing making”]. 

When the Game Boy was released, Yokoi’s colleague came to him “with a grim expression on his face,” Yokoi recalled, and reported that a competitor’s handheld had hit the market. Yokoi asked him if it had a color screen. The man said it did. “Then we’re fine.” Yokoi replied.

I encountered this over the past few months when developing customized coding software for the aforementioned data collection project. While I certainly know how to write coding software using browser-based interfaces—see CIVET, as well as a series of unpublished customized modules I created for coding the Political Instability Task Force Worldwide Atrocities Dataset—I decided to try a far simpler, terminal-based interface for the new project, using the Python variant of the old C-language curses library, which I’d learned back in 2000 when writing TABARI’s coding interface.

The result: a coding program that is much faster to use, and probably physically safer, because my fingers never leave the keyboard, and most commands are one or two keystrokes, not complex mouse [13] movements requiring at least my lower arm and probably my elbow as well. Thus continuing to avoid—fingers crossed, but not too tightly—the dreaded onset of carpal tunnel syndrome which has afflicted so many in this profession.

And critically, the code is far easier to maintain and modify, as I’m working directly with a single library that has been stable for the better part of three decades, rather than the multiple and ever-changing layers of code in modern browsers and servers and the complex model-template-view architectural pattern of Django, as well as three different languages (Python, php and javascript). Really, I just want to get the coding done as efficiently as possible, and as the system matured, the required time to code a month of data dropped almost in half. Like Yokoi, frankly I don’t give a damn what it looks like.

Just sayin’…and we can generalize this to…

The joy of mature software suites: there is no “software crisis”

We have a local Slack channel frequented mostly by remote workers (some not local) and in the midst of the proliferation of “so, how about them 2010s?” articles at the end of 2019, someone posted a series of predictions made on the Y-Combinator Slack-equivalent back in 2010.

Needless to say, most of these were wrong—they did get the ubiquity of internet-enabled personal information devices correct, and some predictions are for technologies still in development but which will likely happen fairly soon—making the predictable errors one expect from this group: naive techno-optimism and expectation of imminent and world-changing “paradigm shifts,” and consistently underestimating the stability of entrenched institutions, whether government, corporate—the demise, replacement, or radical transformation of Microsoft, Facebook, Google, and/or Twitter was a persistent theme—technological or social.[14] But something caught my attention

…in the far future, software will be ancient, largely bug free, and not be changed much over the centuries. Information management software will evolve to a high degree of utility and then remain static because why change bug free software that works perfectly.  … What we think of programming will evolve into using incredible high level scripting languages and frameworks. Programs will be very short.

This hasn’t taken anything close to centuries because in statistics (and rapidly developing, machine-learning), whether R or the extensive Python packages for data analytics and visualization, that’s really where we are already: these needs are highly standardized so the relevant code—or something close enough [15]—is already out there with plenty of practical use examples on the web, so the scripts for very complex analyses are, indeed, just a couple dozen lines.

What is remarkable here—and I think we will look back at the 2010s as the turning point —is that we’ve now evolved (and it was very much an organic evolution, not a grand design) a highly decentralized and robust system for producing stable, inexpensive, high quality software that involves the original ideas generally coming from academia and small companies, then huge investments by large corporations (or well-funded start-ups) to bring the technology to maturity (including eventually establishing either formal or de facto standards), all the while experiencing sophisticated quality control [17]  and pragmatic documentation (read: Stack Overflow). This is most evident at the far end of the analytical pipeline—the aforementioned data analytics and visualization—but, for example, I think we see it very much at work in the evolution of multiple competing frameworks for javascript: this is a good thing, not a bad thing, if sometimes massively annoying at the time. The differences between now and even the 1990s is absolutely stunning.

So why the endless complaints about the “software crisis?” Two things I’m guessing. First, in data analytics we still have, and will always have, a “first mile” and “last mile” problem: at the beginning data needs to be munged in highly idiosyncratic ways in order to be used with these systems, and that process is often very tedious. At the end stages of analysis, the results need to be intelligently presented and interpreted, which also requires a high level of skills often in short supply. And then there’s the age-old problem that most non-technical managers hate skilled programmers, because skilled programmers don’t respond predictably to the traditional Management Triad—anger, greed, and delusion—and at the end of the [working] day, far too many IT managers really just want to employ people, irrespective of technical competence, they will feel comfortable with doing vodka shots in strip clubs. That has only barely changed since the 1990s. Or 1970s.

Whoever has the most labelled cases wins

Fascinating Economist article (alas, possibly paywalled depending on your time and location):

https://www.economist.com/technology-quarterly/2020/01/02/chinas-success-at-ai-has-relied-on-good-data

arguing that the core advantage China has in terms of AI/ML is actually labelled cases, which China has built a huge infrastructure for generating in near-real-time and at low cost, rather than in the algorithms they are using: 

Many of the algorithms used contain little that is not available to any computer-science graduate student on Earth. Without China’s data-labelling infrastructure, which is without peer, they would be nowhere.

Also see this article Andy Haltermann alerted me to: https://arxiv.org/pdf/1805.05377.pdf

Labelled cases—and withered technologies—become highly relevant when we look at the current situation for the automated production of event data. All of the major projects in the 2010s—BBN’s work on ICEWS, UT/Dallas’s near-real-time RIDIR Phoenix, UIUC Cline Center Historical Phoenix—use the parser/dictionary approach first developed in the 1990s by the KEDS and VRA-IDEA projects, then followed through to the TABARI/CAMEO work of the early 2000s. But seen from the perspective of 2020, Lockheed’s successful efforts on the original DARPA ICEWS (2008-2011) went with a rapidly-deployable “withered technology”—TABARI/CAMEO—and initially focused simply on improving the news coverage and actors dictionaries—both technically simple tasks—leaving the core program and its algorithms intact, even to the point where, at DARPA’s insistence, the original Lockheed JABARI duplicated some bugs in TABARI, only later making some incremental improvements: monozukuri + kaizen. Only after the still-mysterious defense contractor skullduggery at the end of the research phase of ICEWS—changing the rules so that BBN, presumably intended as the winner in the DARPA competition all along, could now replace Lockheed—was there a return to the approach of focusing on highly specialized coding algorithms.

But that was then, and despite what I’ve written earlier, probably the Chinese approach—more or less off-the-shelf machine learning algorithms [18], then invest in generating masses of training data (readily available as grist for the event data problem, of course)—is most appropriate. We’ll see. 

David Epstein’s Range: Why generalists triumph in a specialized world is worth a read

Range is sort of an anti-Malcolm-Gladwell—all the more interesting given that Gladwell, much to his credit, favorably blurbs it—in debunking a series of myths about what it takes to become an expert. The first of two major take-aways—the book is quite wide-ranging—are that many of the popular myths are based on expertise gained in “kind” problems where accumulated past experience is a very good guide to how to get a positive outcome in the future: golf, chess, and technical mastery of musical instruments being notoriously kind cases.[19] In “wicked” problems, concentrated experience per se isn’t much help, and, per the title of the book, generalists with a broad range of experience and experimentation in many different types and levels of problems excel instead.

The other myth Epstein thoroughly debunks is the “10,000-hours to expertise” rule extolled by Gladwell. For starters, this is largely an urban legend with little systematic evidence to back it.  And in the “well, duh…” category, the actual amount of time required to achieve expertise depends on the domain—starting with that kind vs. wicked distinction—and the style of the experience/training (Epstein discusses interesting work on effects of mixing hard and easy problems when training), and on the individual: some people absorb useful information more quickly than others.

So where is programming (and data analytics) on this perspective? Curiously with aspects on both ends. Within a fixed environment, it is largely “kind”: the same input will always produce the same output [20]. But the overall environment, particularly for data analytics in recent years, is decidedly wicked: while major programming languages change surprisingly slowly, libraries and frameworks change rapidly and somewhat unpredictably, and this is now occurring in analytics (or at least predictive analytics) as well, with machine-learning supplanting—sometimes inappropriately—classical statistical modeling (which by the 1990s had largely degenerated to almost complete reliance on variants of linear and logistic regression [16]) and rapid changes can also occur in machine-learning, as the rapid ascendency of deep learning neural networks has shown.  

As for what this means for programmers, well…

The mysteries of 1000-hour neuroplasticity

I’ll finish on a bit of a tangent, Goleman and Davidson’s Altered Traits: Science Reveals How Meditation Changes Your Mind, Brain, and Body (one hour talk at Google here).

Goleman and Davidson are specifically interested in meditation methods that have been deliberately refined over millennia to alter how the brain works in a permanent fashion: altered traits as distinct from temporarily altered states in their terminology, and these changes now can be consistently measured with complex equipment rather than self-reporting. But I’m guessing this generalizes to other sustained “deep” cognitive tasks.

What I find intriguing about this research is what I’d call a “missing middle”: There is now a great deal of meditation research on subjects with very short-term experience—typically either secular mindfulness or mantra practice—involving a few tens of hours of instruction, if that, followed by a few weeks or at most months of practice of varying levels of consistency. Davidson, meanwhile, has gained fame for his studies, in collaboration with the Dalai Lama, on individuals, typically Tibetan monastics, with absolutely massive amounts of meditation experience, frequently in excess of 50,000 lifetime hours, including one or more five-year retreats, and intensive study and training.[21]

My puzzle: I think there is a fair amount of anecdotal evidence that the neuroplasticity leading to “altered traits” probably starts kicking in around a level of 1,000 to 2,000 lifetime hours of “deep” work, and this probably occurs in a lot of domains, including programming. But trying to assess this is complicated by at least the following issues

• reliably keeping track of deep practice over a long period of time—a year or two at least, probably more like five years, since we’re looking at time spent in deep work, not PowerPoint-driven meetings or program/performance reviews [22]—and standardizing measures of its quality, per Epstein’s observations in Range
• standardizing a definition of “expertise”: We all know plenty of people who have managed for decades to keep professional jobs apparently involving expertise mostly by just showing up and not screwing up too badly too conspicuously too often
• Figuring out (and measuring for post-1000-to-2000-hour subjects) baselines and adjusting for the likely very large individual variance even among true experts
• Doing these measures with fabulously expensive equipment the validity of which can be, well, controversial. At least in dead salmon.

So, looking at what I just wrote, maybe 1000 to 2000 hour neuroplasticity, if it exists, will remain forever terra incognita, though it might be possible in at least a few domains where performance is more standardized: London taxi drivers again.[reprise 21] But I wonder if this addresses an issue one finds frequently in fields involving sustained mental activity, where a surprisingly high percentage of elaborately-trained and very well compensated people drop out after five to ten years: Is this a point where folks experiencing neuroplasticity—and learning how to efficiently use their modified brains, abandoning inefficient habits from their period of learning and relying more on a now-effective “intuition,” setting aside the proverbial trap to focus on the fish—find tasks increasingly easy, while those who have not experienced this change are still tediously stumbling along, despite putting in equivalent numbers of hours? Just a thought. So to speak.

Happy New Year. Happy 2020s.

Footnotes

0. And about all those damn “footnotes”…

1. And transcends the advice found in much of the start-up porn, which over-emphasizes the returns on networking and utilizing every possible social media angle. Rosenthal does note his networks have been great for locating free-lance jobs, but these were networks of people he’d actually worked with, not social media networks. 

2. By far the worst experience I’ve had with a nominally full-time—I momentarily thought I’d use the word “professional,” but…no…—programmer I was supposedly collaborating with—alas, with no authority over them—was in an academic institution where the individual took three months to provide me with ten lines of code which, in the end, were in a framework I decided wouldn’t work for the task, so even this code was discarded and I ended up doing all of the coding for an extended project myself. The individual meanwhile having used that paid time period to practice for a classical music competition, where they apparently did quite well. They were subsequently “let go”, though only when this could be done in the context of a later grant not coming through.

As it happens, I recently ran into the now-CTO of that institution and, with no prompting from me, they mentioned the institution had a major problem with a large number of programmers on payroll, for years, who were essentially doing nothing, and quite resented any prospects of being expected to do anything. So it was in the institutional culture: wow! Wonder how many similar cases there are like this? And possibly not only in academia.

3. Note this is one of the key reasons programming death marches don’t work, as Brooks initially observed and, Yourdon later elaborated in more detail. [4] In programming, the time you “save” by not taking a break, or just calling it quits for the day, can easily, easily end up genuinely costing you ten or more times the effort down the road. [5]

4. I gather if I were a truly committed blogger, apparently there are ways I could monetize these links to Amazon and, I dunno, buy a private island off the coast of New Zealand or somesuch. But for now they are just links…

5. As with most engineering tasks but unlike, say, retail. Or, surprisingly, law and finance if their 80-hour work weeks are real. Medicine?: they bury their mistakes.

6. I’m pretty sure Kahneman and Tversky did a series of experiments showing the same thing. Also pretty sure, but too tired to confirm, Kahneman discusses these in Thinking Fast and Slow. (Google talk here )

7. I suppose nowadays taking stimulating drugs would be another response. Beyond some caffeine in the morning (only), not something I do: my generation used/uses drugs recreationally, not professionally. But that may just be me: OK, boomer.

8. Far and away the best managed project I’ve been involved with not only paid by the sub-project, but paid in advance! This was a subcontract on a government project and I was subsequently told on another government subcontract that, no, this is impossible, it never happens. Until I gave up arguing the point, I was in the position of discussions with Chico Marx in Duck Soup: “Well, who you gonna believe, me or your own eyes?” Granted, I think I was involved in some sort of “Skunkworks” operation—it was never entirely clear and the work was entirely remote beyond a couple meetings in conference rooms in utterly anonymous office parks—but still, that pre-paying project went on for about two years with several subcontracts. 

9. The “cost-plus” contracts once [?] common in the defense industry are, of course, this moral hazard on steroids.

10. On a learning curve but definitely learning: one of the fascinating things I’ve seen is how quickly people have settled on two fundamental rules for remote meetings :

• Everyone is on a remote connection from their office, even if some of the people are at a central location: meetings with some people in a conference room and the rest coming in via video are a disaster
• Video is on for everyone: audio-only is a recipe for being distracted

These two simple rules go most of the way to explaining why remote meetings work with contemporary technology (Zoom, Hangout) but didn’t with only conference room video or audio-only technology: OMG “speaker phones,” another spawn of Satan or from your alternative netherworld of choice.

10a. So the typical remote worker uses a home office, and tech companies are moving to 100% remote, and yet in downtown Charlottesville there are currently hundreds of thousands of square feet of office space under construction that will be marketed to tech companies: am I missing something here?

11. On the flip side, there is also nothing like getting Mumbai wages while living in San Francisco or Boston.

12. Uh, dude, that’s what Slack and StackOverflow are used for now…

12a. Actual page references evidence that I bought the physical book at the National Book Festival after listening to Epstein talk about it,  rather than just getting a Kindle version.

13. I’ve actually used a trackball for decades, but same difference. Keyboard also works nicely in sardine-class on long flights.

14. One prediction particularly caught my attention: “A company makes a practice of hiring experienced older workers that other companies won’t touch at sub-standard pay rates and the strategy works so well they are celebrated in a Fortune article.” Translation: by 2020, pigs will fly.

15. E.g. I was surprised but exceedingly pleased to find a Python module that was mostly concerned with converting tabular data to data frames, but oh-by-the-way automatically converted qualitative data to dummy variables for regression analysis [16]

16. Yes, I recently did a regression on some data. ANOVA actually: it was appropriate.

17. For all my endless complaints about academic computer science, their single-minded focus on systematically comparing the performance of algorithms is a very valuable contribution to the ecosystem here. Just don’t expect them to write maintainable and documented code: that’s not what computer scientists or their graduate students do.

18. Algorithms from the 2020s, of course, and probably casting a wide net on these, as well as experimenting with how to best pre-process the training data—it’s not like parsing is useless —but general solutions, not highly specialized ones.

19. Firefighting, curiously, is another of his examples of a “kind” environment for learning.

20. If it doesn’t, you’ve forgotten to initialize something and/or are accessing/corrupting memory outside the intended range of your program. The latter is generally all but impossible in most contemporary languages, but certainly not in C! And C is alive and well! Of course, getting different results each time a program is run is itself a useful debugging diagnostic for such languages.

21. Another example of brain re-wiring following intensive focused study involves research on London taxi drivers: Google “brain london taxi drivers” for lots of popular articles, videos etc.

22. If Goleman and Davidson’s conclusions—essentially from a meta-analysis—can be generalized, periods of sustained deep cognitive work, which in meditation occurs in the context of retreats, may be particularly important for neuroplasticity. Such periods of sustained concentration are certainly common in other domains involving intense cognitive effort; the problem would remain reliably tracking these over a period of years. And we’re still stuck with the distinction that neuroplasticity is the objective of most intensive meditation practices, whereas it is an unsystematic side effect of professional cognitive work.