Difficult Problems and Hard Work

Hi everyone,

It seems people really like the fragments format. I'm a little surprised by this, but happy to lean into it for now. I expect in the long run I'll mix up the two formats and go back to writing the occasional essay, but this works pretty well as a default.

There is of course some ambiguity between fragments and essays anyway. Is today's an essay or a collection of fragments? Perhaps somewhere in between. It started out as a collection of fragments, but enough of a theme emerged through writing them that it at least gestures in the direction of an essay.

Hard Work and Difficult Problems

An idiosyncratic distinction I find useful (though don't reliably stick to) is that there is hard work and difficult problems, and these are not all that closely related.

The distinction is roughly that something is hard work if you have to put a lot of time and effort into it and a difficult problem if you have to put a lot of skill or thinking into it. You can generally always succeed at something that is “merely” hard work if you can put in the time and effort, while your ability to solve a difficult problem is at least somewhat unpredictable.

I've suggested in the past (cf. Care Work and Fixing Things, but this note is a mess I don't really recommend it) that there is a useful distinction between care work and repair work as two types of maintenance, and I think this hard work vs difficult problem is one of the important distinguishing features. e.g. Cleaning your flat (care work) is hard work, fixing a broken boiler (repair work) is (or can be) a difficult problem. I think this largely generalises: Care work is usually hard, repair work is often difficult.

I think I'm particularly aware of this distinction because programming is a mix of hard work and difficult problems and it's not always obvious which is which from the outside. There are a lot of things in programming which are “a simple matter of programming” (a phrase that basically means “It's obvious how to do this but it will take time to implement”) and there are some things in programming which are open research problems, and the two tend to blend seamlessly into each other.

Another example that illustrates the point is that writing these fragments is harder work than writing a longer essay, but writing a longer essay can be a difficult problem. I end up writing more, about a range of different topics, but also I have to keep less in my head and as soon as a fragment hits a point where I don't know how to continue I can just wrap it up because the fragment has done its job.

In contrast, weaving things together into a longer essay requires me to maintain enough focus and capacity for intelligent though to weave the whole thing together into a coherent whole. The last 20% of writing an essay can be genuinely very difficult, but not necessarily hard - in optimal conditions I can write a newsletter issue in an hour or two.

Differing Capabilities

As I said, on a good day I can write an entire newsletter in an hour or two. I don't have many days that good right now, and this is part of why the distinction is important: Our ability to do hard work and our ability to solve difficult problems often degrade differently. I find difficult problems often become impossible while depressed, in a way that hard work might not.

This is part of why the distinction is useful to bear in mind: It's tempting to think of our capability to do work as somehow homogenous, but it's not. Often when we're tired, or depressed, or ill, our capacity to do difficult work evaporates, because our thinking is impaired, but there is often less difficult but harder work we could be doing instead.

I think the difference is more or less that difficult problems require a lot of cognitive capacity. You can do hard work while distracted - you can clean your kitchen or process a whole mountain of fruit while watching TV, you can dig a ditch while spaced out thinking of something else.

I think the same is true with intellectual hard work, but it works out slightly differently. It has more of a stop/start characteristic. If I try to write, even relatively easy writing, while watching TV then I don't actually have the capacity to do both, I just swap between them. Some things I can write this way just fine (e.g. I often will hold a conversation over WhatsApp while watching something), some things I cannot (I tried writing this section while watching a YouTube video and it absolutely did not work, despite this not being a particularly difficult section to write), but even then if I'm tired or otherwise compromised I will often zone out, check Twitter, etc.

This doesn't necessarily stop me doing the writing, it just takes longer, but I think part of that is that with this format, like with a conversation, all the context I need is typically on the page and I can just write what logically comes next.

Decision Making is Difficult

One way a problem can be difficult is if it involves a lot of decision making, and the outcome depends crucially on the decisions made.

That last part is important. Just having to make decisions isn't difficult - if I go for a walk I can just meander around, making decisions about which turn to take arbitrarily. This is not a difficult problem, because the decisions made don't matter to the desired outcome, which is just to have a nice walk.

I think decision making is particularly difficult when it doesn't exactly look like decision making. If you have a simple decision to make between two options then this can be emotionally difficult, but it's not typically cognitively taxing (cf. How to make decisions). The problem is more common when you have to choose between myriad possibilities - when writing a long essay there are thousands of directions you could go in, endless ways to reorganise or reframe the text, and you have to decide between all of these.

When things are going well it often doesn't feel like you're deciding at all - you just follow the thread of intuition, pick the option that sparks joy, and do what comes naturally (cf. Intuition as Search Prioritisation). When that stops working, it's easy to become overwhelmed by the possibilities.

This is, I think, why being depressed can so easily compromise our ability to act. Depression mutes our ability to feel positive emotions, so we loose that sense of which way to go.

The Conversion of the Difficult to the Hard

A lifetime ago back in 2019 I wrote How to do Hard Things, one of my most popular posts, which is essentially about how to acquire skills that currently seem beyond your reach by repeatedly improving your current skills in small ways. Its notion of “hard things” doesn't really track the notion of “hard work” I've been using here. In the current formulation, what it's describing is something closer to “How to solve the difficult problem of skill acquisition with hard work”.

Another thing roughly in this space is How to quickly become effective when joining a new company, which is specifically a strategy for the difficult problem of getting up to speed with a new area (in this case a new company, but it's general).

Both of these can be thought of strategies for reducing the amount of critical decision making you have to do at once. “How to do hard things” requires a lot of decisions, but the decisions ultimately don't matter that much because anything you pick that satisfies the conditions works - you are making decisions, but you can pick anything you like in any order. It makes it safe to make arbitrary decisions.

In contrast, with “How to quickly become effective” you have vitally important decisions which affect the success of your project, and the strategy makes those decisions for you. It provides a simple rule: Decide in the direction of what is most immediately useful, disregarding any long-term consequences or utility of that approach.

A third strategy that works the same way is adequate teaching by an expert. Learning a subject can be extremely difficult, but given a good teacher it becomes merely hard. When learning a subject you will often have to make intuitive leaps, form connections, and organise your knowledge of the subject, all of which are difficult problems. Having access to an expert who has already done these things and knows how to guide you through them allows you to delegate those problems. You still have to do the hard work of learning, but your ability to succeed is much less in question.

(Unfortunately most teaching is not adequate in this sense - both because it is very hard to guide people through their individual failures to understand in a group setting, and because providing such guidance itself is a difficult problem which most teachers are not very good at).

I think this points in a fairly general direction: Effective strategies often consist of converting difficult problems into something that can be solved through hard work. They do this by reducing the cognitive cost of decision making. There are a variety of ways of doing this, of which we've seen three here: Focusing on decisions where pretty much anything you can do works, prioritising decisions that quickly lead to good outcomes, and delegating to external sources of decision making. There are almost certainly many more strategies that solve this problem in other ways.

This is potentially particularly useful to bear in mind right now while we're in the tail end of the pandemic and everything is difficult.