This begins a short series of essays on what most programmers get wrong about multi-threading. Over and over again, I’ve seen programmers make the same mistakes in multi-threaded applications on multiple operating systems and in multiple languages. So I decided to give you the benefits of what little insight I have into the problem.
The first problem most programmers have with multi-threading is extremely fundamental: multi-threading an application does not speed it up. In fact, the decision to multi-thread a program is rarely about raw speed. I expect that some of the people reading this are going to decide that I have no idea what I’m talking about at this point. However, I can back up this statement.
Given a single CPU machine and process that is CPU-bound, adding threads must slow down the program. In addition to the work that we were already doing, we now have to spend extra time doing context switches. Therefore, for the CPU-bound program, threading is guaranteed to slow down the program. (Of course, this only applies to preemptive multi-threading.)
If raw speed is not the issue, why would we multi-thread a program? On a single CPU system, there are only two real reasons to use threads.
- responsiveness
- resources that block
Responsiveness
The main reason that preemptive multitasking has become popular in recent years is responsiveness. One process or thread should not be able to lock up the whole machine when the user wants access. The average user does not care if some process running in the background takes a few seconds longer to run as long as the computer responds immediately when a key is pressed or the mouse is moved. Although the computer is actually doing things slower, it feels faster because the computer is more responsive.
Many years ago, I was working in medical research. There was a program written by one of the programmers that normally ran over the weekend. Unfortunately, the guy that wrote it did not write any output to the screen or disk until the program was finished running. So, on Monday morning, we were often confronted with a blank screen and we didn’t know if the program was still running, locked up, or what. We had no feedback at all. (This was back in the days of DOS, so we couldn’t do anything else while the program was running either.) Eventually we changed the program to add a little progress counter to the screen, so we could at least tell if the program was still running. We used to joke about how much faster this made the program. Even though we knew the extra output was slowing down the real work, the feedback made it seem faster.
This concept is what drives most multi-threading development. We want instant feedback and responsive computers. Things can actually run slower, as long as we get these two things.
Blocking
The other reason for multi-threading is that not all processes are CPU-bound. At some time, most programs must do something that blocks progress. We might need to read data from disk to do further calculations, or retrieve data from a database, or even get user input. While this process or thread is blocked, it would be nice to allow the CPU to do other work. Threading makes this possible. (Actually, we used to do this with a cruder form of multitasking years ago. But, threads do make it easier.)
With multi-threaded systems, the CPU can ignore threads that are blocked and continue with threads that are ready for work. This makes better use of the CPU and allows us to appear to be doing more than one thing at a time. In fact, by carefully organizing our code to keep the processor busy, we can appear to be running faster by keeping the CPU busy. In this case, we are not really speeding up the code. We are just no longer wasting CPU cycles.
Multiple CPUs
Of course, the entire discussion above assumes a single CPU. If we have multiple CPUs in a system, the situation is only slightly different. We can get more work done at a time because there is more than one CPU, but the problem remains the same. We can not get more work done than we have CPUs.
In a later essay, I’ll delve into a piece of advice I received a long time ago that you should never have more threads than you have CPUs (plus a few extra to to take advantage of blocking calls). This turns out to be like most multi-threading advice, part right, but mostly wrong.