February 14, 2003
Parallelization

Suppose you have a compilation process that consists of three sequential phases. The first phase is totally under your control and is linearly parallelizable. The second phase is partly under your control and is partly parallelizable. The third phase is not under your control and is not parallelizable.

Question: should you upgrade your build system (1GB RAM, Pentium IV 1.6GHz) to a parallel machine?

Sadly, in our case, the answer is no. Although we occasionally run a large (5 hour) build, the best-case parallelization would reduce that to only 2:40 on two processors and 1:40 on four. And we're unlikely to get best-case parallelization. A more reasonable estimate is 3:00 on 2 and 2:00 on 4.

The initial, fully parallelizable task takes 80 minutes. The final non-parallelizable task takes 1 hour. The rest of the task -- the main compilation -- is 2 1/2 hours plus. It would take significant work to make this phase safe for parallel execution.

Posted by Sam at 03:34 AM