Testing make(1) with different -j options to build OpenBSD kernel on consumer Core 2 Duo G965 system

Below is a sample test of using OpenBSD's make on OpenBSD kernel source tree on G965-based Intel Core 2 Duo Allendale E4300 system with 2GB of Dual-Channel DDR2-800 memory and an HGST 7K160 160GB SATA3 HDD.

dmesg for the system is available on bugs@

The computer was accessed remotely via a Gigabit Ethernet LAN, over an SSH-channel. A second ssh-connection was made to directly run `top -s 1`. /etc/login.conf is set to defaults, the login from which we are making the build is part of staff.

Before taking the measures for the first time, I rebooted the system, and did a `make -j32` on the kernel source, to make sure that the state of the system is not going to influence any further builds.

The source-tree itself was updated several days ago, changes to make(1), specifically to job.c, were integrated today (i.e. my recent patch).

Let's start with the test now. The test was done in tcsh shell with `set time`, so that it shows the time that the command took to execute. Only the last few lines from the final make command are shown, together with the statistics that they produce.

% exec tcsh
% set time
% pushd /usr/src/sys/arch/i386/conf

	#default
% config GENERIC.MP
% pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
252.520u 36.810s 4:45.34 101.3% 0+0k 0+5204io 67pf+0w
 4:28PM  up 21 mins, 2 users, load averages: 2.98, 5.48, 5.03

	#-j2
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j2; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
256.070u 37.760s 5:19.50 91.9%  0+0k 0+10565io 67pf+0w
 4:37PM  up 30 mins, 2 users, load averages: 2.40, 2.84, 3.65

	#-j4
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j4; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
258.110u 40.530s 3:16.47 152.0% 0+0k 0+10303io 67pf+0w
 4:42PM  up 36 mins, 2 users, load averages: 5.81, 4.08, 3.85

	#-j8
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j8; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
260.730u 39.860s 2:40.05 187.8% 0+0k 0+10224io 67pf+0w
 4:47PM  up 40 mins, 2 users, load averages: 12.57, 7.63, 5.31

	#-j16
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j16; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
261.100u 41.010s 2:36.43 193.1% 0+0k 0+10187io 67pf+0w
 4:51PM  up 44 mins, 2 users, load averages: 22.39, 13.77, 8.31

	#-j24
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j24; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
261.940u 40.760s 2:36.86 192.9% 0+0k 0+10211io 67pf+0w
 4:54PM  up 48 mins, 2 users, load averages: 29.94, 20.26, 12.12

	#-j32
% popd && config GENERIC.MP && pushd ../compile/GENERIC.MP
% make clean && make depend
...
% make -j32; uptime
...
text    data    bss     dec     hex
5475544 140952  881968  6498464 6328a0
261.990u 40.650s 2:37.07 192.6% 0+0k 0+10187io 67pf+0w
 4:58PM  up 52 mins, 2 users, load averages: 37.45, 26.39, 16.30

Conclusion: on a Core 2 Duo Allendale E4300 system with the latest HitachiGST 7K160 HDD (Serial ATA 3.0Gb/s, 7200RPM), one may consider using -j16 option to optimise the wall-clock time of the kernel build. Using -j24 also works, but with -j32 you must make sure that you are not going to run out of the default limits that are specified for people in the staff login group (we didn't encounter this limit because the user ran almost no other processes during this testing time; however, -j40 has already started causing make to halt with a "make: Cannot fork" message).

One thing to note is that option -j2 is actually slower than the default compatibility mode (-B).