1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
|
.\" $OpenBSD: bulk.8,v 1.3 2018/07/10 16:01:48 jmc Exp $
.\"
.\" Copyright (c) 2016 Marc Espie <espie@openbsd.org>
.\"
.\"
.\" Permission to use, copy, modify, and distribute this software for any
.\" purpose with or without fee is hereby granted, provided that the above
.\" copyright notice and this permission notice appear in all copies.
.\"
.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
.Dd $Mdocdate: July 10 2018 $
.Dt BULK 8
.Os
.Sh NAME
.Nm bulk
.Nd building OpenBSD packages in bulk
.Sh DESCRIPTION
There are quite a few steps necessary to build packages on a cluster.
They are:
.Pp
.Bl -enum -compact -offset indent
.It
Choose master machine setup and create partitions.
.It
Setup chrooted builds on the master.
.It
Add slaves and do a full bulk.
.It
Clean up and do subsequent bulks.
.It
Perform additional maintenance.
.El
.Ss 1. Choose master machine setup and create partitions
Setup a master machine with enough room for a chroot, say
.Pa /build .
Assuming you are using a cluster of machines,
this chroot should contain NFS exportable partitions for distfiles,
plists, and packages (one single partition can be used for simplicity).
A full setup requires on the order of 50GB for distfiles and 50GB for packages.
.Pp
It is possible to build packages without a chroot, but the space
requirement difference is negligible (a full
.Ox
install is less than 1GB), and having everything chrooted means you may
install useful tools to help with the process outside of the chroot
.Po
for instance
.Xr rsync 1
.Pc .
.Pp
Reserve one "scratch" partition under the chroot for WRKOBJDIR
(for instance, mfs, async, or SSD).
This partition should be roughly 10GB if you want to be able to
build all ports.
This can often double as
.Pa /tmp
under the chroot.
.Pp
Alternately, you can setup your whole chroot as a scratch partition,
and reserve one more permanent space under it for distfiles,
packages, and plists.
.Pp
Choose a strategy for the ports tree itself.
There must be a copy under
.Pa /build .
You can either copy it from outside the chroot, have it in an NFS
partition, or manually make sure all machines on the cluster have the
same ports tree (cvs checkout, rsync ...).
.Pp
Note that logs are only produced on the master, and thus do not
need to be nfs exportable, nor even inside the chroot.
.Pp
.Ox
now comes with default users for package builds, namely _pbuild and _pfetch.
.Pp
The default
.Xr login.conf 5
is appropriate for most setups, but _pbuild's datasize-cur will need
to be bumped for a few ports, like pypy.
Likewise, maxproc-cur is too small for machines with more than 4-6 cpus.
.Pp
Note that _pbuild does not need network access, and is now blocked by default
in
.Xr pf 4 .
.Pp
Recent
.Ox
systems do not need any kind of
.Xr doas 1
setup for bulk ports builds, as
.Xr dpb 1
is run as root and drops permissions appropriately.
.Pp
However, you may still want to setup
.Xr doas 1
for root, if you want to manually fix ports, as
.Ar PORTS_PRIVSEP
relies on it.
.Ss 2. Setup chrooted builds on the master
Populate the initial chroot with
.Xr proot 1 .
Point DISTDIR, PACKAGE_REPOSITORY, PLIST_REPOSITORY, WRKOBJDIR
to appropriate locations.
.Pp
Pay attention to nodev and wxallowed warnings.
A chroot setup that can't have devices won't work at all.
A setup without wxallowed in /usr/local and WRKOBJDIR won't
build a lot of things.
.Pp
Check that this setup can build ports by running
.Li dpb -B /build
as root.
Fix any issues related to file ownership at this point.
See
.Xr dpb 1
for details.
.Pp
If your WRKOBJDIR is a temporary partition, make sure it
belongs to _pbuild:_pbuild after a reboot.
.Ss 3. Add slaves and do a full bulk
Create identical slave machines with the same release material.
Have them import the NFS partitions from the master, they
don't need root access to the partitions.
Set up
.Xr ssh 1
so that the master can connect to the slaves, using ssh protocol 2,
as root, preferably without a password or passphrase (however,
.Xr dpb 1
uses a master connection, so a password would be required just once per host).
.Pp
Note that code on slave machines will only run as _pbuild
(during builds) or root (during dependency installation).
Slave machines only require highly restricted network access.
They just need to act as nfs clients to the master and to be reachable
through ssh from the master.
.Pp
Use a similar
.Xr proot 1 config
to populate each slave.
.Pp
You should now be able to build ports on the slaves.
A simple config will just have
.Bd -literal -offset indent
DEFAULT chroot=/build
localhost
host1
\&...
.Ed
.Pp
Check that the full config can still build ports.
.Pp
You're now ready for a full bulk.
Beware that even fast configs (3 amd64 with 8 cores each) may take over 24 hours
to finish.
It's generally appropriate to run
.Xr dpb 1
under
.Xr tmux 1 .
.Ss 4. Clean up and do subsequent bulks
Before running the next bulk, you should set up rotating logs and move the
PACKAGE_REPOSITORY away.
Save the PLIST_REPOSITORY and DISTDIR though.
PLIST_REPOSITORY will catch problems in packing-lists.
.Pa ${PLIST_REPOSITORY}/${ARCH}/history
is also used to store
.Xr sha256 1
history, necessary to reorder files inside packages to speed updates up.
.Pp
The DISTDIR contains history information as well as DISTDIR/build-stats
to speed further runs up.
.Pp
How you wipe things out depends on your setup.
If you run
.Xr proot 1
again each time, most things will get cleaned up automatically
.Po
.Pa /build/usr/local , /build/var/db/pkg ...
.Pc .
Note that known directories such as WRKOBJDIR do not get cleaned up
automatically, so you may want to set up a STARTUP_SCRIPT in
.Xr dpb 1 .
.Ss 5. Perform additional maintenance
.Xr clean-old-distfiles 1
should be run occasionally since the DISTDIR will continue growing.
.Pp
.Xr pkg_check-problems 1
should be run occasionally to find out conflicts and dependency issues.
.Sh SEE ALSO
.Xr clean-old-distfiles 1 ,
.Xr dpb 1 ,
.Xr pkg_check-problems 1 ,
.Xr proot 1 ,
.Xr register-plist 1 ,
.Xr tmux 1 ,
.Xr bsd.port.mk 5 ,
.Xr release 8
|
