move ports infra manpages into base, as discussed with

schwarze@, deraadt@, landry@...

25 files changed, 4074 insertions, 3 deletions

diff --git a/share/man/man1/Makefile b/share/man/man1/Makefile
index bab3094ab1f..feaa34eb45a 100644
--- a/share/man/man1/Makefile
+++ b/share/man/man1/Makefile
@@ -1,6 +1,15 @@
-#	$OpenBSD: Makefile,v 1.7 2017/02/12 09:57:47 jsg Exp $
+#	$OpenBSD: Makefile,v 1.8 2018/06/26 05:38:49 espie Exp $
 #	$NetBSD: Makefile,v 1.4 1994/12/22 10:48:04 cgd Exp $
 
 MAN=	clang-local.1 help.1 intro.1 gcc-local.1
 
+# ports tools doc
+MAN+=	check-lib-depends.1 check-manpages.1 check-problems.1 \
+	clean-old-distfiles.1 dolock.1 dpb-replay.1 dpb.1 \
+	getpkgpath.1 out-of-date.1 outdated-perl-ports.1 \
+	pkg_subst.1 portbump.1 portcheck.1 portgen.1 \
+	portimport.1 portslogger.1 proot.1 register-plist.1 \
+	resolve-lib.1 retrieve-index.1 update-patches.1 \
+	update-plist.1
+
 .include <bsd.prog.mk>
diff --git a/share/man/man1/check-lib-depends.1 b/share/man/man1/check-lib-depends.1
new file mode 100644
index 00000000000..8362699c495
--- /dev/null
+++ b/share/man/man1/check-lib-depends.1
@@ -0,0 +1,145 @@
+.\"	$OpenBSD: check-lib-depends.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt CHECK-LIB-DEPENDS 1
+.Os
+.Sh NAME
+.Nm check-lib-depends
+.Nd scan port for shared libraries
+.Sh SYNOPSIS
+.Nm check-lib-depends
+.Op Fl fimoqx
+.Op Fl B Ar destdir
+.Op Fl d Ar pkgrepo
+.Op Fl D Ar directory
+.Op Fl F Ar missing
+.Op Fl O Ar dest
+.Op Fl S Ar VAR Ns = Ns Ar value
+.Op Fl s Ar source
+.Op Ar package ...
+.Sh DESCRIPTION
+.Nm
+scans a port or package for shared libraries, and verify that all of them
+are properly recorded.
+.Pp
+.Nm
+can either scan files from a built port after the
+.Ar fake stage
+from the
+.Ev WRKINST
+directory, or look directly inside a
+.Ar package .
+.Pp
+Likewise, it can also verify dependencies off installed packages,
+or figure them out directly from the port directory.
+.Pp
+.Nm
+actually consists of two distinct stages:
+.Bl -bullet
+.It
+scan all binaries inside a package to extract libraries and rpaths,
+using
+.Xr objdump 1
+or
+.Xr ldd 1 .
+.It
+compare those libraries against the port's registered dependencies
+and report problems.
+.El
+.Pp
+The options are as follows:
+.Bl -tag -width keyword
+.It Fl B Ar destdir
+Scan files under
+.Ar destdir .
+.It Fl d Ar pkgrepo
+Look for all needed packages under directory
+.Ar pkgrepo .
+.It Fl D Ar directory
+Create debug log files under
+.Ar directory
+that show in more details the run of
+.Xr objdump 1 .
+.It Fl f
+Give full reports of every file that requires a missing library.
+.It Fl F Ar missing
+Works in tandem with
+.Fl S .
+Allow backsubstituting even if the
+.Ar missing
+library
+is actually not part of the actual
+.Ev WANTLIB
+of the package.
+Mostly used to waive the presence of
+.Ar pthread ,
+a mandatory dependency of C++ libraries on some architectures.
+.It Fl i
+Read packing-list from standard input.
+.It Fl m
+Always show progress-meter, even if not run from terminal.
+.It Fl o
+Scan binaries using
+.Xr ldd 1 ,
+which also works for old a.out binaries, instead of
+.Xr objdump 1 ,
+which only works for
+.Xr elf 5
+binaries.
+.It Fl O Ar dest
+Save result of first stage in file
+.Ar dest .
+.It Fl q
+Be quiet, do not emit
+.Sq Extra:
+wantlibs unless there's also an actual problem.
+.It Fl S Ar var Ns = Ns Ar value
+recognize list of libraries listed in a variable, and replace said list
+with variable value.
+For instance,
+.Fl S Ar COMPILER_LIBCXX Ns = Ns Ar stdc++
+will replace
+.Sq stdc++
+with
+.Sq ${COMPILER_LIBCXX}
+in
+.Ev WANTLIB
+lists for compatibility with both clang and gcc.
+Order matters.
+The first
+.Fl S
+option will be handled first.
+.It Fl s Ar src
+Don't scan, directly read result of first stage from file
+.Ar src .
+.It Fl x
+Don't show progress-meter, even if run from terminal.
+.El
+.Pp
+If no
+.Ar package
+is given, and the
+.Fl i
+option is used,
+.Nm
+will retrieve the packing-list from stdin, and scan the package according
+to other options.
+.Sh SEE ALSO
+.Xr ldd 1 ,
+.Xr objdump 1 ,
+.Xr bsd.port.mk 5 ,
+.Xr elf 5
diff --git a/share/man/man1/check-manpages.1 b/share/man/man1/check-manpages.1
new file mode 100644
index 00000000000..f31d545c4ef
--- /dev/null
+++ b/share/man/man1/check-manpages.1
@@ -0,0 +1,60 @@
+.\"	$OpenBSD: check-manpages.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt CHECK-MANPAGES 1
+.Os
+.Sh NAME
+.Nm check-manpages
+.Nd verify manpage subjects in packages
+.Sh SYNOPSIS
+.Nm check-manpages
+.Op Fl p
+.Op Ar pkgname ...
+.Sh DESCRIPTION
+.Nm
+is used to check manpages in a collection of packages.
+It will extract all manpages from the
+.Ar pkgname
+arguments into a directory named
+.Pa manpages.XXXXXXXXXXXX
+.Po
+see
+.Xr mktemp 1
+.Pc
+and run them through
+.Xr makewhatis 8 .
+.Pp
+Note that this directory is not cleared on exit, so that the problematic
+manpages can be examined more precisely.
+.Pp
+If no
+.Ar pkgname
+argument is passed, it will process all
+.Pa *.tgz
+files in the current directory.
+.Pp
+Options are as follows:
+.Bl -tag -width packing-list
+.It Fl p
+Run
+.Xr makewhatis 8
+in picky mode.
+.El
+.Pp
+Errors reported in picky mode should usually be fixed, as it means the
+subject line stored in the whatis database will be useless to find out
+the corresponding manpage.
diff --git a/share/man/man1/check-problems.1 b/share/man/man1/check-problems.1
new file mode 100644
index 00000000000..d9d8bfae6f5
--- /dev/null
+++ b/share/man/man1/check-problems.1
@@ -0,0 +1,191 @@
+.\"	$OpenBSD: check-problems.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010-2015 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt CHECK-PROBLEMS 1
+.Os
+.Sh NAME
+.Nm check-problems
+.Nd verify problems in package collections
+.Sh SYNOPSIS
+.Nm check-problems
+.Op Fl CDesv
+.Op Fl d Ar plist_dir
+.Op Fl o Ar output
+.Op Fl p Ar ports_dir
+.Op Ar pkgname ...
+.Sh DESCRIPTION
+.Nm
+is used to check a collection of packages for problems that cannot be
+noticed in individual packages, such as conflict and directory registration
+issues.
+.Pp
+A conflict is a file that belongs to two distinct packages.
+.Pp
+By default, packages with the same stem
+.Po
+see
+.Xr packages-specs 7
+.Pc
+are registered to conflict.
+This can be changed to add or remove extra conflict registration with
+.Cm @option no-default-conflict
+or
+.Cm @conflict
+.Po
+see
+.Xr pkg_create 1
+.Pc .
+.Pp
+Conflicts between packages mean that a user can't have both packages installed
+at the same time.
+In some cases (same software, different version), this is expected, but
+the packages should be marked as conflicting, so that
+.Xr pkg_add 1
+knows how to deal with it.
+In other cases, this is an oversight from the ports maintainer, and action
+should be taken to make sure both packages can be installed simultaneously.
+Conflicts also interact with dependencies, and sometimes a missing dependency
+will show up as an unregistered conflict.
+.Pp
+Directories can belong to distinct packages, as long as they are registered
+somewhere.
+Every directory that is not part of the standard
+.Xr mtree 8
+set should appear in a packing-list somewhere.
+.Pp
+Contrary to files, directories can be shared between several packages.
+.Pp
+.Nm
+knows enough about packages to trace directories back to a common dependency,
+or to not flag two packages as conflicting when they depend on already
+conflicting packages.
+.Pp
+.Nm
+requires a set of packing-lists to operate, which it can obtain from
+several sources.
+The preferred source is the packing-list databases
+.Po
+see
+.Ar PACKAGE_REPOSITORY
+in
+.Xr bsd.port.mk 5
+and
+.Xr register-plist 1
+.Pc ,
+as it persists over time and can be used to fix update information as well
+as current version information.
+.Pp
+It should be supplemented with the ports tree itself in order to figure
+out current versions.
+.Pp
+Options are as follows:
+.Bl -tag -width packing-list
+.It Fl C
+Do not compute conflict information.
+.It Fl D
+Do not compute common dir information.
+.It Fl e
+Check for extra logical conflicts, by adding supplementary
+formatted/unformatted manpages, as same name manpages under the
+same base directory will shadow one another.
+.It Fl o Ar output
+Save results to output file as well as writing it on the screen.
+.It Fl s
+Be silent.
+.It Fl v
+Be verbose.
+.It Fl d Ar plist-dir
+Scan directory
+.Ar plist-dir
+for packing-lists.
+.It Fl p Ar portsdir
+Traverse the ports directory for packing-lists, using
+.Ar print-plist-all .
+More than one directory could be specified by delimiting them by colon,
+e.g.:
+.Pa /usr/ports/x11/kde:/usr/ports/x11/kde4 .
+.Pp
+If
+.Fl d Ar plist-dir
+is also used, traverse only the directories corresponding to the packages
+under
+.Ar plist-dir
+to figure out current pkgnames, and annotate those with a
+.Sq \&!
+in the report.
+.El
+.Sh SAMPLE OUTPUT
+.Bd -literal
+Dependency not found zarafa-webaccess-7.1.11p2
+Dependency not found kwebkitpart-1.2p3
+[...]
+Common dirs:
+!claws-mail-gdata-3.11.1p1-ldap(mail/claws-mail,-gdata,ldap): /usr/local/lib/claws-mail
+!claws-mail-pdfviewer-3.11.1p1-ldap(mail/claws-mail,-pdfviewer,ldap): /usr/local/lib/claws-mail
+[...]
+Conflicts:
+!arm-none-eabi-gdb-7.9.1(devel/arm-none-eabi/gdb),!gdb-7.9.1(devel/gdb)
+	/usr/local/include/gdb/jit-reader.h
+	/usr/local/share/gdb/system-gdbinit/elinos.py
+	/usr/local/share/gdb/system-gdbinit/wrs-linux.py
+!libnet-1.0.2ap0v0(net/libnet/1.0),libnet-1.0.2ap2(net/libnet/1.0),libnet-1.0.2av0(net/libnet/1.0)
+	/usr/local/bin/libnet-config-1.0
+	/usr/local/include/libnet-1.0/libnet.h
+	/usr/local/include/libnet-1.0/libnet/libnet-asn1.h
+	/usr/local/include/libnet-1.0/libnet/libnet-functions.h
+	/usr/local/include/libnet-1.0/libnet/libnet-headers.h
+	/usr/local/include/libnet-1.0/libnet/libnet-macros.h
+	/usr/local/include/libnet-1.0/libnet/libnet-ospf.h
+	/usr/local/include/libnet-1.0/libnet/libnet-structures.h
+	/usr/local/lib/libnet-1.0/libnet.a
+	/usr/local/lib/libnet-1.0/libnet.so.0.0
+	/usr/local/lib/libnet-1.0/libpwrite.a
+	/usr/local/lib/libnet.so.0.0
+	/usr/local/man/cat3/libnet.0
+[...]
+zathura-pdf-mupdf-0.2.7(textproc/zathura/plugins/mupdf),zathura-pdf-poppler-0.2.5(textproc/zathura/plugins/poppler)
+	/usr/local/lib/zathura/pdf.so
+.Ed
+.Bl -bullet
+.It
+Dependency not found: means some package names could not be found.
+Most often happens because of panic issues, and fsck not recovering some
+dependency files.
+.It
+claws-mail directory: missing directory registration.
+.Nm
+could not find a common ancestor to claws-mail-gdata and claws-mail-pdfviewer,
+so it should be added to both.
+.It
+Conflict between arm-non-eabi-gdb and gdb:
+note that both packages are current.
+Two choices: either remove the offending files from one package (might break
+the other), or mark an explicit conflict between the packages (will prevent
+people from installing both at the same time).
+.It
+Conflict between libnet-...v0 and libnet:
+Note that only the v0 version is current.
+People forgot to augment the
+.Cm @option no-default-conflict
+with conflicts against the old version when bumping the
+.Ar EPOCH .
+.El
+.Sh SEE ALSO
+.Xr pkg_add 1 ,
+.Xr pkg_create 1 ,
+.Xr bsd.port.mk 5 ,
+.Xr packages-specs 7
diff --git a/share/man/man1/clean-old-distfiles.1 b/share/man/man1/clean-old-distfiles.1
new file mode 100644
index 00000000000..24b45256682
--- /dev/null
+++ b/share/man/man1/clean-old-distfiles.1
@@ -0,0 +1,95 @@
+.\"	$OpenBSD: clean-old-distfiles.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2012 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt CLEAN-OLD-DISTFILES 1
+.Os
+.Sh NAME
+.Nm clean-old-distfiles
+.Nd remove old distfiles
+.Sh SYNOPSIS
+.Nm clean-old-distfiles
+.Op Fl nv
+.Op Fl e Ar except
+.Op Fl h Ar history
+.Op Ar timestamp
+.Sh DESCRIPTION
+.Nm
+uses information provided by
+.Xr dpb 1
+to remove obsolete distfiles:
+each time
+.Xr dpb 1
+is run for a full scan of the ports tree, it appends to
+an history log that records obsolete distfiles.
+.Pp
+.Nm
+can be used to prune that log prior to a given
+.Ar timestamp
+and remove the corresponding distfiles.
+.Pp
+For normal files,
+.Nm
+verifies
+.Xr sha256 1
+checksums prior to removal.
+.Nm
+also removes links under
+.Pa by_cipher/sha256 ,
+but it trusts that the checksums for these are accurate.
+.Pp
+The
+.Ar timestamp
+optional argument is given in seconds since the epoch,
+see
+.Xr date 1 .
+.Pp
+Options are as follows:
+.Bl -tag -width nnnn
+.It Fl e Ar except
+Read exception list from file
+.Ar except .
+Each line is the name of a file that shouldn't be removed.
+.It Fl h Ar history
+Read log from history file
+.Ar history
+instead of the default
+.Pa /usr/ports/distfiles/history .
+Turns on
+.Fl nv ,
+as this is a testing option.
+.It Fl n
+Don't really remove files.
+.It Fl v
+Display what will be removed, as
+.Li rm file
+lines.
+.El
+.Sh FILES
+.Bl -tag -width /usr/ports
+.It Pa /usr/ports
+base of the ports tree, can be overridden with env variable
+.Ev PORTSDIR .
+.It Pa /usr/ports/distfiles
+location of the distfiles repository, can be overridden with env variable
+.Ev DISTDIR .
+.It Pa /usr/ports/distfiles/history
+log of obsolete distfiles.
+.El
+.Sh SEE ALSO
+.Xr dpb 1 ,
+.Xr sha256 1 ,
+.Xr mirroring-ports 7
diff --git a/share/man/man1/dolock.1 b/share/man/man1/dolock.1
new file mode 100644
index 00000000000..67e288e2fc1
--- /dev/null
+++ b/share/man/man1/dolock.1
@@ -0,0 +1,51 @@
+.\"	$OpenBSD: dolock.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt DOLOCK 1
+.Os
+.Sh NAME
+.Nm dolock
+.Nd lock port for concurrent builds
+.Sh SYNOPSIS
+.Nm dolock
+.Op Fl g Ar group
+.Op Fl m Ar mode
+.Ar lockname Op comment ...
+.Sh DESCRIPTION
+.Nm
+creates a lock named
+.Ar lockname
+using
+.Xr open 2 .
+If the lock is already held, it will display one single wait message on
+stderr, sleep and retry until satisfaction.
+.Pp
+If necessary, creates the lock directory, using
+.Fl m Ar mode
+as a mode argument or the default if that option is not specified,
+and
+.Fl g Ar group
+as the group owner if that option is specified.
+.Pp
+This implements a locking mechanism for ports building, see
+.Sq LOCK INFRASTRUCTURE
+in
+.Xr ports 7
+for details.
+See also
+.Xr dpb 1
+for a complementary lock infrastructure when using dpb.
diff --git a/share/man/man1/dpb-replay.1 b/share/man/man1/dpb-replay.1
new file mode 100644
index 00000000000..26c765149ca
--- /dev/null
+++ b/share/man/man1/dpb-replay.1
@@ -0,0 +1,48 @@
+.\"	$OpenBSD: dpb-replay.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2013 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt DPB-REPLAY 1
+.Os
+.Sh NAME
+.Nm dpb-replay
+.Nd replays dpb terminal output
+.Sh SYNOPSIS
+.Nm
+.Op Fl c
+.Op Fl s Ar speedup
+.Ar file
+.Sh DESCRIPTION
+.Nm
+is used to replay
+.Xr dpb 1
+terminal logs
+.Po
+.Pa %L/term-report.log
+.Pc
+later, usually at a much higher speed than the first run,
+for demonstration and optimization purposes.
+The options are as follows
+.Bl -tag -width keywordaaaa
+.It Fl c
+Color display.
+.It Fl s Ar speedup
+Sets the time acceleration to
+.Ar speedup
+(defaults to 10).
+.El
+.Sh SEE ALSO
+.Xr dpb 1
diff --git a/share/man/man1/dpb.1 b/share/man/man1/dpb.1
new file mode 100644
index 00000000000..a27b0873368
--- /dev/null
+++ b/share/man/man1/dpb.1
@@ -0,0 +1,1673 @@
+.\"	$OpenBSD: dpb.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010-2013 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt DPB 1
+.Os
+.Sh NAME
+.Nm dpb
+.Nd distributed ports builder
+.Sh SYNOPSIS
+.Nm dpb
+.Bk -words
+.Op Fl acemqrRsuUvx
+.Op Fl A Ar arch
+.Op Fl b Ar logfile
+.Op Fl B Ar chroot
+.Op Fl C Ar pathlist
+.Op Fl D Ar PARAM Ns = Ns Ar value
+.Op Fl f Ar m
+.Op Fl F Ar m
+.Op Fl h Ar hosts
+.Op Fl I Ar pathlist
+.Op Fl J Ar p
+.Op Fl j Ar n
+.Op Fl L Ar logdir
+.Op Fl l Ar lockdir
+.Op Fl M Ar threshold
+.Op Fl P Ar pathlist
+.Op Fl p Ar parallel
+.Op Fl S Ar logfile
+.Op Fl X Ar pathlist
+.Op Ar pathlist ...
+.Ek
+.Sh DESCRIPTION
+.Nm
+is used to build ports on a cluster of machines, or on a single machine
+with several cores.
+.Nm
+walks the ports tree to figure out dependencies, and starts building ports
+as soon as it can.
+.Pp
+.Nm
+will run with sensible defaults if used without options.
+Note, however, that it will produce logs, lock files, packages, and package
+installations.
+.Pp
+If run as non-root,
+.Nm
+will warn.
+The preferred way is to run it as root (and preferably under a chroot).
+.Nm
+will then change its identity to different users as needed.
+See
+.Sq THE SECURITY MODEL OF DPB
+for details.
+.Pp
+.Nm
+can be restricted to a subset of the tree by giving it
+.Ar pathlist ...
+to build as parameters.
+.Pp
+A
+.Ar pathlist
+is either a
+.Xr pkgpath 7
+to build, or a filename that contains pkgpaths (one per line).
+.Ar pathlist
+parameters can also take the form
+.Li filename*scale
+in order to multiply the weights of all
+.Xr pkgpath 7
+in a file by a given
+.Ar scale ,
+or
+.Li pkgpath=value ,
+in order to set the weight of a given
+.Xr pkgpath 7
+to a specific value.
+.Pp
+.Nm
+supports
+.Sq hot-fixes :
+if a particular port errors out, it is possible to fix the problem, remove
+the corresponding lockfile, and
+.Nm
+will pick it up without needing to be stopped and restarted.
+.Pp
+In order to build on a cluster, the ports tree itself should be identical
+on each machine (shared through NFS or copied at start).
+.Pp
+Some directories must be shared:
+.Ev PACKAGE_REPOSITORY ,
+.Ev DISTDIR ,
+and
+.Ev PLIST_REPOSITORY .
+The
+.Ev WRKOBJDIR
+and
+.Ev LOCKDIR
+should be local to each machine, and on a high-speed partition.
+.Pp
+Also note that
+.Nm Ns 's
+logs and locks are managed by the main
+.Nm
+process, which runs locally, and hence those directories do not need to
+be shared on the cluster.
+.Pp
+Some log files ("rolling logs") are kept from one run to the run and
+stored under
+.Pa ${DISTDIR}/build-stats .
+.Pp
+Option
+.Fl h Ar file
+is used to specify hosts to use, where
+.Ar file
+may contain lots of information,
+but can be as simple as a list of hosts to use, one host per line
+(however, it is recommended to also include a
+.Ar STARTUP
+script).
+.Pp
+Most filenames will go through some control sequence expansions.
+For instance, the default logdir location can be specified as
+.Pa %p/logs/%a .
+The following sequences are recognized:
+.Bl -tag -offset aaaa -width %aa
+.It Cm %a
+architecture being used.
+.It Cm %d
+date at start of
+.Nm ,
+GMtime, formatted as yyyy-mm-dd@hh:mm:ss.
+.It Cm %f
+fetch distfiles location (DISTDIR).
+.It Cm %h
+hostname running
+.Nm .
+.It Cm %L
+logdir location.
+.It Cm %p
+portsdir location.
+.It Cm %t
+timestamp (number of seconds since January 1 1970) at start of
+.Nm .
+.El
+.Pp
+Options are as follows:
+.Bl -tag -width pkgpathlong
+.It Fl A Ar arch
+Build packages for given architecture, selecting relevant hosts from the
+cluster.
+By default, the current host's architecture will be used.
+.It Fl a
+Walk the whole tree and builds all packages (default if no
+.Ar pathlist
+is given).
+.It Fl b Ar logfile
+Explicitly prime the heuristics module with a previous build log,
+so that packages that take a long time to build will happen earlier.
+The rolling log under
+.Pa %f/build-stats/%a
+is automatically used.
+.It Fl B Ar chroot
+chroot to
+.Ar chroot
+before building.
+See
+.Xr proot 1
+for preparing such an environment.
+.It Fl c
+Clean port working directory and log before each build.
+.It Fl C Ar pathlist
+Don't clean port working directories after build.
+Only use simple
+.Xr pkgpath 7
+in the list,
+as this does not take subpackages and flavors into account.
+.It Fl D Ar PARAM Ns = Ns Ar value
+Set defined parameter to value.
+Known parameters are as follows:
+.Bl -tag -width DISP
+.It Ar ALWAYS_CLEAN
+Set to 1 if
+.Nm
+should clean work directories even if the port errored out.
+.It Ar BUILD_USER
+Default value for
+.Ar build_user
+if you want to specify it on the command line, and want to ensure even
+the small "discover PORTSDIR" activity at the beginning of
+.Nm
+is not run as root.
+.It Ar CDROM_ONLY
+Don't fetch distfiles that are not allowed for cdrom.
+.It Ar COLOR
+Set to 1 to have the normal display in color.
+.It Ar CONTROL
+Let
+.Nm
+create a unix socket of the given name for external control.
+.It Ar CONNECTION_TIMEOUT
+Connection timeout for ssh.
+Defaults to 60 seconds.
+.It Ar DISPLAY_TIMEOUT
+Display timeout (in seconds) while waiting for jobs to finish, so that the
+display is updated even if jobs didn't finish.
+Defaults to 10 seconds.
+.It Ar DONT_BUILD_ONCE
+By default,
+.Nm
+will use the
+.Ev BUILD_ONCE
+optimization
+.Po
+see
+.Xr bsd.port.mk 5
+.Pc
+if run with
+.Fl a :
+pseudo-flavors that disable subpackages and are not necessary for bootstrap
+will be disabled, so that the same port is built once, as far as possible.
+This flag disables that optimization, which might be desireable if you want
+to build a small subset of packages which would pull in the kitchen sink
+otherwise.
+.It Ar DONT_CLEAN_LOCKS
+By default,
+.Nm
+will clean old locks from dpb running on the same host that no longer exist,
+provided they didn't end in error.
+This is usually the right thing to do after a crash, or after killing dpb
+abruptly.
+Sometimes, one may want manual control over which locks to remove.
+.It Ar FETCH_JOBS
+Alternate way to specify the number of fetch jobs.
+.It Ar FETCH_USER
+User for all fetch activities if possible
+.Po defaults to
+.Ar _pfetch
+.Pc .
+.It Ar FETCH_TIMEOUT
+Timeout (in seconds) after which fetches that don't show
+any progress will be killed.
+.It Ar FTP_ONLY
+Don't fetch distfiles that are not allowed for ftp.
+.It Ar HISTORY_ONLY
+Don't fetch or build anything.
+Only run
+.Nm
+to figure out old distfiles and update
+.Pa %f/history .
+.It Ar LOGDIR
+Alternate way to specify the logging directory.
+.It Ar LOG_USER
+User
+for all log files if possible
+.Po defaults to
+.Ar build_user
+.Pc .
+.It Ar LOCKDIR
+Alternate way to specify the locking directory.
+.It Ar MIRROR
+Applicable to fetch modes.
+If 0, will only fetch normal
+.Ev DISTFILES
+.Po
+default for
+.Nm Fl f
+.Pc .
+If 1, will also fetch extra
+.Ev SUPDISTFILES
+.Po
+default for
+.Nm Fl F
+.Pc .
+.It Ar NO_CHECKSUM
+Do not run
+.Ar checksum
+again for files already fetched.
+.It Ar NO_CURSOR
+Make the terminal cursor invisible if possible.
+Avoids flickering on slow graphics cards.
+.It Ar NO_BUILD_STATS
+Disable reading/saving of default build stats under
+.Pa ${DISTDIR}/build-stats/${ARCH} .
+.It Ar NO_HISTORY
+Do not update the distfiles history.
+For instance, if
+.Nm
+is run a second time after a problem during the first run.
+.It Ar PORT_USER
+User that can write to the ports tree.
+Not really used for anything yet.
+.It Ar RECORD
+Define a file which will save all terminal output.
+Mostly useful for presentations, as a way to save
+.Nm dpb
+output and replay it later at a faster rate.
+Defaults to
+.Pa %L/term-report.log ,
+can be set to nothing to disable.
+.It Ar STARTUP
+Define a start-up script on the command-line, override any host file contents.
+.It Ar STUCK_TIMEOUT
+Timeout (in seconds * speed factor) after which tasks that don't show
+any progress will be killed.
+This can be instead set on a per-core basis as the
+.Sq stuck
+property.
+Note that this will always be divided by the core's speed factor.
+.It Ar SYSLOG
+Make
+.Nm
+call
+.Xr syslog 3
+on every task start/end while creating packages.
+This does produce lots of messages, it is intended to route the logging
+on another machine, while tracking down panics and other hangs.
+.It Ar WANTSIZE
+Alternate way to specify
+.Fl s .
+.El
+.It Fl e
+The listing job is extra and won't be given back to the pool when it's
+finished.
+.It Fl f Ar m
+Create
+.Ar m
+jobs for fetching files.
+Those are separate from the build jobs, since they don't consume cpu, and they
+run on the localhost.
+Defaults to 2.
+Can be set to 0 to bypass fetching jobs entirely,
+and reduce
+.Nm
+memory footprint by a lot.
+.It Fl F Ar m
+Fetch-only mode, for mirroring hosts.
+Do not build any package but fetch everything, disregarding
+.Ev BROKEN
+and
+.Ev ONLY_FOR_ARCHS
+information.
+Create
+.Ar m
+localhost jobs for fetching files.
+.It Fl h Ar hosts
+File with hosts to use for building.
+One host per line, plus properties, such as:
+.Bd -literal -offset indent
+espie@aeryn jobs=4 arch=i386
+.Ed
+.Pp
+Lines starting with a known variable name such as
+.Bd -literal -offset indent
+STARTUP=path
+.Ed
+or
+.Bd -literal -offset indent
+FETCH_JOBS=5
+.Ed
+can also be set inside a configuration file, to reduce the number of
+options you must pass on the command line.
+.Pp
+The special hostname
+.Ar DEFAULT
+can be used to preset defaults.
+It should be used at the start of the file.
+.Pp
+Use
+.Ar localhost
+to specify the local machine.
+.Nm
+will special-case it and not use
+.Xr ssh 1
+to connect.
+.Pp
+Properties are as follows:
+.Bl -tag -width memory=150
+.It arch=value
+Architecture of the concerned host.
+(there should be a startup task to check consistency, but
+currently this has to be set manually on heterogeneous networks.)
+.It always_clean=n
+Set to 0 or 1 on per-host basis.
+See
+.Ar ALWAYS_CLEAN
+parameter.
+.It build_user=user
+Use
+.Ar user
+for non root jobs if possible (defaults to
+.Xr whoami 1
+value).
+.It chroot=dir
+Chroot to
+.Ar dir
+before building.
+.It jobs=n
+Number of jobs to run on that host, defaults to hw.ncpu.
+.It junk=n
+Junk unused packages each n steps.
+See
+.Fl J
+option.
+.It memory=thr
+Build everything below that wrkdir threshold with
+.Ev USE_MFS Ns = Ns Sq Yes ,
+assuming the ports tree has been configured so that
+.Ev WRKOBJDIR_MFS
+points to a memory filesystem.
+.Ar thr
+is the sum, in KBytes, of ports that will be allowed to build in memory.
+.Nm
+understands suffixes, such as
+.Fl M Ar 2G
+or
+.Fl M Ar 500M .
+.Pp
+Note that you should always allow for some margin, as
+.Nm
+makes its decision based on the size information collected during previous
+builds, so in cases of significant updates, the work directory size will
+usually grow.
+.It nochecksum=0/1
+Defaults to 1.
+During the junk stage, run
+.Xr pkg_delete 1
+with the
+.Fl q
+(no checksum) option.
+.It parallel=p
+Run big ports on several cores.
+See
+.Fl p
+option.
+.It repair=0/1
+Defaults to 1.
+Run
+.Xr pkg_add 1
+with the repair option.
+This is useful on some bulk machines which tend to crash a lot, leaving
+.Pa /var/db/pkg
+in a weird state.
+.It sf=n
+Speed factor.
+An estimate of that machine's speed with that number of jobs
+compared to other machines in the same network.
+Works better with small values, in the range of 1..50.
+The machine (or machines) with the highest speed factor will
+get access to all jobs, whereas other machines will be clamped
+to stuff which does not take too long.
+Requires previous build information to be effective.
+Defaults to 1.
+.It small=s
+Small threshold (in seconds * sf):
+ports known to build under that duration are deemed to be small, so
+.Nm
+won't bother calling fine-grained steps for patch/configure/fake.
+It will go straight to build and package instead.
+Defaults to 120 seconds.
+.It squiggles=n
+Number of squiggles on this host (see
+.Sq the squiggle heuristics
+below).
+Defaults to 1 squiggle for hosts with 4 jobs or more, 0.7 for hosts with more than 1 job,
+0 for single job hosts.
+.It stuck=s
+Stuck timeout (in seconds * sf) after which tasks which show no progress
+will get killed.
+.It timeout=s
+Defines a specific connection timeout for ssh to that host.
+.El
+.Pp
+There are no fine-grained options to control
+.Xr ssh 1
+options, as those can be specified through virtual host declarations in
+.Xr ssh_config 5 .
+.It Fl I Ar pathlist
+List of
+.Xr pkgpath 7
+to install, on the local box.
+This will also add them to the list of things to build.
+.It Fl J Ar p
+Override value for the
+.Dq junk
+property.
+Delete unneeded installed packages during the build.
+Each
+.Ar prepare
+stage is followed by a
+.Ar show-prepare-results
+stage.
+After every
+.Ar p
+new dependencies, it will be followed by a
+.Ar junk
+stage which uses
+.Xr pkg_delete 1
+with the
+.Fl aXI
+options to delete automatically installed packages that are currently
+not needed.
+.Pp
+.Nm
+keeps track of list of dependencies on a given host, by storing each
+dependency list in the lockfile corresponding to the package being built.
+.Pp
+To avoid a race condition between the
+.Ar depends
+and
+.Ar junk
+stages,
+.Nm
+allows only one job on a given host to be in the
+.Ar depends
+\&...
+.Ar junk
+stages at one time, by using a per-host lock.
+.Pp
+Defaults to
+.Ar 150 .
+Can be disabled by setting to
+.Ar 0 .
+.Pp
+Some ports, most notably cmake-based, have an annoying dependency handling
+bug: they compute their makefile dependencies based on all include files
+present, not just the ones that are actually enabled.
+Those ports' build may be broken by a
+.Ar junk
+phase that removes some unused includes that were added as makefile
+prerequisites.
+Those ports should be annotated with
+DPB_PROPERTIES = nojunk
+until that bug is fixed:
+while a port with the
+.Sq nojunk
+property is building,
+.Ar junk
+will be postponed.
+.Pp
+Those ports will be marked with a
+.Sq \&!
+in the display, to make it more obvious why junk seems to be ineffective.
+.Pp
+Note that the
+.Sq nojunk
+property is still active for ports in error, in the belief that trivial fixes
+can be made that will allow the port build to finish.
+.It Fl j Ar n
+Number of jobs to run on a single host (defaults to hw.ncpu).
+.It Fl L Ar logdir
+Choose a log directory.
+.Po
+Defaults to
+.Pa %p/logs/%a
+.Pc .
+.It Fl l Ar lockdir
+Choose a lock directory.
+.Po
+Defaults to
+.Pa %L/locks
+.Pc .
+Override to keep local, as locks don't really like NFS.
+.It Fl M Ar threshold
+Build ports below the memory threshold under a memory
+filesystem, as configured through
+.Ev WRKOBJDIR_MFS
+.Po
+see
+.Xr bsd.port.mk 5
+.Pc .
+.Ar threshold
+is the sum, in KBytes, of ports allowed to build there.
+.It Fl m
+Force tty-style reporting.
+.It Fl P Ar pathlist
+Read list of
+.Xr pkgpath 7
+from file.
+.It Fl p Ar parallel
+Override value for the
+.Dq parallel
+property.
+.Pp
+Run big jobs on several cores on the same host, by using
+MAKE_JOBS=k .
+.Pp
+Once such a job has started,
+.Nm
+will not start new jobs on the same host until the big job has
+stolen enough cores from other finishing jobs.
+.Pp
+Only big ports which are safe for parallel building (annotated with
+DPB_PROPERTIES = parallel in their Makefile) will be affected.
+.Pp
+It is advisable to set k to an integral fraction of the
+number of cores available on a given host.
+.Ar parameter
+can be an integer, or of the form
+.Sq /n ,
+in which case,
+.Nm
+will set k to a fraction of the total number of jobs
+on the machine, but never below 2.
+.Pp
+Defaults to
+.Sq /2 .
+.It Fl q
+Don't quit while errors/locks are around.
+.It Fl r
+Random build order.
+Disregard any kind of smart heuristics.
+Useful to try to find missing build dependencies.
+.It Fl R
+Rebuild existing packages based on discrepancies between the package
+signature and what the port says it should be.
+Concretely, use to run a partial bulk build after some library change.
+.Pp
+Note that
+.Fl R
+won't always work, as rebuilding a package when another version is already
+installed is not supported.
+Building in a chroot is strongly recommended.
+.It Fl S Ar logfile
+Read
+.Ar logfile
+as an initial workdir size log.
+.It Fl s
+Compute workdir sizes before cleaning up, and stash them in log file
+.Pa %L/size.log .
+Also maintain a rolling log of build sizes under
+.Pa %f/build-stats/%a-size .
+In order to save time,
+.Nm
+will actually not always compute new sizes for known directories, but mostly
+for new ones, or when the package name changes.
+.It Fl u
+Update existing packages during dependency solving.
+Can be used to run a bulk-build on a machine with installed packages,
+but might break a bit, since some packages only build on a clean machine
+right now.
+.It Fl U
+Insist on updating existing packages during dependency solving,
+even if the new package apparently didn't change.
+.It Fl x
+No tty report, only report really important things, like hosts going down
+and coming back up, build errors, or builds not progressing.
+.It Fl X Ar pathlist
+Read a list of
+.Xr pkgpath 7
+from file, and pass them along in the junk phase:
+those are packages that should stay on the machine if they've been
+installed by a dependency.
+Can be used to avoid endlessly removing/reinstalling the most common
+packages, e.g.,
+.Pa devel/gmake .
+.El
+.Pp
+.Nm
+figures out in which order to build things on the fly, and constantly
+displays information relative to what's currently building.
+There's a list of what is currently running, one line per job.
+Those jobs are ordered in strict chronological order, which means that
+long running builds will tend to percolate to the top of the list.
+Normal jobs look like this:
+.Bd -literal -offset indent
+www/mozilla-firefox(build) [9452] 41% unchanged for 92 seconds
+.Ed
+.Pp
+This contains:
+.Bl -dash
+.It
+an optional
+.Sq ~
+squiggle marker (see below),
+.It
+the pkgpath being built,
+.It
+the step currently being run,
+.It
+an optional
+.Sq \&!
+for ports with the
+.Sq nojunk
+property.
+.It
+an optional
+.Sq +
+for ports built in memory.
+.It
+the pid running that task (note that this is always a pid on the host
+running dpb: for distributed builds, it will be an
+.Xr ssh 1
+to another machine),
+.It
+the current size of the log file (displayed as a percentage if previous
+build statistics are available).
+.It
+and a possible notice that things might be stuck when
+the log file doesn't change for long periods.
+.El
+.Pp
+And fetch jobs look like this:
+.Bd -literal -offset indent
+<dist-3.0.tgz(#1) [4321] 25%
+.Ed
+.Pp
+This contains:
+.Bl -dash
+.It
+the file being fetched
+.It
+the number of the
+.Ev MASTER_SITE
+being tried
+.It
+the pid of the
+.Xr ftp 1
+process (note that fetch jobs are always local).
+.It
+a progress percentage.
+.El
+.Pp
+This is followed by a host line, containing the name
+of each host used by dpb.
+Host names may be tagged with kde3 or kde4.
+They are followed by a
+.Sq `-'
+for unresponsive hosts, and the pid of the ssh master
+for distant hosts.
+.Pp
+This ends with a summary display:
+.Bl -tag -width BB=
+.It I=
+number of built packages that can be installed.
+.It B=
+number of built packages, not yet known to be installable,
+because of run depends that still need to be built.
+.It Q=
+number of packages in the queue, e.g., stuff that can be built now, assuming
+we have a free slot.
+.It T=
+number of packages to build, where dependencies are not yet resolved.
+.It F=
+number of distfiles to fetch, when
+.Fl f
+is used.
+.It !=
+number of ignored packages.
+Details in
+.Pa engine.log .
+.It L=
+list of packages that cannot currently be built because of locks.
+.It E=
+list of packages in error, that cannot currently be built.
+.It H=
+list of packages that haven't shown up yet, usually due to nfs, but
+watch out for revision bumps.
+.El
+.Pp
+If those three lists are empty, they won't even show up.
+Packages in errors may be followed by a
+.Sq \&!
+if they prevent junk from happening.
+.Pp
+Note that those numbers refer to pkgpaths known to
+.Nm .
+In general, those numbers will be slightly higher than the actual number
+of packages being built, since several paths may lead to the same package.
+.Pp
+.Nm
+uses some heuristics to try to maximise the queue as soon as possible.
+There are also provisions for a feedback-directed build, where information from
+previous builds can be used to try to build long-running jobs first.
+.Pp
+Similarly, fetches will use the continue option of
+.Xr ftp 1 ,
+since distfiles are checksummed after the fetch anyways.
+.Ss THE SQUIGGLE HEURISTICS
+However, on machines with lots of cores, the basic scheduling heuristics
+yields a tail of very small jobs, where
+.Nm
+will mostly wait on
+.Xr pkg_add 1
+to solve dependencies.
+Starting with
+.Ox 5.5 ,
+a new mechanism (squiggles) was introduced to counter-balance this effect:
+big machines devote some of their cores to
+.Sq squiggles ,
+jobs that walk the queue in reverse, thus building smallest ports first.
+As a result, small ports are built as a trickle alongside the largest ports,
+thus offsetting the negative effect of the exponential queue for a large part.
+.Pp
+Note that
+.Sq squiggles
+can be a non-integral value, usually lower than 1, in which case they
+represent the fraction of cores that should be affected to squiggles,
+as decided randomly at the start of each build.
+0.7 or 0.8 might be a good choice for dual core machines.
+.Ss DPB PROPERTIES
+The
+.Xr bsd.port.mk 5
+variable
+.Ev DPB_PROPERTIES
+may hold several annotations that only
+.Nm
+will look at.
+These properties are as follows:
+.Bl -tag -width pkgpathlong
+.It Ar lonesome
+Large port that stresses the memory limits of the machine, should be built
+alone.
+Prevents
+.Nm
+from scheduling anything else on the same host after it starts building.
+\".It Ar memoryhog
+.It Ar nojunk
+Port that hardcodes includes in its Makefile mechanisms.
+Prevents
+.Ar junk
+from running while port is building.
+.It Ar parallel
+Port that can be built in parallel, uses
+.Ev MAKE_JOBS
+and several build slots.
+.It Ar tag:kde3
+kde3 port that conflicts with kde4 ports.
+Prevent scheduling ports with
+.Ar tag:kde4
+on the same host.
+.It Ar tag:kde4
+kde4 port that conflicts with kde3 ports.
+Prevent scheduling ports with
+.Ar tag:kde3
+on the same host.
+.El
+.Sh THE SECURITY MODEL OF DPB
+When
+.Nm
+is run as root, it uses a privilege drop model instead of the
+dangerous privilege elevation model of
+.Xr doas 1 .
+When run as root, by default,
+.Ar _pbuild
+is used as the build and log user, and
+.Ar _pfetch
+is used as the fetch user.
+.Bl -bullet
+.It
+Start
+.Nm
+as root.
+.It
+.Nm
+will drop privileges for every operation except
+.Xr pkg_add 1 ,
+.Xr pkg_delete 1
+and the
+.Ar STARTUP
+script.
+.It
+For cluster builds,
+provide an
+.Xr ssh 1
+connection to distant hosts from root as root.
+.It
+.Ar build_user
+is used to build stuff locally or distantly (can be per-host), using:
+.Li chroot -u build_user /build_root
+(with
+.Pa /build_root
+=
+.Pa /
+if there is no actual chroot needed).
+It must have read access to ${DISTDIR} and ${PORTSDIR}, and write
+access to ${WRKOBJDIR}, ${PACKAGE_REPOSITORY}, and ${PLIST_REPOSITORY}.
+It does not require network access.
+.It
+.Ar LOG_USER
+is used to open all log files.
+.Ar LOG_USER
+only needs to exist locally.
+It needs write access to the log directories, including
+${DISTDIR}/build-stats.
+It does not need network access.
+.It
+.Ar FETCH_USER
+is used to fetch distfiles and handle corresponding log info.
+It needs write access to ${DISTDIR}, and network access.
+Thus,
+.Xr ftp 1
+does not happen as root.
+.It
+.Ar _dpb
+is used as a fail-safe for any other activities that do not require any rights.
+.It
+.Nm
+creates local directories as root, then gives them to the appropriate user.
+.El
+.Sh LOCKS AND ERRORS
+.Nm
+still uses the normal ports tree mechanism while building, which includes
+.Ev LOCKDIR .
+When starting up
+.Nm
+will normally detect stale locks from old dpb runs, and remove them.
+If this does not happen, builds will stay stuck in their initial stage,
+that is:
+.Ar show-prepare-results , patch , build
+depending on the port.
+A telltale message
+.Sq Awaiting lock ...
+can be found in the corresponding logfile
+.Pa paths/pkgpath.log
+.Pp
+In addition, when building a package,
+.Nm
+produces a lockfile in the locks directory, whose name is deduced from
+the basic pkgpath with slashes replaced by dots.
+This lockfile is filled with such info as the build start time or the host,
+or the needed dependencies for this pkgpath.
+.Pp
+The lockfile will also contain the name of a parent pkgpath, for paths that
+were discovered as dependencies.
+This is particularly useful for bogus paths, where it would be hard to
+know where the path came from otherwise.
+.Pp
+At the end of a successful build, these lockfiles are removed.
+The lock will stay around in case of errors.
+.Po
+raw
+value from
+.Xr wait 2
+.Pc ,
+and the name of the next task in the build pipeline (with todo=<nothing>
+in case of failure during clean-up).
+Normal list of tasks is:
+.Ar depends prepare fetch patch configure build fake package clean .
+.Pp
+At the end of each job,
+.Nm
+rechecks the locks directory for existing lockfiles.
+If some locks have vanished,
+it will put the corresponding paths back in the queue and attempt
+another build.
+.Pp
+This eases manual repairs: if a package does not build, the user can look
+at the log, go to the port directory, fix the problem, and then remove the lock.
+.Nm
+will pick up the ball and keep building without interruption.
+.Pp
+It is perfectly safe to run several
+.Nm
+in parallel on the same machine.
+This is not optimal, since each
+.Nm
+ignores the others, and only uses the lock info to avoid the other's
+current work, but it can be handy: in an emergency, one can start a second
+.Nm
+to obtain a specific package right now, in parallel with the original
+.Nm .
+.Pp
+Note that
+.Nm
+is very careful not to run two builds from the same pkgpath at the
+same time, even on different machines:
+in some cases, MULTI_PACKAGES and FLAVOR combinations may lead to the
+same package being built simultaneously, and since the package repository
+is shared, this can easily lead to trouble.
+.Pp
+Handling of shared log files and history is also done very carefully by
+systematically appending to files or using atomic mv operations.
+.Pp
+For obvious reasons, this won't work as well with masters running on distinct
+machines sharing their logs through NFS.
+.Ss BUILD CYCLES
+There are some various interdependencies in package builds that can be hard
+to trace in case something goes wrong.
+Refer to
+.Pa summary.log
+to fix those specific issues.
+.Sh AFFINITY
+.Nm
+now maintains a list of pkgpath-per-host that are currently building in the
+.Pa affinity
+directory of its log directory, along with building-in-memory status.
+.Pp
+That information is only wiped out when a given build finishes successfully.
+.Pp
+Otherwise
+.Nm
+will try to restart that build on the same host, which can be handy if you
+interrupt
+.Nm
+while it is building a large port, or if you remove a lock after fixing a
+problem.
+.Sh TAGS FOR BUILDING KDE
+Currently, kde3 and kde4 can't be built simultaneously.
+Conflicting ports have been annotated with
+DPB_PROPERTIES=tag:kde3 ,
+DPB_PROPERTIES=tag:kde4
+respectively.
+.Pp
+.Nm
+now keeps track of those tags, and will postpone ports with the wrong
+tag while a given host is used by the other tag.
+.Pp
+This heavily relies on the
+.Ar junk
+stage to clean-up hosts periodically,
+and it can even forcibly provoke a
+.Ar junk
+stage even if junk=0.
+.Pp
+This
+.Sq force-junk
+stage is actually implemented as a pseudo path called
+.Ar junk-proxy ,
+which only does junk.
+.Pp
+In order for builds to proceed gracefully, machines should start
+in a clean slate, without kde3 or kde4 installed.
+.Pp
+As a special-case, failing ports with a kde3 or kde4 tag will not
+interfere with clean-up, so that hosts do not get locked down to
+a specific tag.
+This also means that their dependencies
+may vanish before human intervention addresses the problem.
+.Pp
+This is supposed to be a temporary hack, as kde4 is large and
+having official packages helps a great deal in debugging it.
+.Sh EXTERNAL CONTROL
+If
+.Fl D Ar CONTROL Ns = Ns Ar path
+if used,
+.Nm
+will create a Unix socket at the given
+.Pa path ,
+only accessible by
+.Ar LOG_USER ,
+that can accept a few commands, .e.g.,
+usable as
+.Li nc -U path
+.Pp
+Currents commands are as follows:
+.Bl -tag -offset aaaa -with addhost
+.It Cm addhost Ar hostline
+Add a new host
+.It Cm addpath Ar fullpkgpath ...
+Add new fullpkgpath to scan
+.It Cm bye
+close the socket connection.
+.It Cm dontclean Ar pkgpath ...
+Add new pkgpath to list of paths that should not be cleaned after build
+.It Cm help
+Self explanatory
+.It Cm stats
+Show the current stats line
+.El
+.Pp
+.Sh SHUTTING DOWN GRACEFULLY
+.Nm
+periodically checks for a file named
+.Pa stop
+in its log directory.
+If this file exists, then it won't start new jobs, and shutdown when
+the current jobs are finished unless
+.Fl q .
+.Pp
+.Nm
+also checks for files named
+.Pa stop-<hostname>
+in its log directory.
+If such a file exists, then it won't start new jobs on
+the corresponding machine.
+.Sh FILES
+Apart from producing packages,
+.Nm
+may create temporary files as
+.Pa ${FULLDISTDIR}/${DISTFILE}.part .
+.Pp
+In fetch mode
+.Po
+.Fl f
+and
+.Fl F
+.Pc ,
+.Nm
+populates
+.Pa ${DISTDIR}/by_cipher/sha256
+with links.
+It also uses
+.Pa ${DISTDIR}/distinfo
+and
+.Pa ${DISTDIR}/history
+as a
+.Sq permanent log :
+.Bl -tag -width distinfo
+.It distinfo
+cache of distfiles checksum.
+Contains all
+.Xr sha256 1
+checksums of known files under
+.Pa ${DISTDIR} .
+Fetching uses this to avoid re-checksumming known files.
+.It history
+Log of old files under distinfo.
+After successfully scanning a full ports tree
+.Po
+.Nm Fl a
+.Pc ,
+the fetch engine knows precisely which files are needed by the build
+(and their checksums).
+Anything that is
+.Bl -bullet
+.It
+recorded in distinfo but unneeded
+.It
+recorded in distinfo but with the wrong checksum
+.It
+not recorded in distinfo, but not needed
+.El
+will be entered at the end of history as a line:
+.Pp
+.Li ts SHA256 (file) = value
+.Pp
+with
+.Ar ts
+a timestamp from Unix epoch.
+.Pp
+When cleaning up old files, with a tool such as
+.Xr clean-old-distfiles 1 ,
+it is vital to check both the checksum and
+the file name: since mirroring stores permanent links under
+.Pa by_cipher ,
+files which are still needed will appear in history under their old
+checksums, as an indication the link should be removed, but possibly not
+the file itself.
+.El
+.Pp
+If
+.Pa ${DISTDIR}
+ever becomes corrupted,
+removing
+.Pa ${DISTDIR}/distinfo
+will force
+.Nm
+into checking all files again.
+.Pp
+All those files belong to the
+.Ar FETCH_USER
+if it is defined.
+They should be readable for the
+.Ar build_user .
+.Pp
+.Nm
+also records rolling build statistics under
+.Pa ${DISTDIR}/build-stats/${ARCH} ,
+and uses them automatically in the absence of
+.Fl b Ar logfile .
+That file belongs to the
+.Ar LOG_USER
+if it is defined.
+.Pp
+If
+.Fl s
+is used, size information for successful builds will be recorded under
+.Pa ${DISTDIR}/build-stats/${ARCH}-size
+.Po
+by default, location adjustable with
+.Fl S Ar sizelog
+.Pc .
+This is then reused for the mfs threshold option.
+That file also belongs to the
+.Ar LOG_USER
+if it is defined.
+.Pp
+.Nm
+also maintains a list of pkgpath frequencies
+.Pa ${DISTDIR}/build-stats/${ARCH}-dependencies ,
+filled at end of LISTING if
+.Fl a .
+This list will be automatically reused when restarting a build:
+a quick LISTING of the most important dependencies will happen
+before the general LISTING,
+in order to prime further LISTING steps with most common ports first.
+.Pp
+.Nm
+will also create a large number of log files under
+.Pa ${PORTSDIR}/logs/${ARCH} ,
+which will belong
+to
+.Ar LOG_USER
+if it is defined:
+.Bl -tag -width engine.log
+.It Pa affinity/
+Affinity information.
+One file per full pkgpath, with slash replaced by dots
+like so:
+.Pa affinity/lang.ghc,-main .
+.It Pa affinity.log
+On startup
+.Nm
+reads existing affinity information, and records it in that log,
+together with its pid.
+This log just exists to verify, along with
+.Pa engine.log ,
+whether correct affinity was heeded.
+.It Pa awaiting-locks.log
+This is purely for gathering performance statistics, about how much
+lock contention happened around
+.Xr pkg_add 1
+and
+.Xr pkg_delete 1
+usage.
+Plotting cumulated time may help in fine-tuning squiggles parameters.
+.It Pa build.log
+Actual build log.
+Each line summarizes build of a single pkgpath, as:
+.Sq pkgpath host time logsize (detailed timing)[!]
+where time is the actual build time in seconds, host is the machine name
+where this occurred, logsize is the corresponding log file size,
+and a ! is appended in case the build didn't succeed.
+.Pp
+The detailed timing info gives a run-down of the build, with clean, fetch,
+prepare, patch (actually extract+patch), configure, build, fake, package, clean
+detailed timing info.
+Note that the actual build time starts at
+.Sq extract
+and finishes at
+.Sq package .
+.It Pa concurrent.log
+Shows the actual concurrency achieved as a result of job starvation /
+parallel handling.
+Only gets a new line when the value changes: pid timestamp jobs
+.It Pa debug.log
+contains various information related to the main engine spinning (RTFS, haven't
+figured that one yet) along with the more useful warning and die traces that
+happen when something wrong occurs.
+Especially useful for the warning messages that tend to be overwritten by
+subsequent displays.
+.It Pa dist/<distfile>.log
+Log of the
+.Xr ftp 1
+process(es) that attempted to fetch the distfile.
+.It Pa dump.log
+A long log file generated at the end of build that yields any information
+pertinent to ports still in the
+.Sq to build
+and the
+.Sq built
+queues.
+See also
+.Pa summary.log
+for an expurged version of same.
+.It Pa engine.log
+Build engine log.
+Each line corresponds to a state change for a pkgpath and starts with the pid
+of
+.Nm ,
+plus a timestamp of the log entry.
+.Bl -tag -width BB:
+.It ^
+pkgpath temporarily put aside, because a job is running in the same directory.
+.It !
+pkgpath ignored, either directly, or indirectly because a dependency was
+ignored.
+End of the line states reason why ignored.
+.It A
+affinity mismatch: path considered for build, but not the right host,
+followed by the affinity information.
+.It B
+pkgpath built / distfile found.
+.It C
+forcible clean-up before building a port with a kde tag.
+.It E
+error in build or fetch.
+.It F
+distfile queued for download.
+.It H
+package still not found due to nfs on this run.
+.It I
+pkgpath can be installed.
+.It J
+job to build pkgpath started.
+Also records the host used for the build.
+.It K
+kde mismatch, no build until host has been cleaned up.
+.It L
+job did not start, existing lock detected.
+.It N
+job did not finish.
+The host may have gone down.
+.It P
+built package is no longer required for anything.
+.It Q
+pkgpath queued as buildable whenever a slot is free.
+.It T
+pkgpath to build / distfile to download.
+.It V
+pkgpath put back in the buildable queue, after job that was running in
+the same directory returned.
+.It X
+only happens when rescanning after an error.
+The engine temporarily locks paths that are incomplete (detained).
+These will be kept in a separate list for later examination until the
+end of the new scan.
+.It x
+only happens when rescanning after an error.
+Releases a path for building after the new scan is finished.
+.It Y
+affinity mismatch, but job will start on the wrong host anyways, as the queue
+contains no other buildable path.
+.El
+.Pp
+Please note that the engine is no longer run after each package build event
+because of performance considerations, so the
+.Sq Q
+and
+.Sq I
+changes may be delayed by a few
+.Sq B .
+.It Pa equiv.log
+Lists of equivalent pkgpaths for the build, when default flavors and default subpackages have been resolved.
+.It Pa fetch/bad.log
+List of URLs that did not lead to a correct distfile, either because
+they were not responding, or because of incorrect checksums.
+.It Pa fetch/good.log
+List of URLs that fetched correctly, along with timing statistics.
+.It Pa fetch/manually.log
+List of pkgpaths that require manual intervention, in human-readable form.
+.It Pa <hostname>.sig.log
+Complete library signature of the host.
+.It Pa init.<hostname>.log
+Captured output of the initialization job for each host.
+.It Pa junk.log
+Option
+.Fl J
+counts the number of dependencies directly added to decide when to run
+.Nm pkg_delete Fl a .
+This file sums up how many ports were built, and how many ports had
+dependencies each time
+.Nm
+decides to junk.
+.It Pa locks/
+Directory where locks are created.
+There are three types of locks:
+.Bl -bullet
+.It
+pkgpath locks for building, where the slash in a pkgpath is replaced
+with a dot like so:
+.Pa locks/devel.make
+to flatten the structure.
+.It
+distfile locks for fetching, using the distfile name without the path like so:
+.Pa locks/distfile.dist .
+.It
+host locks for dependency handling and junking, like so:
+.Pa locks/host:hostname .
+.El
+.It Pa packages/pkgname.log
+one file or symlink per pkgname.
+.It Pa paths/some/path.log
+one file or symlink per pkgpath.
+.It Pa performance.log
+Some parts of
+.Nm
+are computationally intensive, such as the engine runs to determine
+new stuff that can be built, and the actual display reports.
+.Pp
+Both those activities are rate-limited, so that
+.Nm
+doesn't run its engine at each new package build,
+and doesn't update its display every time there is a phase change.
+.Pp
+Lines tagged with
+.Sq ENG
+correspond to the engine;
+lines tagged with
+.Sq REP
+correspond to the display reports.
+.Pp
+Lines ending with a dash
+.Sq -
+correspond to new activity that didn't trigger
+a computation.
+.Pp
+Other lines will feature a plus
+.Sq +
+for normal runs, or an exclamation point
+.Sq !
+for forced runs, followed by two numbers:
+the next timestamp at which we'll be allowed to run, and
+a measure of how much time it took to run this pass.
+.Pp
+That information is mostly relevant while
+.Nm
+is building lots of small packages very quickly.
+.It Pa signature.log
+Discrepancies between hosts that prevent them from starting up.
+.It Pa size.log
+Size of work directory at the end of each build, built only with
+.Fl s .
+.It Pa stats.log
+Simple log of the B=... line summaries.
+Mostly useful for making plots and tweaking performance.
+.It Pa stop
+Not a logfile at all, but a file created by the user to stop
+.Nm
+creating new jobs.
+.It Pa stop-<hostname>
+Not a logfile at all, but created by the user to stop hostname creating
+new jobs.
+.It Pa summary.log
+A summary file generated at end of build that lists packages not built
+or not installable, along with a reason for it.
+This summarizes packages not built because of existing locks, because of
+errors, but also because they depend on something that was not built.
+.Pp
+In that last case,
+.Pa summary.log
+contains a chain of dependencies leading to the problematic package, or
+in case of build cycles, stopping at the first loop.
+.It Pa term-report.log
+Saves all terminal output, so that it can be replayed at hi speed with
+.Xr dpb-replay 1 .
+.It Pa vars.log
+Logs the directories that were walked in the ports tree for dependency
+information, including the path to a dependency that triggered this
+particular step.
+.El
+.Sh DIAGNOSTICS
+.Bl -tag -offset aaaa -width truc
+.It Waiting for hosts to finish STARTUP...
+Displayed on the console while
+.Nm
+is setting up hosts, getting essential data from the ports tree,
+running a
+.Ar STARTUP
+script, collecting base library signatures.
+.It stuck on <lockfilename>
+Display on the console when
+.Nm
+detects a "frozen" port has happened outside of
+.Nm Ns 's
+purview, namely because the ports tree itself has that specific
+port locked without
+.Nm Ns 's
+knowledge.
+See
+.Xr bsd.port.mk 5 ,
+.Xr dolock 1 .
+.It (Junk lock obtained for <host> at <time>)
+.It (Junk lock released for <host> at <time>)
+Printed in a
+.Pa paths/pkgpath.log
+file when attempting to get a
+.Sq junk lock .
+On a given host, all dependency operations are serialized.
+The dependency computation itself is handled by the main
+.Nm
+process, which needs to know exactly which dependencies are used
+at a given point, so that
+.Ar junk
+can clean up the host correctly.
+In particular,
+.Ar junk
+will not clean up dependencies already scheduled for installation.
+Ports that do no obtain the lock on first try are put to sleep.
+.It Received IO
+Printed in a
+.Pa paths/pkgpath.log
+file when woken up before trying attempting to obtain a
+.Ar junk
+lock again...
+.It Woken up <fullpkgpath>
+Printed in a
+.Pa paths/pkgpath.log
+when waking another task by sending it SIGIO,
+so that it may attempt to obtain the junk lock again.
+.It (Junk lock failure for <host> at <time>)
+All ports sleeping for a
+.Ar junk
+lock are woken at the same time, so only one of them will obtain the lock,
+and the others will fail and be put to sleep again.
+.It Short-cut: depends already handled by <fullpkgpath>
+Printed in a
+.Pa paths/pkgpath.log
+when a port wakes up after others that ran
+.Xr pkg_add 1 .
+As
+.Nm
+maintains dependencies for a given host globally, it coalesces depends lists
+together.
+.It Don't run junk because nojunk in <fullpkgpath>
+Printed in a
+.Pa paths/pkgpath.log
+while evaluating whether to run
+.Ar junk .
+Normally,
+.Ar junk
+happens at regular intervals, but ports marked
+.Sq nojunk
+will delay that.
+.Nm
+still keeps track of attempted junks.
+.It Still tainted: <bool>
+A host may have a tag (kde3/kde4) that prevents building differently tagged
+ports.
+This will be cleansed by
+.Ar junk
+eventually.
+This prints in
+.Ar path/pkgpath.log
+to indicate whether this particular
+.Ar junk
+will keep the host tainted with a tag or not.
+.It Forced junk, retainting: <tag>
+Printed at end of
+.Ar prepare-results ,
+when an eventual junk was run even though some ports still hold a tag.
+.It Can't run junk because of lock on <fullpkgpath>
+.Ar junk
+can't happen because
+.Ar fullpkgpath
+is locked and is marked
+.Sq nojunk .
+.It Avoided depends for <dependencies>
+As dependencies are handled globally per-host, some ports can avoid
+.Xr pkg_add 1
+altogether because another port already installed the correct dependencies.
+.It SPINNING ON MAIN
+Printed in
+.Ar debug.log ,
+this is an actual bug: the engine said it can build, there are cores available,
+but
+.Nm
+can't start a new build job.
+.It SPINNING ON FETCH
+Printed in
+.Ar debug.log ,
+this is an actual bug: the engine said it can fetch, there are fetching
+cores available, but
+.Nm
+can't start a new fetch job.
+.It KILLED: <job> stuck at <somewhere>
+Printed in
+.Ar path/pkgpath.log
+when a port exceeds its timeout.
+.El
+.Sh BUGS AND LIMITATIONS
+.Nm
+performs best with lots of paths to build.
+When just used to build a few ports, there's a high risk of starvation
+as there are bottlenecks in parts of the tree.
+.Pp
+Fetch jobs don't deal with checksum changes yet:
+if a fetch fails because of a wrong checksum, if you update the distinfo
+file and remove the lock,
+.Nm
+won't pick it up.
+.Pp
+Note that
+.Nm
+does not manage installed packages in any intelligent way, it will just
+call
+.Xr pkg_add 1
+during its depend stage to install its dependencies.
+With
+.Fl u ,
+it will call pkg_add -r.
+With
+.Fl U ,
+it will call pkg_add -r -D installed,
+but there is nothing else going on.
+This is especially true when using
+.Fl R ,
+ensure the machine is clean of possibly older packages first, or run
+.Nm
+with
+.Fl U .
+.Pp
+In particular
+.Fl R
+and
+.Fl J
+together may lead to strange issues.
+.Pp
+On heterogeneous networks, calibration of build info and choice of speed
+factors is not perfect, and somewhat a dark art.
+Using distinct speed factors on a build log that comes from a single
+machine works fine, but using the build info coming from several machines
+does not work all that well.
+.Pp
+.Nm
+should check
+.Pa /usr/include
+and
+.Pa /usr/X11R6/include
+for consistency, but it doesn't.
+.Pp
+When a host fails consistency check, there is not yet a way to re-add it
+after fixing the problem.
+You have to stop
+.Nm ,
+cleanup and restart.
+.Pp
+The default limits in
+.Pa login.conf
+are too small for bulk builds on any kind of parallel machines.
+Bump number of processes, file descriptors, and memory.
+.Pp
+Even though
+.Nm
+tries really hard to check heterogeneous networks for sanity (checking
+shared libraries and .la files), it is still dependent on the user to
+make sure all the hosts build ports the same way.
+.Pp
+Make sure your NFS setup is consistent.
+The ports dir itself should be exported or synchronized.
+Distfiles, the package repository,  and the plist repository should be exported,
+but WRKOBJDIR should not be on NFS unless you have absolutely no choice,
+or if you exhibit deep masochistic tendencies.
+Pay particular attention to discrepancies in
+.Pa /etc/mk.conf .
+.Pp
+Also,
+.Nm
+connects to external hosts through
+.Xr ssh 1 ,
+relying on
+.Xr ssh_config 5
+for any special cases.
+.Pp
+When fetching distfiles,
+.Nm
+may freeze and spin in a tight loop while the last distfiles are being fetched.
+This is definitely a bug, which has been around for quite some time, which
+is a bit difficult to reproduce, and hasn't been fixed yet.
+So if
+.Nm
+stops updating its display right around the end of fetch, you've hit the bug.
+Just kill
+.Nm
+and restart it.
+.Sh SEE ALSO
+.Xr clean-old-distfiles 1 ,
+.Xr dpb-replay 1 ,
+.Xr proot 1 ,
+.Xr pkgpath 7
+.Sh HISTORY
+The original
+.Nm dpb
+command was written by Nikolay Sturm.
+This version is a complete rewrite from scratch using all the stuff
+we learnt over the years to make it better.
+.Sh AUTHORS
+.An Marc Espie
diff --git a/share/man/man1/getpkgpath.1 b/share/man/man1/getpkgpath.1
new file mode 100644
index 00000000000..685116f70a6
--- /dev/null
+++ b/share/man/man1/getpkgpath.1
@@ -0,0 +1,36 @@
+.\"	$OpenBSD: getpkgpath.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt GETPKGPATH 1
+.Os
+.Sh NAME
+.Nm getpkgpath
+.Nd deduces PKGPATH from current directory
+.Sh SYNOPSIS
+.Nm getpkgpath
+.Ar dir
+.Sh DESCRIPTION
+.Nm
+deduces the
+.Ev PKGPATH
+corresponding to a given directory
+.Ar dir ,
+by trying to strip directories from the environment variable
+.Ev PORTSDIR_PATH .
+.Pp
+This is mostly a helper script for
+.Pa bsd.port.subdir.mk .
diff --git a/share/man/man1/out-of-date.1 b/share/man/man1/out-of-date.1
new file mode 100644
index 00000000000..0ff6638e37c
--- /dev/null
+++ b/share/man/man1/out-of-date.1
@@ -0,0 +1,50 @@
+.\"	$OpenBSD: out-of-date.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2011 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt OUT-OF-DATE 1
+.Os
+.Sh NAME
+.Nm out-of-date
+.Nd compare installed packages and the ports tree
+.Sh SYNOPSIS
+.Nm out-of-date
+.Bk -words
+.Op Fl mqxv
+.Ek
+.Sh DESCRIPTION
+.Nm
+checks installed packages and compares them to the ports tree.
+It displays packages that may need updating.
+Typical output is a subdirlist with comments, suitable for bulk building.
+.Pp
+Options are as follows:
+.Bl -tag -width pkgpathlonger
+.It Fl m
+Causes
+.Nm
+to always display the progress meter in cases it would not do so by default.
+.It Fl q
+Quick check, only verifies the names and not full update signatures.
+Use is discouraged.
+.It Fl v
+Verbose mode.
+Does not do anything yet.
+.It Fl x
+Disable progress meter.
+.El
+.Sh AUTHORS
+.An Bernd Ahlers
diff --git a/share/man/man1/outdated-perl-ports.1 b/share/man/man1/outdated-perl-ports.1
new file mode 100644
index 00000000000..d8ce1bfaecc
--- /dev/null
+++ b/share/man/man1/outdated-perl-ports.1
@@ -0,0 +1,46 @@
+.\"	$OpenBSD: outdated-perl-ports.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2011 Jasper Lievisse Adriaanse <jasper@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt OUTDATED-PERL-PORTS 1
+.Os
+.Sh NAME
+.Nm outdated-perl-ports
+.Nd compare CPAN releases and the ports tree
+.Sh SYNOPSIS
+.Nm outdated-perl-ports
+.Bk -words
+.Op Fl d
+.Op Fl m Ar maintainer
+.Ek
+.Sh DESCRIPTION
+.Nm
+retrieves and compares the packages list provided by CPAN with the ports
+recorded in
+.Pa ${PORTSDIR}/INDEX
+and reports which ports have a newer version available upstream.
+.Pp
+Options are as follows:
+.Bl -tag -width maintainer
+.It Fl d
+Causes
+.Nm
+to show the path (relative to ${PORTSDIR}) for the outdated port.
+.It Fl m Ar maintainer
+Only display ports that are being maintained by
+.Ar maintainer .
+This can be specified as name or e-mail address.
+.El
diff --git a/share/man/man1/pkg_subst.1 b/share/man/man1/pkg_subst.1
new file mode 100644
index 00000000000..ae641b89fbf
--- /dev/null
+++ b/share/man/man1/pkg_subst.1
@@ -0,0 +1,101 @@
+.\"	$OpenBSD: pkg_subst.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PKG_SUBST 1
+.Os
+.Sh NAME
+.Nm pkg_subst
+.Nd substitute variables' values in files
+.Sh SYNOPSIS
+.Nm pkg_subst
+.Op Fl ci
+.Op Fl B Ar basedir
+.Op Fl D Ar name Ns Op = Ns Ar value
+.Op Fl g Ar group
+.Op Fl m Ar mode
+.Op Fl o Ar owner
+.Op Ar file ...
+.Sh DESCRIPTION
+.Nm
+is used within the ports tree to substitute variable values, using
+the exact same conventions used by
+.Xr pkg_create 1 .
+Constructs like
+.Li ${VAR}
+will be replaced with the variable value, according to
+.Fl D Ar name Ns Op = Ns Ar value
+options.
+.Pp
+The options are as follows
+.Bl -tag -width Dname[=value]xxx
+.It Fl B Ar basedir
+Specify a basedir in which we know file properties.
+It is forbidden to copy files from elsewhere without
+.Fl m Ar mode .
+.It Fl c
+Copy and substitute.
+Number of
+.Ar file
+arguments must be even.
+They are paired as
+.Ar src1
+.Ar dest1 ,
+.Ar src2
+.Ar dest2 ...
+and each
+.Ar src
+file is substituted and copied into the corresponding
+.Ar dest
+file.
+.It Xo
+.Fl D
+.Ar name Ns Op = Ns Ar value
+.Xc
+Define
+.Ar name
+to
+.Ar value ,
+or to 1 if no value is given.
+.It Fl g Ar group
+Change group of destination file to
+.Ar group .
+.It Fl i
+Ignore changes to group or owner that failed.
+.It Fl m Ar mode
+Change mode of destination file to
+.Ar mode .
+.It Fl o Ar owner
+Change owner of destination file to
+.Ar owner .
+.El
+.Pp
+If no
+.Ar file
+arguments is given,
+.Nm
+acts as a filter from stdin to stdout.
+.Pp
+Without
+.Fl c ,
+each
+.Ar file
+is backed up as
+.Ar file.beforesubst
+before the substitution occurs.
+.Sh SEE ALSO
+.Xr install 1 ,
+.Xr pkg_create 1
diff --git a/share/man/man1/portbump.1 b/share/man/man1/portbump.1
new file mode 100644
index 00000000000..eebd3af32ad
--- /dev/null
+++ b/share/man/man1/portbump.1
@@ -0,0 +1,155 @@
+.\"     $OpenBSD: portbump.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2014 Vadim Zhukov
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PORTBUMP 1
+.Os
+.Sh NAME
+.Nm portbump
+.Nd tweaks port revisions and library versions
+.Sh SYNOPSIS
+.Nm
+.Op Fl lMmnrv
+.Op Fl W Ar wlib
+.Op Fl w Ar wlib
+.Op Ar portref ...
+.Pp
+.Nm
+.Fl o Ar file
+.Op Fl lMmnrv
+.Op Fl W Ar addlib
+.Op Fl w Ar rmlib
+.Op Ar portref ...
+.Sh DESCRIPTION
+.Nm
+is used to increase ('bump')
+.Ev REVISION
+and
+.Ev SHARED_LIBS
+values in
+.Ox
+ports.
+It also can remove
+.Ev REVISION
+marks and add, update or remove
+.Ev WANTLIB
+items.
+.Pp
+.Ar portref
+has the same syntax as
+.Ev FULLPKGPATH
+port variable, see
+.Xr bsd.port.mk 5 .
+Actually, you can feed a list of pkgpaths as parameters; flavor
+information will be ignored.
+.Nm
+doesn't care about actual subdirectory portion of
+.Ar portref ,
+so you can freely pass
+.Pa \&. ,
+.Pa ../foo
+or anything else like this.
+.Pp
+If subpackage is not specified in port reference, then all subpackages
+of a port will be processed.
+If no port references are given, the port in current directory will
+be processed.
+.Pp
+Default mode of operation is increasing
+.Ev REVISION
+mark values (
+.Fl r ) .
+Operations on
+.Ev REVISION
+marks and on
+.Ev SHARED_LIBS
+could be combined by explicit specifying corresponding flags, see below.
+.Pp
+The following options are available:
+.Bl -tag -width Ds
+.It Fl d
+Delete all
+.Ev REVISION
+marks in
+.Pa Makefile .
+Mutually exclusive with
+.Fl r .
+.It Fl l
+Recreate all
+.Ev WANTLIB
+items based on output of
+.Sq make port-lib-depends-check .
+If used twice or more, the
+.Sq make lib-depends-check
+will be used instead.
+The
+.Fl l
+mode is incompatible with
+.Fl W
+and
+.Fl w .
+.It Fl M
+Increment by one major component of all
+.Ev SHARED_LIBS ,
+resetting minor one to zero if needed.
+Mutually exclusive with
+.Fl m .
+.It Fl m
+Increment by one minor component of all
+.Ev SHARED_LIBS .
+Mutually exclusive with
+.Fl M .
+.It Fl n
+Do not replace
+.Pa Makefile
+but save modified version in the
+.Pa Makefile.bump
+instead.
+.It Fl o Ar file
+Send modified Makefile contents to a given file instead of creating
+.Pa Makefile.bump
+in port's directory.
+This could be only used if not more than one port is specified;
+multiple subpackages of a single port could be specified, though.
+.It Fl r
+Increment
+.Ev REVISION
+values for all subpackages, or only for given ones.
+Mutually exclusive with
+.Fl d .
+.It Fl W Ar wlib
+.It Fl w Ar wlib
+Adds or updates (the
+.Fl W
+flag) or removes (the
+.Fl w
+flag)
+.Ev WANTLIB
+items in whole port, or in particular subpackages, if given.
+Both options could be specified multiple times, accumulating
+corresponding lists.
+The
+.Ev WANTLIB
+entries will be fully re-generated based on previous contents.
+Those modes are incompatible with
+.Fl l .
+.It Fl v
+Enable printing of diagnostic messages to standard error output.
+.El
+.Sh HISTORY
+.Nm
+first appeared in
+.Ox 5.6 .
diff --git a/share/man/man1/portcheck.1 b/share/man/man1/portcheck.1
new file mode 100644
index 00000000000..0e9749cde06
--- /dev/null
+++ b/share/man/man1/portcheck.1
@@ -0,0 +1,152 @@
+.\"     $OpenBSD: portcheck.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2013 Vadim Zhukov
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PORTCHECK 1
+.Os
+.Sh NAME
+.Nm portcheck
+.Nd validate a port before submitting
+.Sh SYNOPSIS
+.Nm
+.Op Fl dNP
+.Op Fl p Ar portsdir
+.Op Fl x Ar pattern
+.Pp
+.Nm
+.Fl A
+.Op Fl dP
+.Op Fl p Ar portsdir
+.Op Fl x Ar pattern
+.Op Ar subdir ...
+.Sh DESCRIPTION
+.Nm
+is used to validate the
+.Ox
+port or port hierarchy in current directory.
+It should be used before submitting ports for review to avoid making
+common mistakes.
+.Nm
+verifies that directory and file structure for a port is in place and
+that no bogus files exist.
+.Pp
+When it's done,
+.Nm
+will print detected value of port's
+.Ev PKGPATH
+to standard output, unless it fails in detection.
+In the latter case, the
+.Fl p
+option should be provided.
+All other (error) messages from
+.Nm
+end up on standard error output.
+.Pp
+By default,
+.Nm
+automatically picks up nearest parent directory named
+.Dq ports ,
+with an optional
+.Dq mystuff
+or
+.Dq openbsd-wip
+subdirectory component, as the ports root directory.
+For example: if the port being imported is located in
+.Pa /home/joe/cvs/ports/openbsd-wip/devel/p5-Foo ,
+then the root ports directory will be detected as being
+.Pa /home/joe/cvs/ports/openbsd-wip .
+To override this behaviour, see the
+.Fl p
+option.
+.Pp
+The following options are available:
+.Bl -tag -width Ds
+.It Fl A
+Intended for running
+.Nm
+on the whole ports tree, i.e., the one lying in
+.Ev PORTSDIR .
+This option adds several ignore patterns (see
+.Fl x
+option description) and disables some other checks (e.g., for missing
+distinfo).
+.Ev PKGPATH
+determining and printing won't be done.
+Implicit change of working directory to the ports tree root is done
+before starting any checks.
+Also, in this mode one or more
+.Ar subdir
+arguments could be specified, to narrow the check only for given
+subdirectories of ports tree root.
+.It Fl d
+Show debugging information such as calling of check routines.
+.It Fl P
+Disable expensive checks that use
+.Dq print-plist-with-depends
+target, e.g., proper usage of
+.Xr gtk-update-icon-cache 1 ,
+.Xr update-desktop-database 1
+and
+.Xr update-mime-database 1 .
+.It Fl p Ar portsdir
+Forces the given directory to be treated as ports root directory.
+Cancels autodetection of the root ports directory made by default.
+This option is useful, e.g., when you have a temporary ports tree in
+a non-standard location.
+.It Fl N
+Intended to be used when working on new ports.
+Enables the checks like the presence of REVISION markers and non-0.0
+.Ev SHARED_LIBS .
+It also enables checks for the presence of CVS directories that could
+be left by mistake when creating a new port based on another one.
+.It Fl x
+Excludes files and subdirectories matching given shell globbing pattern
+from any checks.
+Note that matching is done against relative path, and not against
+absoulte path or base name either.
+I.e., to exclude the
+.Dq x11/kde4/libs/logs
+from checks, you must pass the whole line as argument, not just
+.Dq logs .
+Multiple -x options may be specified.
+.El
+.Sh EXAMPLES
+To validate a new port you've just prepared, go to port's directory and
+run:
+.Bd -literal -offset indent
+$ portcheck -N
+.Ed
+.Pp
+If you were working on updating of an existing port in CVS tree:
+.Bd -literal -offset indent
+$ portcheck
+.Ed
+.Pp
+To run a global check of the whole
+.Dq devel
+category in ports tree, use the
+.Fl A
+option instead:
+.Bd -literal -offset indent
+$ portcheck -Ap /usr/ports devel
+.Ed
+.Sh SEE ALSO
+.Xr portimport 1
+.Sh HISTORY
+This utility was split from
+.Xr portimport 1
+in 2013 and first appeared in
+.Ox 5.5 .
diff --git a/share/man/man1/portgen.1 b/share/man/man1/portgen.1
new file mode 100644
index 00000000000..79df5fd3011
--- /dev/null
+++ b/share/man/man1/portgen.1
@@ -0,0 +1,91 @@
+.\" $OpenBSD: portgen.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2015 Giannis Tsaraias <tsg@openbsd.org>
+.\" Copyright (c) 2015 Vadim Zhukov <zhuk@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PORTGEN 1
+.Os
+.Sh NAME
+.Nm portgen
+.Nd port modules on the fly
+.Sh SYNOPSIS
+.Nm portgen
+.Ar type
+.Ar module-name
+.Sh DESCRIPTION
+The
+.Nm
+utility creates ports for the given
+.Ar module-name
+from
+.Ar type
+module framework.
+The module is downloaded, analyzed and translated into a port.
+Any missing dependencies are handled recursively.
+New ports will land under ${PORTSDIR}/mystuff.
+The
+.Ev PORTSDIR
+value will be taken from current user environment, and defaults to
+.Pa /usr/ports .
+.Pp
+The
+.Nm
+utility currently supports the following
+.Ar type
+values:
+.Pp
+.Bl -inset -offset indent -compact
+.It Cm p5
+for Perl modules on CPAN.
+.It Cm ruby
+for Ruby gems.
+.El
+.Pp
+The following packages must be installed prior to the invocation of
+.Nm :
+.Pp
+.Bl -item -offset indent -compact
+.It
+.Cm sqlports-compact
+or
+.Cm sqlports
+.It
+.Cm p5-DBI
+.It
+.Cm p5-DBD-SQLite
+.El
+.Sh EXIT STATUS
+.Ex -std
+.Sh EXAMPLES
+The following command will download the latest version of Config::AutoConf
+from CPAN, parse it and create a port for it:
+.Bd -ragged -offset indent
+.Nm
+p5 Config::AutoConf
+.Ed
+.Sh SEE ALSO
+.Xr port-modules 5 ,
+.Xr ports 7
+.Sh HISTORY
+The
+.Nm
+utility first appeared in
+.Ox 5.9
+as a result of Google Summer of Code 2015.
+.Sh AUTHORS
+.An Giannis Tsaraias Aq Mt tsg@openbsd.org
+.Sh CAVEATS
+Not every module could be converted to a port.
diff --git a/share/man/man1/portimport.1 b/share/man/man1/portimport.1
new file mode 100644
index 00000000000..24cef5e1ef6
--- /dev/null
+++ b/share/man/man1/portimport.1
@@ -0,0 +1,79 @@
+.\"     $OpenBSD: portimport.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2013 Robert Peichaer
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PORTIMPORT 1
+.Os
+.Sh NAME
+.Nm portimport
+.Nd import a new port to the ports cvs repository
+.Sh SYNOPSIS
+.Nm
+.Op Fl p Ar portsdir
+.Op Fl u Ar username
+.Sh DESCRIPTION
+.Nm
+is used to import the directories and files of a new port to the
+.Ox
+ports
+.Xr cvs 1
+repository, avoiding common mistakes.
+It has to be executed from within the new port's directory.
+.Pp
+.Nm
+calls the
+.Xr portcheck 1
+internally to verify that directory and file structure for a port is in
+place and that no bogus files exist, and also to detect the
+.Ev PKGPATH
+for a port.
+.Pp
+The import is done in two steps.
+In the first step, the
+.Fl n
+option of the cvs command is used to only check for possible problems
+without changing anything and shows the cvs path where the new port will
+be imported.
+In the second step, the current ports directory is imported to the
+.Ox
+ports cvs repository.
+After the import, the new port is checked out in the respective directory
+of the local ports tree.
+.Pp
+The following options are available:
+.Bl -tag -width Ds
+.It Fl p Ar portsdir
+Forces the given directory to be treated as ports root directory.
+Passed through to
+.Xr portcheck 1 .
+.It Fl u Ar username
+Set the username used for
+.Xr ssh 1 ,
+and to compose vendor and release tags.
+Defaults to the local username.
+.El
+.Sh SEE ALSO
+.Xr cvs 1 ,
+.Xr portcheck 1
+.Sh HISTORY
+This command is based on the portimport script of Marc Espie, lightly
+modified by Stuart Henderson and rewritten by Robert Peichaer.
+Later, actual checks were moved to a separate utility,
+.Xr portcheck 1 .
+.Sh CAVEATS
+The
+.Ev CVSROOT
+environment variable is not used.
diff --git a/share/man/man1/portslogger.1 b/share/man/man1/portslogger.1
new file mode 100644
index 00000000000..1a48b41f1fb
--- /dev/null
+++ b/share/man/man1/portslogger.1
@@ -0,0 +1,61 @@
+.\"	$OpenBSD: portslogger.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PORTSLOGGER 1
+.Os
+.Sh NAME
+.Nm portslogger
+.Nd logs ports builds selectively
+.Sh SYNOPSIS
+.Nm portslogger
+.Op Fl s
+.Ar directory
+.Sh DESCRIPTION
+.Nm
+saves logs from ports build in separate files.
+.Pp
+.Nm
+handles directory change messages from the ports tree infrastructure and
+switches to a corresponding log file situated in
+.Ar directory
+accordingly, e.g., it will log the build from screen in
+.Pa directory/screen-4.0.3p1.log .
+.Nm
+will filter progress-bar noise from
+.Xr ftp 1 ,
+.Xr pkg_add 1 ,
+and
+.Xr pkg_create 1 .
+.Pp
+By default,
+.Nm
+acts as a filter and still passes its input to stdout, unless
+run with
+.Fl s .
+.Pp
+It is most often used as
+.Pp
+.Dl $ make package 2>&1 | portslogger logdir
+.Pp
+or in conjunction with
+.Xr tmux 1 ,
+such as
+.Pp
+.Dl $ tmux pipe-pane 'portslogger -s logdir'
+.Sh SEE ALSO
+.Xr dpb 1 ,
+.Xr tmux 1
diff --git a/share/man/man1/proot.1 b/share/man/man1/proot.1
new file mode 100644
index 00000000000..d84bd8603f5
--- /dev/null
+++ b/share/man/man1/proot.1
@@ -0,0 +1,287 @@
+.\"	$OpenBSD: proot.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2016 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt PROOT 1
+.Os
+.Sh NAME
+.Nm proot
+.Nd ports chroot builder
+.Sh SYNOPSIS
+.Nm proot
+.Op Fl B Ar chroot
+.Op Fl c Ar configfile
+.Op Fl S Ar srcroot
+.Op Ar attribute Ns = Ns Ar value ...
+.Sh DESCRIPTION
+.Nm
+can fill up a chroot directory for ports building usage.
+It will perform a set of
+.Cm actions
+that should fill up a destination
+.Ar chroot
+directory from the base system (or an optional
+.Ar srcroot ) .
+.Pp
+As far as possible,
+.Nm
+will create hardlinks instead of copying files, so that cloning an existing
+chroot will often only consume i-nodes.
+.Pp
+Some attributes can take multiple values.
+As a shorthand, several values can be specified in a row without having
+to repeat the
+.Ar attribute
+name, for instance:
+.Pp
+.Dl proot -B /build actions=unpopulate_light check_symlinks
+.Pp
+A
+.Ar configfile
+mostly contains one
+.Ar attribute Ns = Ns Ar value
+assignment per line, with comments starting with a
+.Sq #
+character.
+.Pp
+Possible
+.Cm actions
+(in the order of execution) are as follows:
+.Pp
+.Bl -tag -width unpopulate_light -compact -offset indent
+.It Cm check_mount
+Verify the state of mount points in the system, specifically whether
+the chroot area is not nodev.
+.It Cm unpopulate
+Remove everything from the chroot apart from selected data.
+See
+.Sx Preserved areas and unpopulate
+for details.
+.It Cm snapshot
+Copy things from a snapshot,
+to be retrieved from an url or from the local filesystem.
+.It Cm locate
+Copy things from the base system, perusing the system
+.Xr locate 1
+databases.
+.Bf Sy
+Note that this might erase data, in case it doesn't match the base system
+and is in the way.
+.Ef
+.It Cm resolv
+Copy the system
+.Xr resolv.conf 5
+and
+.Xr hosts 5
+files.
+.It Cm copy_ports
+Copy the ports tree.
+.It Cm copy_sys
+Copy system include files (deprecated).
+.It Cm unpopulate_light
+Remove everything from the chroot apart from selected data.
+See
+.Sx Preserved areas and unpopulate
+for details.
+.It Cm ldconfig
+Rerun
+.Xr ldconfig 8 .
+.It Cm devs
+Regenerate devices using
+.Xr MAKEDEV 8
+and
+.Xr dev_mkdb 8 .
+.It Cm checkout_ports
+Check out a ports tree from cvs using a provided
+.Cm portscvs
+location.
+.It Cm ports_subdirs
+Create ports infrastructure subdirs, according to users required for
+.Xr dpb 1 .
+If
+.Cm chown_all
+is set to 1,
+.Xr chown 2
+any content within to the appropriate users.
+.It Cm stragglers
+Double-check filled up chroot for files we don't know about.
+.It Cm write_mk
+If some directory values are different from the default,
+write a skeleton
+.Xr mk.conf 5
+file.
+.It Cm check_symlinks
+Report absolute symlinks that exist under the
+.Ar chroot ,
+white-listing known base system links.
+.El
+.Pp
+By default,
+.Nm
+will run
+.Cm check_mount , unpopulate_light , devs , ldconfig , ports_subdirs ,
+.Cm resolv , write_mk .
+If
+.Cm unpopulate
+is used,
+.Cm unpopulate_light
+won't be run.
+It will also run
+.Cm snapshot
+if a snapshot location is provided,
+or
+.Cm locate
+otherwise.
+.Pp
+Add actions with
+.Cm actions Ns = Ns Ar value ,
+remove them with
+.Cm actions Ns =- Ns Ar value .
+.Pp
+Other attributes are:
+.Pp
+.Bl -tag -width mkconf_lines -offset indent -compact
+.It Cm chroot
+same as
+.Fl B Ar chroot
+.It Cm srcroot
+same as
+.Fl S Ar srcroot
+.It Cm PORT_USER
+Who the ports tree should belong to
+.It Cm BUILD_USER
+Who to build as (defaults to _pbuild)
+.It Cm FETCH_USER
+Who to fetch as (defaults to _pfetch)
+.It Cm LOG_USER
+Who to write log as (defaults to
+.Cm BUILD_USER )
+.It Cm PORTSDIR
+.It Cm DISTDIR
+.It Cm PACKAGE_REPOSITORY
+.It Cm PLIST_REPOSITORY
+.It Cm LOCKDIR
+.It Cm WRKOBJDIR
+same as in
+.Xr bsd.port.mk 5
+.Pp
+.It Cm chown_all
+set to 1 to have the action
+.Cm ports_subdirs
+change owners of every content.
+.It Cm extra
+extra file to copy into the
+.Ar chroot .
+.It Cm mkconf_lines
+Add lines to the generated
+.Pa /etc/mk.conf .
+.It Cm mkconf_tail
+Add file to the generated
+.Pa /etc/mk.conf .
+.It Cm preserve
+Supplemental directory to preserve during
+.Cm unpopulate .
+.It Cm sets
+Add or remove sets compared to the default list required to build ports
+(defaults to base comp etc xbase xfont xshare xetc).
+.It Cm snapshot
+Location of snapshot for the
+.Cm snapshot
+action.
+.It Cm portscvs
+Where to get a ports snapshot for the
+.Cm checkout_ports
+action.
+.El
+.Ss Preserved areas and unpopulate
+The following entries won't be erased during a
+.Cm locate
+or
+.Cm unpopulate*
+action:
+.Pp
+.Bl -dash -compact -offset indent
+.It
+Items explicitly marked with the
+.Cm preserve
+attribute.
+.It
+A
+.Cm snapshot
+directory containing the base sets.
+.It
+All the ports specific sub directories if they are defined,
+namely
+.Cm DISTDIR , WRKOBJDIR, LOGDIR, PACKAGE_REPOSITORY , PLIST_REPOSITY , LOCKDIR .
+.It
+The
+.Cm PORTSDIR
+itself, unless
+.Cm copy_ports
+is used.
+.It
+Any mountpoint.
+.It
+Data explicitly copied through other actions, e.g.,
+.Cm extra
+files, key files...
+.El
+.Sh EXAMPLES
+The following
+.Ar configfile
+sets things up for an initial build on a
+.Xr dpb 1
+cluster.
+.Bd -literal -offset index
+chroot=/build
+PORT_USER=espie
+extra=/home/espie/startup
+WRKOBJDIR=/tmp/pobj
+LOCKDIR=/tmp/locks
+PLIST_REPOSITORY=/data/plist
+DISTDIR=/data/distfiles
+PACKAGE_REPOSITORY=/data/packages
+actions=unpopulate_light
+	copy_ports
+.Ed
+.Pp
+It assumes
+.Pa /build/data
+is a separate partition that won't be cleaned up by
+.Cm unpopulate_light ,
+and that can be mounted on slaves.
+Note also that
+.Pa /tmp
+should be a fast local directory on every machine of the cluster.
+.Pp
+.Pa /home/espie/startup
+is a startup script for
+.Xr dpb 1 .
+.Pp
+Initial invocation on the master could be
+.Pp
+.D1 Nm Fl c Ar configfile Cm chown_all Ns =1
+.Pp
+to ensure correct permissions for existing data under various
+ports directories.
+.Pp
+Slaves should not need the extra arguments, as the directories will
+already have correct owners.
+.Sh SEE ALSO
+.Xr dpb 1 ,
+.Xr chroot 8
+.Sh AUTHOR
+Marc Espie
diff --git a/share/man/man1/register-plist.1 b/share/man/man1/register-plist.1
new file mode 100644
index 00000000000..81cbd4b552f
--- /dev/null
+++ b/share/man/man1/register-plist.1
@@ -0,0 +1,75 @@
+.\"	$OpenBSD: register-plist.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt REGISTER-PLIST 1
+.Os
+.Sh NAME
+.Nm register-plist
+.Nd smart diff and registration of packing-lists
+.Sh SYNOPSIS
+.Nm register-plist
+.Ar dir
+.Op Ar pkgname ...
+.Nm
+.Fl t
+.Ar p1
+.Ar p2
+.Sh DESCRIPTION
+.Nm
+is used to check that a packing-list for a given package name doesn't change.
+By default, it is invoked at the end of
+.Li make package,
+see
+.Ev PACKAGE_REPOSITORY
+in
+.Xr bsd.port.mk 5 .
+Package names (package stem plus version) are used to uniquely identify
+packages.
+When something in the package changes, the package name should change,
+usually by increasing
+.Ev REVISION .
+.Pp
+.Nm
+records a packing-list for each
+.Ar pkgname
+into the given directory
+.Ar dir .
+If no previous packing-list for the given pkgname exists, the packing-list
+is recorded.
+If a previous packing-list exists,
+.Nm
+compares both packing-list for discrepancies.
+Whenever something significant changes,
+.Nm
+will display a diff of the differences, and exit with an error.
+.Pp
+Some items in the packing-list may change without actual problems.
+For instance, dependencies are allowed to go forward.
+On the other hand, some items like the package DESCR can't change at all.
+Since this is a generated file, porters often overlook that items such as
+the HOMEPAGE or the MAINTAINER's email address are embedded into that file,
+and thus that changing these requires a REVISION bump.
+.Pp
+.Nm
+can also be invoked with
+.Fl t
+and two packing-lists
+.Ar p1
+.Ar p2
+to directly compare those packing-lists.
+.Sh SEE ALSO
+.Xr bsd.port.mk 5
diff --git a/share/man/man1/resolve-lib.1 b/share/man/man1/resolve-lib.1
new file mode 100644
index 00000000000..8e9451551e0
--- /dev/null
+++ b/share/man/man1/resolve-lib.1
@@ -0,0 +1,68 @@
+.\"	$OpenBSD: resolve-lib.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt RESOLVE-LIB 1
+.Os
+.Sh NAME
+.Nm resolve-lib
+.Nd resolve library specs during ports build
+.Sh SYNOPSIS
+.Nm resolve-lib
+.Op Fl needed
+.Op Fl silent
+.Ar spec ...
+.Sh DESCRIPTION
+.Nm
+resolve library specs against a list of installed libraries.
+.Nm
+takes a list of file names with full path on stdin,
+and matches each library
+.Ar spec
+against those libraries,
+using
+.Xr library-specs 7
+conventions.
+.Pp
+In particular,
+.Ev X11BASE
+and
+.Ev LOCALBASE
+are used to locate X11 libraries and port libraries.
+.Pp
+.Nm
+will print the best libraries found in a format suitable for
+.Cm @wantlib
+.Po
+see
+.Xr pkg_create 1
+.Pc .
+.Pp
+Each
+.Ar spec
+not found will result in an error message and exit code.
+.Pp
+If the option
+.Fl silent
+is used, nothing will be displayed.
+.Pp
+The option
+.Fl needed
+is used to sort out
+.Ev LIB_DEPENDS :
+.Nm
+will succeed if at least one of the libraries matches against a shared
+library passed on the command line.
diff --git a/share/man/man1/retrieve-index.1 b/share/man/man1/retrieve-index.1
new file mode 100644
index 00000000000..ff7cd0eaa1f
--- /dev/null
+++ b/share/man/man1/retrieve-index.1
@@ -0,0 +1,45 @@
+.\"	$OpenBSD: retrieve-index.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt RETRIEVE-INDEX 1
+.Os
+.Sh NAME
+.Nm retrieve-index
+.Nd find index entry in ports index according to selected criteria
+.Sh SYNOPSIS
+.Nm retrieve-index
+.Cm index Ns = Ns Ar file
+.Op Cm category Ns = Ns Ar cat
+.Op Cm maintainer Ns = Ns Ar person
+.Op Cm key Ns = Ns Ar value
+.Sh DESCRIPTION
+.Nm
+scans index file
+.Ar file
+for index entry (or entries) according to selected criteria,
+such as category, maintainer name or actual name.
+.Pp
+Matching is done using perl regular expressions, and corresponding entries
+get printed as a list on a single line on standard output.
+.Nm
+is a helper script used for searches from the ports infrastructure.
+It relies on
+.Pa ${PORTSDIR}/INDEX
+being accurate.
+.Sh SEE ALSO
+.Xr perlre 1 ,
+.Xr ports 7
diff --git a/share/man/man1/update-patches.1 b/share/man/man1/update-patches.1
new file mode 100644
index 00000000000..6ba4317b057
--- /dev/null
+++ b/share/man/man1/update-patches.1
@@ -0,0 +1,100 @@
+.\"	$OpenBSD: update-patches.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2010 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt UPDATE-PATCHES 1
+.Os
+.Sh NAME
+.Nm update-patches
+.Nd create or update patches for a given port.
+.Sh SYNOPSIS
+.Nm update-patches
+.Sh DESCRIPTION
+.Nm
+is a helper script for the target
+.Ar update-patches
+in
+.Xr bsd.port.mk 5 .
+It takes no argument, but uses quite a few environment variables as
+implicit parameters.
+.Pp
+.Nm
+looks under
+.Ev WRKDIST
+for patched files.
+Those are files with extension
+.Ev PATCHORIG ,
+except for those matching also
+.Ev DISTORIG .
+.Pp
+Once those files are found, they are matched against the contents of
+directory
+.Ev PATCHDIR ,
+most specifically files matching wildcard
+.Ev PATCH_LIST .
+Matching files are regenerated on the basis of
+.Pp
+.Dl $ diff -u -p -a ${DIFF_ARGS} $file$PATCHORIG $file
+.Pp
+and compared.
+Modified patches are updated and reported, new patches are generated,
+and old patches that no longer seem to exist are also reported.
+.Pp
+As an exception, if a
+.Pa $file.beforesubst
+original file is found, it is used instead of $file in the diff line:
+.Pp
+.Dl $ diff -u -p -a ${DIFF_ARGS} $file$PATCHORIG $file.beforesubst
+.Pp
+The patch header is normalized to always be:
+.Bd -literal -offset indent
+Index: $file
+--- $file.orig
++++ $file
+.Ed
+.Pp
+Patches with problematic
+.Xr cvs 1
+keywords may be fixed if possible, and reported if not.
+.Pp
+.Nm
+prints the list of changed patches for further study.
+.Sh ENVIRONMENT
+.Bl -tag -width FORCE_REGEN
+.It Ev DIFF_ARGS
+Extra parameters to insert into
+.Xr diff 1 .
+.It Ev DISTORIG
+Extension to original files to ignore entirely.
+.It Ev FORCE_REGEN
+Force patches to be regenerated even if they didn't change.
+.It Ev PATCHDIR
+Path to existing and generated patches.
+.It Ev PATCHORIG
+Extension to match for finding original files.
+.It Ev PATCH_LIST
+Only compare new patches to existing patches matching that pattern.
+.It Ev PATCH_VERBOSE
+Force
+.Nm
+to tell about everything it does.
+.It Ev WRKDIST
+Directory to scan for files to compare.
+.El
+.Sh SEE ALSO
+.Xr cvs 1 ,
+.Xr diff 1 ,
+.Xr patch 1
diff --git a/share/man/man1/update-plist.1 b/share/man/man1/update-plist.1
new file mode 100644
index 00000000000..e9a57dcd7af
--- /dev/null
+++ b/share/man/man1/update-plist.1
@@ -0,0 +1,251 @@
+.\"	$OpenBSD: update-plist.1,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\" Copyright (c) 2018 Marc Espie <espie@openbsd.org>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 26 2018 $
+.Dt UPDATE-PLIST 1
+.Os
+.Sh NAME
+.Nm update-plist
+.Nd create or update packing-list(s) for a given port.
+.Sh SYNOPSIS
+.Ev PORTSDIR Ns = Ns Ar path
+.Op Ev FAKE_TREE_OWNER Ns = Ns Ar user Ev PORTS_TREE_OWNER Ns = Ns Ar user Nm doas
+.Nm
+.Op Fl Fmnqrvx
+.Op Fl C Ar dir
+.Op Fl E Ar ext
+.Op Fl e Ar ext
+.Op Fl i Ar var
+.Op Fl j Ar jobs
+.Op Fl S Ar var
+.Op Fl s Ar var
+.Op Fl X Ar path
+.Op Fl w Ar suffix
+.Fl -
+.Ar pkg_create_args ...
+.Sh DESCRIPTION
+.Nm
+is a helper script for the target
+.Ar update-plist
+in
+.Xr bsd.port.mk 5 .
+.Pp
+Along with options and environment variables,
+.Nm
+uses the exact same options and arguments that would be passed to
+.Xr pkg_create 1 .
+Since
+.Xr pkg_create 1
+normally takes one single actual
+.Ar pkgname ,
+there is no ambiguity in
+.Ev MULTI_PACKAGES
+situations.
+.Pp
+The first set of arguments correspond to the
+.Sq default package .
+.Pp
+If
+.Ev PORTS_PRIVSEP
+is in effect,
+.Nm
+is run as root, but will switch back to
+.Ev FAKE_TREE_OWNER
+for scanning the installation directory and to
+.Ev PORTS_TREE_OWNER
+for writing packing-lists.
+.Pp
+.Nm
+may need to know the
+.Ev PORTSDIR
+location in order to scan dependencies to trim common directories.
+.Pp
+The options are as follows:
+.Bl -tag -width Ds
+.It Fl C Ar dir
+Use a cache directory
+.Ar dir
+for scanning dependencies.
+Beware that this directory should then be cleaned manually if the
+dependencies's packing-lists change.
+But this will speed up packing-list regeneration for ports with
+lots of dependencies significantly.
+.It Fl E Ar ext
+Write new files with
+.Ar ext
+extension instead of the default
+.Pa .new .
+.It Fl e Ar ext
+Preserve old files as
+.Ar ext
+extension instead of the default
+.Pa .orig .
+.It Fl F
+Do not try to run
+.Xr pkg_locate 1 .
+.It Fl i Ar var
+Ignore variable
+.Ar var
+for new substitutions.
+Only existing substitutions will be considered for the new lists.
+.It Fl j Ar jobs
+Use
+.Ar jobs
+concurrent
+.Xr pkg_locate 1
+pipes.
+Default is
+.Sq hw.ncpu .
+.It Fl n
+Do not move the final packing-lists in position. Instead, leave the
+.Pa <file>.new
+files for comparison.
+.It Fl q
+Quiet mode.
+Do not display status progress messages.
+Note that this is not the opposite of verbose mode.
+.It Fl r
+Run the
+.Pa fake
+directory scan as root.
+.It Fl S Ar var
+Consider variable for substitution only at the end of paths,
+e.g., for suffixes like
+.Pa .pyc
+or
+.Pa .pyo .
+.It Fl s Ar var
+Consider variable for substitution only at start of paths,
+e.g., for directories like
+.Pa /etc/rc.d
+or
+.Pa /usr/local .
+.It Fl v
+Verbose mode.
+Explain about directories stripped from dependencies.
+Note that this is not the opposite of quiet mode.
+.It Fl w Ar suffix
+Warn about suffixes such as
+.Ar .orig
+that (usually) should not end into packing-lists.
+.It Fl X Ar path
+Exclude file at absolute
+.Ar path
+from the information recorded in the packing-list.
+.El
+.Pp
+.Nm
+assumes all multi-packages live under the same installation directory
+.Po
+.Xr pkg_create 1 Ns 's
+.Fl B
+option
+.Pc ,
+and it will scan all paths under that directory and dispatch
+them into corresponding subpackages, using the following heuristics:
+.Bl -bullet
+.It
+files that were in existing packing-list(s) will be kept in the same
+packing-list.
+.It
+new files and directories will be put in the packing-list that owns
+the corresponding directory.
+.It
+baring that, new files will be put into the first packing-list whose PREFIX
+fits.
+.El
+.Pp
+As far as possible, everything that looks like actual files will be sorted
+in alphabetical order, after variable substitutions.
+.Pp
+.Nm
+will warn for a lot of conditions, such as pre-formatted manpages, or files
+ending in pre-patch suffixes.
+Adding a
+.Cm @comment Ar intentional<reason>
+will silence the warning.
+For instance:
+.Bd -literal -offset indent
+@man man/ja_JP.EUC/cat1/kakasi.0
+@comment intentional: mandoc does't handle this locale
+.Ed
+.Pp
+.Nm
+will also scan dependencies, in order to strip common directories.
+.Pp
+If the
+.Ar pkglocatedb
+package is installed,
+.Nm
+will make use of it and report unregistered conflicts.
+This can be disabled with
+.Fl F
+as it is time-consuming.
+.Pp
+Most
+.Ev SUBST_VARS
+variables will be back-substituted into the regenerated packing-lists,
+starting from the longest variable values.
+.Nm
+tries to keep existing substitutions.
+It does not add new substitutions on
+.Ev BASE_PKGPATH
+and
+.Ev BUILD_PKGPATH ,
+it does only substitute
+.Ev FULLPKGNAME
+in
+.Pa share/doc/pkg-readmes
+and it currently does not add new substitutions for
+.Ev ARCH
+nor
+.Ev MACHINE_ARCH .
+.Pp
+Specific items such as shared libraries or binaries will gain annotations
+and special handling, for instance
+.Ev LIBlibname_VERSION .
+.Pp
+Existing packing-lists are scanned for non-file entries, such as
+.Cm newuser ,
+.Cm mode ,
+or
+.Cm exec
+markers.
+Those will be inserted into the updated packing-lists at the most likely
+position.
+.Pp
+.Nm
+will first write files as
+.Pa <file>.new ,
+then it will display which files are new and which files have changed.
+If old packing-lists already exist, it will move old packing lists into
+.Pa <file>.orig
+unless these already exist, then move the new files into position.
+.Sh SEE ALSO
+.Xr pkg_add 1 ,
+.Xr pkg_create 1 ,
+.Xr bsd.port.mk 5
+.Sh BUGS AND LIMITATIONS
+.Nm
+might be the most complicated piece of the ports infrastructure.
+There will always be cases that require manual intervention.
+.
+Since
+.Ar fake
+is now run as non-root,
+.Nm
+can't figure out users and groups for new files, so these should be
+considered carefully.
diff --git a/share/man/man8/Makefile b/share/man/man8/Makefile
index b50fa156128..116f098c2a8 100644
--- a/share/man/man8/Makefile
+++ b/share/man/man8/Makefile
@@ -1,8 +1,8 @@
-#	$OpenBSD: Makefile,v 1.97 2017/04/26 14:53:11 sthen Exp $
+#	$OpenBSD: Makefile,v 1.98 2018/06/26 05:38:49 espie Exp $
 #	$NetBSD: Makefile,v 1.13 1996/03/28 21:36:40 mark Exp $
 #	@(#)Makefile	8.1 (Berkeley) 6/5/93
 
-MAN=	afterboot.8 autoinstall.8 boot_config.8 \
+MAN=	afterboot.8 autoinstall.8 boot_config.8 bulk.8 \
 	crash.8 daily.8 \
 	diskless.8 genassym.sh.8 intro.8 netstart.8 rc.8 \
 	rc.conf.8 rc.d.8 rc.shutdown.8 rc.subr.8 release.8 \
diff --git a/share/man/man8/bulk.8 b/share/man/man8/bulk.8
new file mode 100644
index 00000000000..62716313241
--- /dev/null
+++ b/share/man/man8/bulk.8
@@ -0,0 +1,202 @@
+.\"	$OpenBSD: bulk.8,v 1.1 2018/06/26 05:38:49 espie Exp $
+.\"
+.\"	Copyright (c) 2016 Marc Espie <espie@openbsd.org>
+.\"
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.Dd $Mdocdate: June 26 2018 $
+.Dt BULK 8
+.Os
+.Sh NAME
+.Nm bulk
+.Nd building OpenBSD packages in bulk
+.Sh DESCRIPTION
+There are quite a few steps necessary to build packages on a cluster.
+They are:
+.Pp
+.Bl -enum -compact -offset indent
+.It
+Choose master machine setup and create partitions.
+.It
+Setup chrooted builds on the master.
+.It
+Add slaves and do a full bulk.
+.It
+Clean up and do subsequent bulks.
+.It
+Perform additional maintenance.
+.El
+.Ss 1. Choose master machine setup and create partitions
+Setup a master machine with enough room for a chroot, say
+.Pa /build .
+Assuming you are using a cluster of machines,
+this chroot should contain NFS exportable partitions for distfiles,
+plists, and packages (one single partition can be used for simplicity).
+A full setup requires on the order of 50GB for distfiles and 50GB for packages.
+.Pp
+It is possible to build packages without a chroot, but the space
+requirement difference is negligible (a full
+.Ox
+install is less than 1GB), and having everything chrooted means you may
+install useful tools to help with the process outside of the chroot
+.Po
+for instance
+.Xr rsync 1
+.Pc .
+.Pp
+Reserve one "scratch" partition under the chroot for WRKOBJDIR
+(for instance, mfs, async, or SSD).
+This partition should be roughly 10GB if you want to be able to
+build all ports.
+This can often double as
+.Pa /tmp
+under the chroot.
+.Pp
+Alternately, you can setup your whole chroot as a scratch partition,
+and reserve one more permanent space under it for distfiles,
+packages, and plists.
+.Pp
+Choose a strategy for the ports tree itself.
+There must be a copy under
+.Pa /build .
+You can either copy it from outside the chroot, have it in an NFS
+partition, or manually make sure all machines on the cluster have the
+same ports tree (cvs checkout, rsync ...).
+.Pp
+Note that logs are only produced on the master, and thus do not
+need to be nfs exportable, nor even inside the chroot.
+.Pp
+.Ox
+now comes with default users for package builds, namely _pbuild and _pfetch.
+.Pp
+The default
+.Xr login.conf 5
+is appropriate for most setups, but _pbuild's datasize-cur will need
+to be bumped for a few ports, like pypy.
+Likewise, maxproc-cur is too small for machines with more than 4-6 cpus.
+.Pp
+Note that _pbuild does not need network access, and is now blocked by default
+in
+.Xr pf 4 .
+.Ed
+.Pp
+Recent
+.Ox
+systems do not need any kind of
+.Xr doas 1
+setup for bulk ports builds, as
+.Xr dpb 1
+is run as root and drops permissions appropriately.
+.Pp
+However, you may still want to setup
+.Xr doas 1
+for root, if you want to manually fix ports, as
+.Ar PORTS_PRIVSEP
+relies on it.
+.Ss 2. Setup chrooted builds on the master
+Populate the initial chroot with
+.Xr proot 1 .
+Point DISTDIR, PACKAGE_REPOSITORY, PLIST_REPOSITORY, WRKOBJDIR
+to appropriate locations.
+.Pp
+Pay attention to nodev and wxallowed warnings.
+A chroot setup that can't have devices won't work at all.
+A setup without wxallowed in /usr/local and WRKOBJDIR won't
+build a lot of things.
+.Pp
+Check that this setup can build ports by running
+.Li dpb -B /build
+as root.
+Fix any issues related to file ownership at this point.
+See
+.Xr dpb 1
+for details.
+.Pp
+If your WRKOBJDIR is a temporary partition, make sure it
+belongs to _pbuild:_pbuild after a reboot.
+.Ss 3. Add slaves and do a full bulk
+Create identical slave machines with the same release material.
+Have them import the NFS partitions from the master, they
+don't need root access to the partitions.
+Set up
+.Xr ssh 1
+so that the master can connect to the slaves, using ssh protocol 2,
+as root, preferably without a password or passphrase (however,
+.Xr dpb 1
+uses a master connection, so a password would be required just once per host).
+.Pp
+Note that code on slave machines will only run as _pbuild
+(during builds) or root (during dependency installation).
+Slave machines only require highly restricted network access.
+They just need to act as nfs clients to the master and to be reachable
+through ssh from the master.
+.Pp
+Use a similar
+.Xr proot 1 config
+to populate each slave.
+.Pp
+You should now be able to build ports on the slaves.
+A simple config will just have
+.Bd -literal -offset indent
+DEFAULT chroot=/build
+localhost
+host1
+\&...
+.Ed
+.Pp
+Check that the full config can still build ports.
+.Pp
+You're now ready for a full bulk.
+Beware that even fast configs (3 amd64 with 8 cores each) may take over 24 hours
+to finish. It's generally appropriate to run
+.Xr dpb 1
+under
+.Xr tmux 1 .
+.Ss 4. Clean up and do subsequent bulks
+Before running the next bulk, you should set up rotating logs and move the
+PACKAGE_REPOSITORY away.
+Save the PLIST_REPOSITORY and DISTDIR though.
+PLIST_REPOSITORY will catch problems in packing-lists.
+.Pa ${PLIST_REPOSITORY}/${ARCH}/history
+is also used to store
+.Xr sha256 1
+history, necessary to reorder files inside packages to speed updates up.
+.Pp
+The DISTDIR contains history information as well as DISTDIR/build-stats
+to speed further runs up.
+.Pp
+How you wipe things out depends on your setup.
+If you run
+.Xr proot 1
+again each time, most things will get cleaned up automatically
+.Po
+.Pa /build/usr/local , /build/var/db/pkg ...
+.Pc .
+Note that known directories such as WRKOBJDIR do not get cleaned up
+automatically, so you may want to set up a STARTUP_SCRIPT in
+.Xr dpb 1 .
+.Ss 5. Perform additional maintenance
+.Xr clean-old-distfiles 1
+should be run occasionally since the DISTDIR will continue growing.
+.Pp
+.Xr check-problems 1
+should be run occasionally to find out conflicts and dependency issues.
+.Sh SEE ALSO
+.Xr check-problems 1 ,
+.Xr clean-old-distfiles 1 ,
+.Xr dpb 1 ,
+.Xr proot 1 ,
+.Xr register-plist 1 ,
+.Xr tmux 1 ,
+.Xr bsd.port.mk 5 ,
+.Xr release 8