Document the way distfiles are mirrored.

Document a few things that changed in the ports tree.

3 files changed, 239 insertions, 24 deletions

diff --git a/share/man/man7/Makefile b/share/man/man7/Makefile
index 329f6c31b05..0f3627cf58e 100644
--- a/share/man/man7/Makefile
+++ b/share/man/man7/Makefile
@@ -4,7 +4,8 @@
 # missing: eqnchar.7 man.7 ms.7 term.7
 
 MAN=	ascii.7 environ.7 hier.7 hostname.7 intro.7 mailaddr.7 \
-	mdoc.7 mdoc.samples.7 operator.7 ports.7 securelevel.7
+	mdoc.7 mdoc.samples.7 operator.7 securelevel.7 \
+	ports.7 mirroring-ports.7
 MLINKS=	mdoc.7 mandoc.7 mdoc.samples.7 mandoc.samples.7
 
 .include <bsd.prog.mk>
diff --git a/share/man/man7/mirroring-ports.7 b/share/man/man7/mirroring-ports.7
new file mode 100644
index 00000000000..19902a31898
--- /dev/null
+++ b/share/man/man7/mirroring-ports.7
@@ -0,0 +1,178 @@
+.\" $OpenBSD: mirroring-ports.7,v 1.1 2000/04/19 01:39:58 espie Exp $
+.\"
+.\" Copyright (c) 2000 Marc Espie
+.\"
+.\" All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR
+.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT,
+.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+.\"
+.Dd February 19, 2000
+.Dt MIRRORING-PORTS 7
+.Os
+.Sh NAME
+.Nm mirroring-ports
+.Nd how to build a mirror for ports distfiles
+.Sh DESCRIPTION
+The
+.Nm OpenBSD Ports Collection
+offers some powerful tools to mirror software sources.
+The ports infrastructure provides a
+.Ar mirror-maker
+target that can be used to build a Makefile to facilitate mirroring
+distfiles.
+This target builds
+.Ic ${DISTDIR}/Makefile ,
+which can be used on almost any Unix-like machine to mirror
+.Ox
+distfiles.
+.Pp
+A sample Makefile entry is formatted like this:
+.Bd -literal
+all:: audio/tracker/tracker-5.3
+.PHONY: audio/tracker/tracker-5.3
+audio/tracker/tracker-5.3::  tracker-5.3.tgz
+
+tracker-5.3.tgz: $F
+        @MAINTAINER="espie@openbsd.org" \\
+         SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ " \\
+         CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323" \\
+         ${FETCH} "$@"
+
+
+.Ed
+This Makefile is usually invoked by the user from the directory where
+they wish to mirror distfiles, with variables
+.Ev FETCH
+and
+.Ev F
+set on the command line, e.g.,
+.Bd -literal -offset indent
+	cd mirror && make -j 5 -f path_to_makefile FETCH=fetch_script
+.Ed
+
+Targets are set up so that each port is referenced by its full name, and
+retrieves all its distfiles.
+The default
+.Ar all
+target is used to retrieve all distfiles.
+Targets
+.Ar ftp
+and
+.Ar cdrom
+can be used to retrieve all distfiles necessary to build ftp and cdrom
+packages, respectively.
+Actual fetching is usually invoked with a parallel-make option, so that
+several retrievals through ftp can proceed simultaneously.
+.Pp
+The
+.Ev F
+variable can be set to a dummy name, or a recent filename, to force
+re-fetching of anything which is older than the filename.
+Its intended use is to force re-fetching existing files,
+or to checksum all files.
+.Pp
+The
+.Ev ${FETCH}
+script should be supplied by the user, and will download and verify the
+archive file.
+It must obey the following variables:
+.Bl -tag -width DIST_SUBDIR
+.It Ev MAINTAINER
+Port maintainer, used to report errors,
+.It Ev ERROR
+Some ports problems can be detected while building the Makefile, in which
+case this variable will be set to a proper error message.
+.It Ev DIST_SUBDIR
+See
+.Xr ports 7
+for more details.
+The
+.Ev ${FETCH}
+script is responsible for creating this subdirectory and cd'ing to it
+before performing the actual fetch.
+.It Ev SITES
+A list of sites to try for fetching the distribution file.
+.It Ev CIPHER
+The checksumming utility to use for verifying the distribution file.
+It will normally be set to
+.Xr sha1 1
+unless you tinker with
+.Ev PREFERRED_CIPHER
+while building the mirroring Makefile.
+.It Ev CKSUM
+The corresponding checksum.
+If neither
+.Ev CIPHER
+nor
+.Ev CKSUM
+nor
+.Ev ERROR
+are set, the distribution file needs not be checked.
+.El
+.Pp
+A standard fetch strategy is to try all sites in order: whenever the
+distribution file is found, download it; verify the checksum; erase the
+file and try the next site if it doesn't match.
+.Pp
+Mirroring sites should update their master Makefile fairly often.
+Activities a proper mirror should offer (in order of decreasing importance):
+.Bl -tag -width XXXXXXXXX
+.It Mirror new files
+Use a proper fetch script to download missing files,
+.It Verify all checksums
+All checksums should be verified from time to time, and maintainers
+notified of persistant discrepancies,
+.It Check mastersites liveliness
+Use a tool such as
+.Sq mirror
+to check that the master sites haven't fallen
+off the Earth.
+Even though the first site in the site list is the
+most important site, good mirrors will scan all sites and report all
+problems,
+.It Remove old files
+To gain room this, the mirror should maintain a list of
+.Sq active
+files (easy enough, just provide a fetch script that just lists the
+file names), and remove files that are no longer active.
+Since
+.Ox
+releases happen every six months, this delay should be longer than that.
+.El
+Sample scripts are provided in the
+.Pa ${PORTSDIR}/infrastructure/fetch
+directory.
+.Sh FILES
+.Bl -tag -width XXXXXXXXX -compact
+.It Pa ${DISTDIR}/Makefile
+Main mirroring Makefile
+.It Pa ${PORTSDIR}/infrastructure/fetch/fetch-all
+Sample script usable to retrieve distfiles.
+.It Pa ${PORTSDIR}/infrastructure/fetch/check-all
+Sample script to check all distfiles checksums.
+.El
+.Sh HISTORY
+This infrastructure was introduced for
+.Ox 2.7
+by Marc Espie, with feedback from Bob Beck, Todd Fries, Camiel Doobelbar,
+and a few other people.
+.Sh SEE ALSO
+.Xr ports 7
diff --git a/share/man/man7/ports.7 b/share/man/man7/ports.7
index 7174c3aea5f..38fdb1dbccd 100644
--- a/share/man/man7/ports.7
+++ b/share/man/man7/ports.7
@@ -23,7 +23,7 @@
 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
 .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 .\"
-.\" $OpenBSD: ports.7,v 1.9 2000/03/19 19:25:37 aaron Exp $
+.\" $OpenBSD: ports.7,v 1.10 2000/04/19 01:39:58 espie Exp $
 .\" $FreeBSD: ports.7,v 1.7 1998/06/23 04:38:50 hoek Exp $
 .\"
 .Dd January 25, 1998
@@ -67,8 +67,8 @@ Package System documentation
 .Pp
 .Sh PORTS MASTER MAKEFILE
 The ports master Makefile, normally located in
-.Pa /usr/ports/Makefile 
-(but see 
+.Pa /usr/ports/Makefile
+(but see
 .Ev PORTSDIR
 below)
 offers a few useful targets.
@@ -76,23 +76,59 @@ offers a few useful targets.
 .It Ar index
 rebuild the ports complete index,
 .Pa /usr/ports/INDEX
+.It Ar mirror-maker
+see
+.Xr mirroring-distfiles 7 ,
 .It Ar print-index
-display the contents of the index in a user-friendly way.
+display the contents of the index in a user-friendly way,
 .It Ar search
-invoked with a key, e.g., 
+invoked with a key, e.g.,
 .Ic make search key=foo ,
-retrieve information relevant to a given port.
+retrieve information relevant to a given port (obsolescent).
+.El
+.Sh SELECTING A SET OF PORTS
+If
+.Pa /usr/ports/INDEX
+is up to date, it is possible to select subsets by setting the following
+variables on the command line:
+.Bl -tag -width category
+.It Va key
+package name matching the given key,
+.It Va category
+port belonging to category,
+.It Va maintainer
+port maintained by a given person.
+.El
+
+For instance, to invoke clean on all ports in the x11 category, one can say:
+.Bd -literal -offset indent
+	make category=x11 clean
+.Ed
+
+The index search is done by a perl script, so all regular expressions from
+.Xr perlre 1
+apply.
 .Sh TARGETS
 Individual ports are controlled through a few documented targets.
 Some of these targets work recursively through subdirectories, so that
 someone can, for examples, install all of the net
 ports.
+.Pp
+In case of failure in a subdirectory, the shell fragment held in
+.Ev REPORT_PROBLEM
+is executed.
+Default behavior is to call exit, but this can be overriden on the command
+line, e.g., to avoid stopping after each problem.
+.Bd -literal -offset indent
+	make REPORT_PROBLEM=true
+.Ed
+
 The targets that do this are
-.Ar build , checksum , clean , configure , extract , fetch , fetch-list , 
+.Ar build , checksum , clean , configure , extract , fetch , fetch-list ,
 .Ar install , distclean , deinstall, reinstall, mirror-distfiles, obj ,
-.Ar list-distfiles 
+.Ar list-distfiles , package , cdrom-packages
 and
-.Ar package .
+.Ar ftp-packages .
 .Pp
 Target names starting with _ are private to the ports infrastructure,
 should not be invoked directly, and are liable to change without notice.
@@ -107,12 +143,12 @@ by
 .Ar install ,
 and so on all the way to
 .Ar fetch .
-Typical use only runs 
+Typical use only runs
 .Ar install
-explicitly (if root), or 
+explicitly (if root), or
 .Ar build
 (as user), then
-.Ar install 
+.Ar install
 (as root).
 .Bl -tag -width configure
 .It Ar fetch
@@ -144,7 +180,7 @@ targets, this is run in scattered pieces as
 .Ar fetch-depends , lib-depends , build-depends , run-depends , misc-depends .
 Defining
 .Ev NO_DEPENDS
-to 
+to
 .Dv Yes
 will skip this step.
 .It Ar extract
@@ -170,7 +206,7 @@ Install the port and register it with the package system.
 The following targets are not run during the normal install process.
 .Bl -tag -width fetch-list
 .It Ar fetch-list
-Build a script that lists the files needing to be fetched in order to 
+Build a script that lists the files needing to be fetched in order to
 build the port.
 .It Ar depends-list package-depends
 Print an ordered list of all the compile and run dependencies.
@@ -203,13 +239,13 @@ and
 .Ev PKGFILE .
 .El
 .Sh PORT VARIABLES
-These can be changed in the environment, or in 
+These can be changed in the environment, or in
 .Pa /etc/mk.conf
 for persistence.
 They can also be set on make's command line, e.g.,
 .Ic make VAR_FOO=foo
 .Pp
-Boolean variables should be set to 
+Boolean variables should be set to
 .Dv Yes
 instead of simply being defined, for uniformity and future compatibility.
 .Pp
@@ -275,25 +311,25 @@ Primary location(s) for distribution patch files if not found
 locally.
 .It Ev MASTER_SITE_OPENBSD
 If set to
-.Dv Yes , 
+.Dv Yes ,
 go to the master OpenBSD site for all files.
 .It Ev MASTER_SITE_FREEBSD
 If set to
-.Dv Yes , 
+.Dv Yes ,
 go to the master FreeBSD site for all files.
 .It Ev MASTER_SITE_OVERRIDE
 Try going to this site for all files and patches, first.
 .It Ev CLEANDEPENDS
-If set to 
+If set to
 .Dv Yes ,
-let 
-.Sq clean 
+let
+.Sq clean
 recurse to dependencies.
 .It Ev NOCLEANDEPENDS
 If defined, don't let
 .Sq clean
-recurse to dependencies (deprecated, use 
-.Ev CLEANDEPENDS 
+recurse to dependencies (deprecated, use
+.Ev CLEANDEPENDS
 instead).
 .It Ev FETCH_CMD
 Command to use to fetch files.