.\" $OpenBSD: bsd.regress.mk.5,v 1.14 2017/07/03 18:19:55 bluhm Exp $ .\" .\" Copyright (c) 2002 Anil Madhavapeddy .\" Copyright (c) 2000 Marc Espie .\" .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. .\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .\" .Dd $Mdocdate: July 3 2017 $ .Dt BSD.REGRESS.MK 5 .Os .Sh NAME .Nm bsd.regress.mk .Nd regression test master Makefile fragment .Sh SYNOPSIS .Fd .include .Sh DESCRIPTION .Nm holds the standard routines used by the source regression tests. Some variables and targets are for its internal use only. The rest are documented here. .Pp Since this file also includes .Nm bsd.prog.mk , all of the usual .Ox Makefile variables may be used to build the regression test programs. .Sh TARGETS .Bl -tag -width regress .It Cm depend Build any dependencies required to carry out the current set of regression tests. .It Cm regress Executes all of the regression targets defined in the Makefile. If one of the tests fails, the line .Qq FAILED is printed to the standard output. By default, execution continues with the next test and, after running all the tests, .Sy make Cm regress exits with an exit status of 0 even if some of the tests failed. .It Cm run-regress-* Runs an individual regression test. If the exit status of the program indicates an error or timeout, then a failure is logged, otherwise the test is marked as a success. .El .Sh VARIABLES .Bl -tag -width Ds .It Ev REGRESS_FAIL_EARLY If this variable is set to anything but .Dq no , the .Cm regress target will abort as soon as a test fails. .It Ev REGRESS_LOG Points to the fully-qualified path of a file to which regression results are appended. Defaults to .Pa /dev/null . .It Ev REGRESS_MAXTIME Maximum limit of CPU seconds to spend on the regression test. Exceeding this time will result in a failure being logged. .It Ev REGRESS_ROOT_TARGETS Targets for which root access is required to run the test. The .Ev SUDO variable is invoked for these targets. See also .Ev SUDO . .It Ev REGRESS_SKIP_SLOW If this variable is not empty, skip over all the regression tests which have been marked as being 'slow' using the .Ev REGRESS_SLOW_TARGETS variable. .It Ev REGRESS_SLOW_TARGETS Targets which are defined as 'slow'. All of these tests can be skipped by setting the .Ev REGRESS_SKIP_SLOW variable. .It Ev REGRESS_TARGETS Targets which are invoked to run the set of regression tests for this Makefile. Defaults to .Cm run-regress-${PROG} . .It Ev SUDO Location of a command used to switch to root for certain test targets which require it. See .Xr doas 1 . .El .Pp Some variables are intended to be set at runtime in the environment or in .Xr mk.conf 5 , but not in the regress Makefile itself. .Sh GUIDELINES If an individual test passes, .Sy make Ar testname should return with an exit status of 0. If it fails, it should return with a non-zero exit status. .Pp If a test cannot be executed because a package is not installed or some environment variable is not set, .Sy make Ar testname should print .Qq SKIPPED to stdout and exit with status 0. To skip everything, implement the .Cm regress target with a command that prints .Qq SKIPPED . .Pp Some tests may require a special setup on the test machine that has to be done manually before testing. This requirement has to be documented in the Makefile or in a .Pa README file. The test should find out whether the setup exists before running and print .Qq SKIPPED and exit if it is missing. .Pp Tests should not fail because an intended feature has not been implemented yet. To avoid such false failures, a test should show the reason, print .Qq DISABLED to stdout and exit with status 0. .Pp Tests must be able to run with an .Pa obj directory. In case the test is not linked to the build or the tester forgot to run .Sy make Cm obj before, this directory or symlink may not exist. Then the test should run anyway. .Pp Test are executed with .Sy make Cm regress , but running .Sy make Cm all or .Sy make should have the same effect. Tests must be runnable by root, and may also succeed when run as a regular user. Tests must not assume they have a controlling tty, to allow them to be run by .Xr cron 8 . An individual regress test may create a pseudo tty if it needs one. .Pp Tests should use the binaries installed and the kernel running on the local system. They may use environment variables to test alternative binaries or remote kernels running on other machines. In some cases a test may need binaries or libraries or object files to be present in .Pa /usr/obj/ that exist only after .Sy make Cm build was run in .Pa /usr/src/ . The test must not assume that they have already been built, but should run .Sy make in the appropriate source directory as a dependency. For missing generated source or header files a target called .Sy make Cm generated is common. The .Pa /usr/src/ tree can be found with a relative path or with the .Ev BSDSRCDIR variable. .Sh SEE ALSO .Xr bsd.port.mk 5 .Sh HISTORY The regression system originally came from .Nx , with many tests added by .Ox since. The current Makefile framework was written by Artur Grabowski and Marc Espie for .Ox 3.1 . .Sh BUGS AND LIMITATIONS The build system is unable to distinguish between timeouts due to .Ev REGRESS_MAXTIME being exceeded, or a genuine failure occurring.