Fix the return value for the FUTEX_WAIT/FUTEX_WAIT_PRIVATE futex(2)

operation.  System calls should return -1 and set errno when they fail.
They should not return an errno value directly.  This matches how
the Linux version of futex(2) behaves and what Mesa expects.  This fixes
a bug in Mesa where a timeout wouldn't be reported properly.

Technically this is an ABI break.  But libc and libpthread were changed
to be compatible with both the old and new ABI, and code outside of base
almost certainly expects Linux compatible behaviour.  If you have not
rebuilt libc and the last few days, upgrade using a snap.

Mesa issue discovered by jsg@
ok mpi@, deraadt@

2 files changed, 11 insertions, 7 deletions

diff --git a/lib/libc/sys/futex.2 b/lib/libc/sys/futex.2
index 6dfc12f0617..dfe45adc4ce 100644
--- a/lib/libc/sys/futex.2
+++ b/lib/libc/sys/futex.2
@@ -1,4 +1,4 @@
-.\" $OpenBSD: futex.2,v 1.5 2019/01/18 05:06:37 cheloha Exp $
+.\" $OpenBSD: futex.2,v 1.6 2021/05/26 18:11:59 kettenis Exp $
 .\"
 .\" Copyright (c) 2017 Martin Pieuchot
 .\"
@@ -14,7 +14,7 @@
 .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 .\"
-.Dd $Mdocdate: January 18 2019 $
+.Dd $Mdocdate: May 26 2021 $
 .Dt FUTEX 2
 .Os
 .Sh NAME
@@ -98,7 +98,10 @@ returns zero if woken by a matching
 .Dv FUTEX_WAKE
 or
 .Dv FUTEX_REQUEUE
-call, otherwise an error number is returned to indicate the error.
+call.
+Otherwise, a value of \-1 is returned and
+.Va errno
+is set to indicate the error.
 .Sh ERRORS
 .Fn futex
 will fail if:
diff --git a/sys/kern/sys_futex.c b/sys/kern/sys_futex.c
index 5ef16fffa9a..c4cbe36364d 100644
--- a/sys/kern/sys_futex.c
+++ b/sys/kern/sys_futex.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: sys_futex.c,v 1.17 2021/03/10 10:21:47 jsg Exp $ */
+/*	$OpenBSD: sys_futex.c,v 1.18 2021/05/26 18:11:59 kettenis Exp $ */
 
 /*
  * Copyright (c) 2016-2017 Martin Pieuchot
@@ -98,6 +98,7 @@ sys_futex(struct proc *p, void *v, register_t *retval)
 	const struct timespec *timeout = SCARG(uap, timeout);
 	void *g = SCARG(uap, g);
 	int flags = 0;
+	int error = 0;
 
 	if (op & FUTEX_PRIVATE_FLAG)
 		flags |= FT_PRIVATE;
@@ -107,7 +108,7 @@ sys_futex(struct proc *p, void *v, register_t *retval)
 	case FUTEX_WAIT_PRIVATE:
 		KERNEL_LOCK();
 		rw_enter_write(&ftlock);
-		*retval = futex_wait(uaddr, val, timeout, flags);
+		error = futex_wait(uaddr, val, timeout, flags);
 		rw_exit_write(&ftlock);
 		KERNEL_UNLOCK();
 		break;
@@ -124,11 +125,11 @@ sys_futex(struct proc *p, void *v, register_t *retval)
 		rw_exit_write(&ftlock);
 		break;
 	default:
-		*retval = ENOSYS;
+		error = ENOSYS;
 		break;
 	}
 
-	return 0;
+	return error;
 }
 
 /*