KERN_CPTIME2: set ENODEV if the CPU is offline.

This lets userspace distinguish between idle CPUs and those that are
not schedulable because hw.smt=0.

A subsequent commit probably needs to add documentation for this
to sysctl.2 (and perhaps elsewhere) after the dust settles.

Also included here are changes to systat(1) and top(1) that account
for the ENODEV case and adjust behavior accordingly:

 - systat(1)'s cpu view prints placeholder marks ('-') instead of
   percentages for each state if the given CPU is offline.

 - systat(1)'s vmstat view checks for offline CPUs when computing the
   machine state total and excludes them, so the CPU usage graph
   only represents the states for online CPUs.

 - top(1) does not draw CPU rows for offline CPUs when the view is
   redrawn.  If CPUs "go offline", percentages for each state are
   replaced by placeholder marks ('-'); the view will need to be
   redrawn to remove these rows.  If CPUs "go online" the view will
   need to be redrawn to show these new CPUs.  In "combined CPU" mode,
   the count and the state totals only represent online CPUs.

Ports using KERN_CPTIME2 will need to be updated.  The changes
described above to make systat(1) and top(1) aware of the ENODEV
case *and* gracefully handle a changing HW_NCPUONLINE while the
application is running are not necessarily appropriate for each
and every port.

The changes described above are so extensive in part to demonstrate
one way a program *might* be made robust to changing CPU availability.
In particular, changing hw.smt after boot is an extremely rare event,
and this needs to be weighed when updating ports.

The logic needed to account for the KERN_CPTIME2 ENODEV case is
very roughly:

	if (sysctl(...) == -1) {
		if (errno != ENODEV) {
			/* Actual error occurred. */
		} else {
			/* CPU is offline. */
		}
	} else {
		/* CPU is online and CPU states were set by sysctl(2). */
	}

Prompted by deraadt@.  Basic idea for ENODEV from kettenis@.  Discussed at
length with kettenis@.  Additional testing by tb@.

No complaints from hackers@ after a week.

ok kettenis@, "I think you should commit [now]" deraadt@

2 files changed, 10 insertions, 2 deletions

diff --git a/sys/kern/kern_sched.c b/sys/kern/kern_sched.c
index ce9e80066a3..20ae05d0001 100644
--- a/sys/kern/kern_sched.c
+++ b/sys/kern/kern_sched.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: kern_sched.c,v 1.51 2018/07/12 01:23:38 cheloha Exp $	*/
+/*	$OpenBSD: kern_sched.c,v 1.52 2018/09/26 17:23:13 cheloha Exp $	*/
 /*
  * Copyright (c) 2007, 2008 Artur Grabowski <art@openbsd.org>
  *
@@ -832,6 +832,12 @@ sysctl_hwncpuonline(void)
 	return cpuset_cardinality(&sched_all_cpus);
 }
 
+int
+cpu_is_online(struct cpu_info *ci)
+{
+	return cpuset_isset(&sched_all_cpus, ci);
+}
+
 #ifdef __HAVE_CPU_TOPOLOGY
 
 #include <sys/sysctl.h>
diff --git a/sys/kern/kern_sysctl.c b/sys/kern/kern_sysctl.c
index f83cb70a9a6..8b0da7c6475 100644
--- a/sys/kern/kern_sysctl.c
+++ b/sys/kern/kern_sysctl.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: kern_sysctl.c,v 1.347 2018/09/20 18:59:10 bluhm Exp $	*/
+/*	$OpenBSD: kern_sysctl.c,v 1.348 2018/09/26 17:23:13 cheloha Exp $	*/
 /*	$NetBSD: kern_sysctl.c,v 1.17 1996/05/20 17:49:05 mrg Exp $	*/
 
 /*-
@@ -2397,6 +2397,8 @@ sysctl_cptime2(int *name, u_int namelen, void *oldp, size_t *oldlenp,
 	}
 	if (!found)
 		return (ENOENT);
+	if (!cpu_is_online(ci))
+		return (ENODEV);
 
 	return (sysctl_rdstruct(oldp, oldlenp, newp,
 	    &ci->ci_schedstate.spc_cp_time,