amd64: simplify TSC synchronization testing

Computing a per-CPU TSC skew value is error-prone, especially on
multisocket machines and VMs.  My best guess is that larger latencies
appear to the current skew measurement test as TSC desync, and so the
TSC is demoted to a kernel timecounter on these machines or marked
non-monotonic.

This patch eliminates per-CPU TSC skew values.  Instead of trying to
measure and correct for TSC desync we only try to detect desync, which
is less error-prone.  This approach should allow a wider variety of
machines to use the TSC as a timecounter when running OpenBSD.

In the new sync test, both CPUs repeatedly try to detect whether their
TSC is trailing the other CPU's TSC.  The upside to this approach is
that it yields no false positives.  The downside to this approach is
that it takes more time than the current skew measurement test.  Each
test round takes 1ms, and we run up to two rounds per CPU, so this
patch slows boot down by 2ms per AP.

If any CPU fails the sync test, the TSC is marked non-monotonic and a
different timecounter is activated.  The TC_USER flag remains intact.
There is no middle ground where we fall back to only using the TSC in
the kernel.

Before running the test, we check for the IA32_TSC_ADJUST register and
reset it if necessary.  This is a trivial way to work around firmware
bugs that desync the TSC before we reach the kernel.  Unfortunately,
at the moment this register appears to only be available on Intel
processors.  I cannot find an equivalent but differently-named MSR for
AMD processors.

Because there is no per-CPU skew value, there is also no concept of
TSC drift anymore.

Miscellaneous notes:

- This patch adds a new timecounter utility function, tc_reset_quality().
  Used after sync test failure to mark the TSC non-monotonic.

- I have left TSC_DEBUG enabled for now.  Unsure if we should leave it
  enabled for release or not.  If we disable it we no longer run the
  sync test after failing it once.  Running the test even after failure
  provides information about the desync on every CPU.

- Taking 1ms per test round is fairly conservative.  We can experiment
  with and discuss shorter test rounds.  My main goal with a relatively
  long test round is ensuring VMs actually run the test.  It would be
  bad if a hypervisor interrupted the test for so long that it concealed
  desync.

- The use of two test rounds is mostly a diagnostic tool: it would be
  very strange if a CPU passed the first round but failed the second.
  If we ever saw this in the wild it would indicate something odd.

- Most of the desync seen in test reports is on Ryzen CPUs.  I
  believe, but cannot prove, that this is due to a widespread
  firmware bug on AMD motherboards.  Hopefully AMD and/or the
  downstream vendors fix it.

- Fixing TSC desync by writing the TSC directly with WRMSR is very
  difficult.  The TSC is a moving target incrementing very quickly and
  compensating for WRMSR overhead is non-trivial.  We can experiment
  with this, but my confidence is low that we can make it work reliably.

Prompted by deraadt@ and kettenis@ in 2021. Shepherded along by
deraadt@ throughout.  Reprompted by Yuichiro Naito several times.
With input from Yuichiro Naito, naddy@, sthen@, dv@, and deraadt@.

Tested by florian@, gnezdo@, sthen@, Josh Rickmar, dv@, Mohamed Aslan,
Hrvoje Popovski, Yuichiro Naito, semarie@, mlarkin@, asou@, jmatthew@,
Renato Aguiar, and Timo Myyra.

Patch v1: https://marc.info/?l=openbsd-tech&m=164330092208035&w=2
Patch v2: https://marc.info/?l=openbsd-tech&m=164558519712957&w=2
Patch v3: https://marc.info/?l=openbsd-tech&m=165698681018991&w=2
Patch v4: https://marc.info/?l=openbsd-tech&m=165835507113680&w=2
Patch v5: https://marc.info/?l=openbsd-tech&m=165923705118770&w=2

"just commit it" deraadt@

6 files changed, 280 insertions, 128 deletions

diff --git a/sys/arch/amd64/amd64/cpu.c b/sys/arch/amd64/amd64/cpu.c
index 413938981b6..4d0d9062980 100644
--- a/sys/arch/amd64/amd64/cpu.c
+++ b/sys/arch/amd64/amd64/cpu.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: cpu.c,v 1.157 2022/08/07 23:56:06 guenther Exp $	*/
+/*	$OpenBSD: cpu.c,v 1.158 2022/08/12 02:20:36 cheloha Exp $	*/
 /* $NetBSD: cpu.c,v 1.1 2003/04/26 18:39:26 fvdl Exp $ */
 
 /*-
@@ -774,9 +774,9 @@ cpu_init(struct cpu_info *ci)
 	lcr4(cr4 & ~CR4_PGE);
 	lcr4(cr4);
 
-	/* Synchronize TSC */
+	/* Check if TSC is synchronized. */
 	if (cold && !CPU_IS_PRIMARY(ci))
-	      tsc_sync_ap(ci);
+	      tsc_test_sync_ap(ci);
 #endif
 }
 
@@ -856,18 +856,14 @@ cpu_start_secondary(struct cpu_info *ci)
 #endif
 	} else {
 		/*
-		 * Synchronize time stamp counters. Invalidate cache and
-		 * synchronize twice (in tsc_sync_bp) to minimize possible
-		 * cache effects. Disable interrupts to try and rule out any
-		 * external interference.
+		 * Test if TSCs are synchronized.  Invalidate cache to
+		 * minimize possible cache effects.  Disable interrupts to
+		 * try to rule out external interference.
 		 */
 		s = intr_disable();
 		wbinvd();
-		tsc_sync_bp(ci);
+		tsc_test_sync_bp(ci);
 		intr_restore(s);
-#ifdef TSC_DEBUG
-		printf("TSC skew=%lld\n", (long long)ci->ci_tsc_skew);
-#endif
 	}
 
 	if ((ci->ci_flags & CPUF_IDENTIFIED) == 0) {
@@ -892,7 +888,6 @@ void
 cpu_boot_secondary(struct cpu_info *ci)
 {
 	int i;
-	int64_t drift;
 	u_long s;
 
 	atomic_setbits_int(&ci->ci_flags, CPUF_GO);
@@ -907,18 +902,11 @@ cpu_boot_secondary(struct cpu_info *ci)
 		db_enter();
 #endif
 	} else if (cold) {
-		/* Synchronize TSC again, check for drift. */
-		drift = ci->ci_tsc_skew;
+		/* Test if TSCs are synchronized again. */
 		s = intr_disable();
 		wbinvd();
-		tsc_sync_bp(ci);
+		tsc_test_sync_bp(ci);
 		intr_restore(s);
-		drift -= ci->ci_tsc_skew;
-#ifdef TSC_DEBUG
-		printf("TSC skew=%lld drift=%lld\n",
-		    (long long)ci->ci_tsc_skew, (long long)drift);
-#endif
-		tsc_sync_drift(drift);
 	}
 }
 
@@ -944,13 +932,12 @@ cpu_hatch(void *v)
 #endif
 
 	/*
-	 * Synchronize the TSC for the first time. Note that interrupts are
-	 * off at this point.
+	 * Test if our TSC is synchronized for the first time.
+	 * Note that interrupts are off at this point.
 	 */
 	wbinvd();
 	atomic_setbits_int(&ci->ci_flags, CPUF_PRESENT);
-	ci->ci_tsc_skew = 0;	/* reset on resume */
-	tsc_sync_ap(ci);
+	tsc_test_sync_ap(ci);
 
 	lapic_enable();
 	lapic_startclock();
diff --git a/sys/arch/amd64/amd64/tsc.c b/sys/arch/amd64/amd64/tsc.c
index 32736b938aa..fd38dc6359d 100644
--- a/sys/arch/amd64/amd64/tsc.c
+++ b/sys/arch/amd64/amd64/tsc.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: tsc.c,v 1.24 2021/08/31 15:11:54 kettenis Exp $	*/
+/*	$OpenBSD: tsc.c,v 1.25 2022/08/12 02:20:36 cheloha Exp $	*/
 /*
  * Copyright (c) 2008 The NetBSD Foundation, Inc.
  * Copyright (c) 2016,2017 Reyk Floeter <reyk@openbsd.org>
@@ -36,13 +36,6 @@ int		tsc_recalibrate;
 uint64_t	tsc_frequency;
 int		tsc_is_invariant;
 
-#define	TSC_DRIFT_MAX			250
-#define TSC_SKEW_MAX			100
-int64_t	tsc_drift_observed;
-
-volatile int64_t	tsc_sync_val;
-volatile struct cpu_info	*tsc_sync_cpu;
-
 u_int		tsc_get_timecount(struct timecounter *tc);
 void		tsc_delay(int usecs);
 
@@ -236,22 +229,12 @@ cpu_recalibrate_tsc(struct timecounter *tc)
 u_int
 tsc_get_timecount(struct timecounter *tc)
 {
-	return rdtsc_lfence() + curcpu()->ci_tsc_skew;
+	return rdtsc_lfence();
 }
 
 void
 tsc_timecounter_init(struct cpu_info *ci, uint64_t cpufreq)
 {
-#ifdef TSC_DEBUG
-	printf("%s: TSC skew=%lld observed drift=%lld\n", ci->ci_dev->dv_xname,
-	    (long long)ci->ci_tsc_skew, (long long)tsc_drift_observed);
-#endif
-	if (ci->ci_tsc_skew < -TSC_SKEW_MAX || ci->ci_tsc_skew > TSC_SKEW_MAX) {
-		printf("%s: disabling user TSC (skew=%lld)\n",
-		    ci->ci_dev->dv_xname, (long long)ci->ci_tsc_skew);
-		tsc_timecounter.tc_user = 0;
-	}
-
 	if (!(ci->ci_flags & CPUF_PRIMARY) ||
 	    !(ci->ci_flags & CPUF_CONST_TSC) ||
 	    !(ci->ci_flags & CPUF_INVAR_TSC))
@@ -268,111 +251,264 @@ tsc_timecounter_init(struct cpu_info *ci, uint64_t cpufreq)
 		calibrate_tsc_freq();
 	}
 
-	if (tsc_drift_observed > TSC_DRIFT_MAX) {
-		printf("ERROR: %lld cycle TSC drift observed\n",
-		    (long long)tsc_drift_observed);
-		tsc_timecounter.tc_quality = -1000;
-		tsc_timecounter.tc_user = 0;
-		tsc_is_invariant = 0;
-	}
-
 	tc_init(&tsc_timecounter);
 }
 
-/*
- * Record drift (in clock cycles).  Called during AP startup.
- */
 void
-tsc_sync_drift(int64_t drift)
+tsc_delay(int usecs)
 {
-	if (drift < 0)
-		drift = -drift;
-	if (drift > tsc_drift_observed)
-		tsc_drift_observed = drift;
+	uint64_t interval, start;
+
+	interval = (uint64_t)usecs * tsc_frequency / 1000000;
+	start = rdtsc_lfence();
+	while (rdtsc_lfence() - start < interval)
+		CPU_BUSY_CYCLE();
 }
 
+#ifdef MULTIPROCESSOR
+
+#define TSC_DEBUG 1
+
 /*
- * Called during startup of APs, by the boot processor.  Interrupts
- * are disabled on entry.
+ * Protections for global variables in this code:
+ *
+ *	a	Modified atomically
+ *	b	Protected by a barrier
+ *	p	Only modified by the primary CPU
  */
-void
-tsc_read_bp(struct cpu_info *ci, uint64_t *bptscp, uint64_t *aptscp)
-{
-	uint64_t bptsc;
 
-	if (atomic_swap_ptr(&tsc_sync_cpu, ci) != NULL)
-		panic("tsc_sync_bp: 1");
+#define TSC_TEST_MSECS		1	/* Test round duration */
+#define TSC_TEST_ROUNDS		2	/* Number of test rounds */
 
-	/* Flag it and read our TSC. */
-	atomic_setbits_int(&ci->ci_flags, CPUF_SYNCTSC);
-	bptsc = (rdtsc_lfence() >> 1);
+/*
+ * tsc_test_status.val is isolated to its own cache line to limit
+ * false sharing and reduce the test's margin of error.
+ */
+struct tsc_test_status {
+	volatile uint64_t val;		/* [a] Latest RDTSC value */
+	uint64_t pad1[7];
+	uint64_t lag_count;		/* [b] Number of lags seen by CPU */
+	uint64_t lag_max;		/* [b] Biggest lag seen by CPU */
+	int64_t adj;			/* [b] Initial IA32_TSC_ADJUST value */
+	uint64_t pad2[5];
+} __aligned(64);
+struct tsc_test_status tsc_ap_status;	/* Test results from AP */
+struct tsc_test_status tsc_bp_status;	/* Test results from BP */
+uint64_t tsc_test_cycles;		/* [p] TSC cycles per test round */
+const char *tsc_ap_name;		/* [b] Name of AP running test */
+volatile u_int tsc_egress_barrier;	/* [a] Test end barrier */
+volatile u_int tsc_ingress_barrier;	/* [a] Test start barrier */
+volatile u_int tsc_test_rounds;		/* [p] Remaining test rounds */
+int tsc_is_synchronized = 1;		/* [p] Have we ever failed the test? */
+
+void tsc_report_test_results(void);
+void tsc_reset_adjust(struct tsc_test_status *);
+void tsc_test_ap(void);
+void tsc_test_bp(void);
 
-	/* Wait for remote to complete, and read ours again. */
-	while ((ci->ci_flags & CPUF_SYNCTSC) != 0)
-		membar_consumer();
-	bptsc += (rdtsc_lfence() >> 1);
+void
+tsc_test_sync_bp(struct cpu_info *ci)
+{
+	if (!tsc_is_invariant)
+		return;
+#ifndef TSC_DEBUG
+	/* No point in testing again if we already failed. */
+	if (!tsc_is_synchronized)
+		return;
+#endif
+	/* Reset IA32_TSC_ADJUST if it exists. */
+	tsc_reset_adjust(&tsc_bp_status);
+
+	/* Reset the test cycle limit and round count. */
+	tsc_test_cycles = TSC_TEST_MSECS * tsc_frequency / 1000;
+	tsc_test_rounds = TSC_TEST_ROUNDS;
+
+	do {
+		/*
+		 * Pass through the ingress barrier, run the test,
+		 * then wait for the AP to reach the egress barrier.
+		 */
+		atomic_inc_int(&tsc_ingress_barrier);
+		while (tsc_ingress_barrier != 2)
+			CPU_BUSY_CYCLE();
+		tsc_test_bp();
+		while (tsc_egress_barrier != 1)
+			CPU_BUSY_CYCLE();
+
+		/*
+		 * Report what happened.  Adjust the TSC's quality
+		 * if this is the first time we've failed the test.
+		 */
+		tsc_report_test_results();
+		if (tsc_ap_status.lag_count || tsc_bp_status.lag_count) {
+			if (tsc_is_synchronized) {
+				tsc_is_synchronized = 0;
+				tc_reset_quality(&tsc_timecounter, -1000);
+			}
+			tsc_test_rounds = 0;
+		} else
+			tsc_test_rounds--;
+
+		/*
+		 * Clean up for the next round.  It is safe to reset the
+		 * ingress barrier because at this point we know the AP
+		 * has reached the egress barrier.
+		 */
+		memset(&tsc_ap_status, 0, sizeof tsc_ap_status);
+		memset(&tsc_bp_status, 0, sizeof tsc_bp_status);
+		tsc_ingress_barrier = 0;
+		if (tsc_test_rounds == 0)
+			tsc_ap_name = NULL;
+
+		/*
+		 * Pass through the egress barrier and release the AP.
+		 * The AP is responsible for resetting the egress barrier.
+		 */
+		if (atomic_inc_int_nv(&tsc_egress_barrier) != 2)
+			panic("%s: unexpected egress count", __func__);
+	} while (tsc_test_rounds > 0);
+}
 
-	/* Wait for the results to come in. */
-	while (tsc_sync_cpu == ci)
-		CPU_BUSY_CYCLE();
-	if (tsc_sync_cpu != NULL)
-		panic("tsc_sync_bp: 2");
+void
+tsc_test_sync_ap(struct cpu_info *ci)
+{
+	if (!tsc_is_invariant)
+		return;
+#ifndef TSC_DEBUG
+	if (!tsc_is_synchronized)
+		return;
+#endif
+	/* The BP needs our name in order to report any problems. */
+	if (atomic_cas_ptr(&tsc_ap_name, NULL, ci->ci_dev->dv_xname) != NULL) {
+		panic("%s: %s: tsc_ap_name is not NULL: %s",
+		    __func__, ci->ci_dev->dv_xname, tsc_ap_name);
+	}
 
-	*bptscp = bptsc;
-	*aptscp = tsc_sync_val;
+	tsc_reset_adjust(&tsc_ap_status);
+
+	/*
+	 * The AP is only responsible for running the test and
+	 * resetting the egress barrier.  The BP handles everything
+	 * else.
+	 */
+	do {
+		atomic_inc_int(&tsc_ingress_barrier);
+		while (tsc_ingress_barrier != 2)
+			CPU_BUSY_CYCLE();
+		tsc_test_ap();
+		atomic_inc_int(&tsc_egress_barrier);
+		while (atomic_cas_uint(&tsc_egress_barrier, 2, 0) != 2)
+			CPU_BUSY_CYCLE();
+	} while (tsc_test_rounds > 0);
 }
 
 void
-tsc_sync_bp(struct cpu_info *ci)
+tsc_report_test_results(void)
 {
-	uint64_t bptsc, aptsc;
-
-	tsc_read_bp(ci, &bptsc, &aptsc); /* discarded - cache effects */
-	tsc_read_bp(ci, &bptsc, &aptsc);
+	u_int round = TSC_TEST_ROUNDS - tsc_test_rounds + 1;
 
-	/* Compute final value to adjust for skew. */
-	ci->ci_tsc_skew = bptsc - aptsc;
+	if (tsc_bp_status.adj != 0) {
+		printf("tsc: cpu0: IA32_TSC_ADJUST: %lld -> 0\n",
+		    tsc_bp_status.adj);
+	}
+	if (tsc_ap_status.adj != 0) {
+		printf("tsc: %s: IA32_TSC_ADJUST: %lld -> 0\n",
+		    tsc_ap_name, tsc_ap_status.adj);
+	}
+	if (tsc_ap_status.lag_count > 0 || tsc_bp_status.lag_count > 0) {
+		printf("tsc: cpu0/%s: sync test round %u/%u failed\n",
+		    tsc_ap_name, round, TSC_TEST_ROUNDS);
+	}
+	if (tsc_bp_status.lag_count > 0) {
+		printf("tsc: cpu0/%s: cpu0: %llu lags %llu cycles\n",
+		    tsc_ap_name, tsc_bp_status.lag_count,
+		    tsc_bp_status.lag_max);
+	}
+	if (tsc_ap_status.lag_count > 0) {
+		printf("tsc: cpu0/%s: %s: %llu lags %llu cycles\n",
+		    tsc_ap_name, tsc_ap_name, tsc_ap_status.lag_count,
+		    tsc_ap_status.lag_max);
+	}
 }
 
 /*
- * Called during startup of AP, by the AP itself.  Interrupts are
- * disabled on entry.
+ * Reset IA32_TSC_ADJUST if we have it.
+ *
+ * XXX We should rearrange cpu_hatch() so that the feature
+ * flags are already set before we get here.  Check CPUID
+ * by hand until then.
  */
 void
-tsc_post_ap(struct cpu_info *ci)
+tsc_reset_adjust(struct tsc_test_status *tts)
 {
-	uint64_t tsc;
-
-	/* Wait for go-ahead from primary. */
-	while ((ci->ci_flags & CPUF_SYNCTSC) == 0)
-		membar_consumer();
-	tsc = (rdtsc_lfence() >> 1);
-
-	/* Instruct primary to read its counter. */
-	atomic_clearbits_int(&ci->ci_flags, CPUF_SYNCTSC);
-	tsc += (rdtsc_lfence() >> 1);
-
-	/* Post result.  Ensure the whole value goes out atomically. */
-	(void)atomic_swap_64(&tsc_sync_val, tsc);
-
-	if (atomic_swap_ptr(&tsc_sync_cpu, NULL) != ci)
-		panic("tsc_sync_ap");
+	uint32_t eax, ebx, ecx, edx;
+
+	CPUID(0, eax, ebx, ecx, edx);
+	if (eax >= 7) {
+		CPUID_LEAF(7, 0, eax, ebx, ecx, edx);
+		if (ISSET(ebx, SEFF0EBX_TSC_ADJUST)) {
+			tts->adj = rdmsr(MSR_TSC_ADJUST);
+			if (tts->adj != 0)
+				wrmsr(MSR_TSC_ADJUST, 0);
+		}
+	}
 }
 
 void
-tsc_sync_ap(struct cpu_info *ci)
+tsc_test_ap(void)
 {
-	tsc_post_ap(ci);
-	tsc_post_ap(ci);
+	uint64_t ap_val, bp_val, end, lag;
+
+	ap_val = rdtsc_lfence();
+	end = ap_val + tsc_test_cycles;
+	while (__predict_true(ap_val < end)) {
+		/*
+		 * Get the BP's latest TSC value, then read the AP's
+		 * TSC.  LFENCE is a serializing instruction, so we
+		 * know bp_val predates ap_val.  If ap_val is smaller
+		 * than bp_val then the AP's TSC must trail that of
+		 * the BP and the counters cannot be synchronized.
+		 */
+		bp_val = tsc_bp_status.val;
+		ap_val = rdtsc_lfence();
+		tsc_ap_status.val = ap_val;
+
+		/*
+		 * Record the magnitude of the problem if the AP's TSC
+		 * trails the BP's TSC.
+		 */
+		if (__predict_false(ap_val < bp_val)) {
+			tsc_ap_status.lag_count++;
+			lag = bp_val - ap_val;
+			if (tsc_ap_status.lag_max < lag)
+				tsc_ap_status.lag_max = lag;
+		}
+	}
 }
 
+/*
+ * This is similar to tsc_test_ap(), but with all relevant variables
+ * flipped around to run the test from the BP's perspective.
+ */
 void
-tsc_delay(int usecs)
+tsc_test_bp(void)
 {
-	uint64_t interval, start;
-
-	interval = (uint64_t)usecs * tsc_frequency / 1000000;
-	start = rdtsc_lfence();
-	while (rdtsc_lfence() - start < interval)
-		CPU_BUSY_CYCLE();
+	uint64_t ap_val, bp_val, end, lag;
+
+	bp_val = rdtsc_lfence();
+	end = bp_val + tsc_test_cycles;
+	while (__predict_true(bp_val < end)) {
+		ap_val = tsc_ap_status.val;
+		bp_val = rdtsc_lfence();
+		tsc_bp_status.val = bp_val;
+
+		if (__predict_false(bp_val < ap_val)) {
+			tsc_bp_status.lag_count++;
+			lag = ap_val - bp_val;
+			if (tsc_bp_status.lag_max < lag)
+				tsc_bp_status.lag_max = lag;
+		}
+	}
 }
+
+#endif /* MULTIPROCESSOR */
diff --git a/sys/arch/amd64/include/cpu.h b/sys/arch/amd64/include/cpu.h
index e504cea14c2..b8db48f2714 100644
--- a/sys/arch/amd64/include/cpu.h
+++ b/sys/arch/amd64/include/cpu.h
@@ -1,4 +1,4 @@
-/*	$OpenBSD: cpu.h,v 1.146 2022/08/07 23:56:06 guenther Exp $	*/
+/*	$OpenBSD: cpu.h,v 1.147 2022/08/12 02:20:36 cheloha Exp $	*/
 /*	$NetBSD: cpu.h,v 1.1 2003/04/26 18:39:39 fvdl Exp $	*/
 
 /*-
@@ -216,8 +216,6 @@ struct cpu_info {
 	paddr_t		ci_vmxon_region_pa;
 	struct vmxon_region *ci_vmxon_region;
 
-	int64_t		ci_tsc_skew;		/* counter skew vs cpu0 */
-
 	char		ci_panicbuf[512];
 
 	paddr_t		ci_vmcs_pa;
@@ -235,7 +233,6 @@ struct cpu_info {
 #define CPUF_CONST_TSC	0x0040		/* CPU has constant TSC */
 #define CPUF_INVAR_TSC	0x0100		/* CPU has invariant TSC */
 
-#define CPUF_SYNCTSC	0x0800		/* Synchronize TSC */
 #define CPUF_PRESENT	0x1000		/* CPU is present */
 #define CPUF_RUNNING	0x2000		/* CPU is running */
 #define CPUF_PAUSE	0x4000		/* CPU is paused in DDB */
diff --git a/sys/arch/amd64/include/cpuvar.h b/sys/arch/amd64/include/cpuvar.h
index 54fa2a583c1..882ddd5757b 100644
--- a/sys/arch/amd64/include/cpuvar.h
+++ b/sys/arch/amd64/include/cpuvar.h
@@ -1,4 +1,4 @@
-/*	$OpenBSD: cpuvar.h,v 1.11 2021/05/16 04:33:05 jsg Exp $	*/
+/*	$OpenBSD: cpuvar.h,v 1.12 2022/08/12 02:20:36 cheloha Exp $	*/
 /* 	$NetBSD: cpuvar.h,v 1.1 2003/03/01 18:29:28 fvdl Exp $ */
 
 /*-
@@ -97,8 +97,7 @@ void identifycpu(struct cpu_info *);
 void cpu_init(struct cpu_info *);
 void cpu_init_first(void);
 
-void tsc_sync_drift(int64_t);
-void tsc_sync_bp(struct cpu_info *);
-void tsc_sync_ap(struct cpu_info *);
+void tsc_test_sync_bp(struct cpu_info *);
+void tsc_test_sync_ap(struct cpu_info *);
 
 #endif
diff --git a/sys/kern/kern_tc.c b/sys/kern/kern_tc.c
index 58204d0ea9f..b89170a573a 100644
--- a/sys/kern/kern_tc.c
+++ b/sys/kern/kern_tc.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: kern_tc.c,v 1.76 2022/07/23 22:58:51 cheloha Exp $ */
+/*	$OpenBSD: kern_tc.c,v 1.77 2022/08/12 02:20:36 cheloha Exp $ */
 
 /*
  * Copyright (c) 2000 Poul-Henning Kamp <phk@FreeBSD.org>
@@ -458,6 +458,38 @@ tc_init(struct timecounter *tc)
 	timecounter = tc;
 }
 
+/*
+ * Change the given timecounter's quality.  If it is the active
+ * counter and it is no longer the best counter, activate the
+ * best counter.
+ */
+void
+tc_reset_quality(struct timecounter *tc, int quality)
+{
+	struct timecounter *best = &dummy_timecounter, *tmp;
+
+	if (tc == &dummy_timecounter)
+		panic("%s: cannot change dummy counter quality", __func__);
+
+	tc->tc_quality = quality;
+	if (timecounter == tc) {
+		SLIST_FOREACH(tmp, &tc_list, tc_next) {
+			if (tmp->tc_quality < 0)
+				continue;
+			if (tmp->tc_quality < best->tc_quality)
+				continue;
+			if (tmp->tc_quality == best->tc_quality &&
+			    tmp->tc_frequency < best->tc_frequency)
+				continue;
+			best = tmp;
+		}
+		if (best != tc) {
+			enqueue_randomness(best->tc_get_timecount(best));
+			timecounter = best;
+		}
+	}
+}
+
 /* Report the frequency of the current timecounter. */
 u_int64_t
 tc_getfrequency(void)
diff --git a/sys/sys/timetc.h b/sys/sys/timetc.h
index dccee1364bd..d88b69809b0 100644
--- a/sys/sys/timetc.h
+++ b/sys/sys/timetc.h
@@ -1,4 +1,4 @@
-/*	$OpenBSD: timetc.h,v 1.12 2020/07/06 13:33:09 pirofti Exp $ */
+/*	$OpenBSD: timetc.h,v 1.13 2022/08/12 02:20:36 cheloha Exp $ */
 
 /*
  * Copyright (c) 2000 Poul-Henning Kamp <phk@FreeBSD.org>
@@ -120,6 +120,7 @@ extern struct timekeep *timekeep;
 u_int64_t tc_getfrequency(void);
 u_int64_t tc_getprecision(void);
 void	tc_init(struct timecounter *tc);
+void	tc_reset_quality(struct timecounter *, int);
 void	tc_setclock(const struct timespec *ts);
 void	tc_setrealtimeclock(const struct timespec *ts);
 void	tc_ticktock(void);