Add per-CPU caches to the pmemrange allocator.

The caches are used primarily to reduce contention on uvm_lock_fpageq() during
concurrent page faults.  For the moment only uvm_pagealloc() tries to get a
page from the current CPU's cache.  So on some architectures the caches are
also used by the pmap layer.

Each cache is composed of two magazines, design is borrowed from jeff bonwick
vmem's paper and the implementation is similar to the one of pool_cache from
dlg@.  However there is no depot layer and magazines are refilled directly by
the pmemrange allocator.

This version includes splvm()/splx() dances because the buffer cache flips
buffers in interrupt context.  So we have to prevent recursive accesses to
per-CPU magazines.

Tested by naddy@, solene@, krw@, robert@, claudio@ and Laurence Tratt.

ok claudio@, kettenis@

1 files changed, 4 insertions, 1 deletions

diff --git a/sys/arch/i386/include/cpu.h b/sys/arch/i386/include/cpu.h
index 5709bf5480c..60f0d044a4d 100644
--- a/sys/arch/i386/include/cpu.h
+++ b/sys/arch/i386/include/cpu.h
@@ -1,4 +1,4 @@
-/*	$OpenBSD: cpu.h,v 1.187 2024/04/19 10:22:50 mpi Exp $	*/
+/*	$OpenBSD: cpu.h,v 1.188 2024/05/01 12:54:27 mpi Exp $	*/
 /*	$NetBSD: cpu.h,v 1.35 1996/05/05 19:29:26 christos Exp $	*/
 
 /*-
@@ -69,6 +69,7 @@
 #include <sys/sched.h>
 #include <sys/sensors.h>
 #include <sys/srp.h>
+#include <uvm/uvm_percpu.h>
 
 struct intrsource;
 
@@ -99,6 +100,8 @@ struct cpu_info {
 
 #if defined(MULTIPROCESSOR)
 	struct srp_hazard ci_srp_hazards[SRP_HAZARD_NUM];
+#define __HAVE_UVM_PERCPU
+	struct uvm_pmr_cache	ci_uvm;
 #endif
 
 	/*