| Age | Commit message (Collapse) | Author |
|---|
|
Protect all remaining write access to inp_faddr and inp_laddr with
inpcb table mutex. Document inpcb locking for foreign and local
address and port and routing table id. Reading will be made MP
safe by adding per socket rw-locks in a next step.
OK sashan@ mvs@
|
|
in_pcbbind(), in_pcbconnect(), and in6_pcbconnect() have to set
addresses and ports within the same critical section as the inpcb
hash table calculation. Also lookup and address selection have to
be protected to avoid bindings and connections that are not unique.
For that in_pcbpickport() and in_pcbbind_locked() expect that the
table mutex is already taken. The functions in_pcblookup_lock(),
in_pcblookup_local_lock(), and in_pcbaddrisavail_lock() grab the
mutex iff the lock parameter is IN_PCBLOCK_GRAB. Otherwise the
parameter is IN_PCBLOCK_HOLD has the lock has to be taken already.
Note that in_pcblookup_lock() and in_pcblookup_local() return an
inp with increased reference iff they take and release the lock.
Otherwise the caller protects the life time of the inp.
This gives enough flexibility that in_pcbbind() and in_pcbconnect()
can hold the table mutex when they need it. The public inpcb API
does not change.
OK sashan@ mvs@
|
|
Since soreceive() runs in parallel for raw sockets, sbappendaddr()
has to be protected by inpcb mutex. This was missing in multicast
forwarding which is running with a combination of shared net lock
and kernel lock. soreceive() uses shared net lock and mutex per
inpcb. Grab mutex before sbappendaddr() in socket_send() and
socket6_send().
panic receive 1 reported by Jo Geraerts
OK mvs@ claudio@
|
|
There exists no struct in6pcb in OpenBSD, this was an old kame idea.
Calling the local variable in6p does not make sense, it is actually
a struct inpcb. Also in6p is not used consistently in inet6 code.
Having the same convention for IPv4 and IPv6 is less confusing.
OK sashan@ mvs@
|
|
During initialization in_pcballoc() sets INP_IPV6 once to avoid
reaching through inp_socket->so_proto->pr_domain->dom_family. Use
this flag consistently.
OK sashan@ mvs@
|
|
The inpcb hash table is protected by table->inpt_mtx. The hash is
based on addresses, ports, and routing table. These fields were
not sychronized with the hash. Put writes and hash update into the
same critical section.
Move the updates from ip_ctloutput(), ip6_ctloutput(), syn_cache_get(),
tcp_connect(), udp_disconnect() to dedicated inpcb set functions.
There they use the same table mutex as in_pcbrehash().
in_pcbbind(), in_pcbconnect(), and in6_pcbconnect() need more work
and are not included yet.
OK sashan@ mvs@
|
|
The public interface is in_pcbconnect(). It dispatches to
in6_pcbconnect() if necessary. Call the former from tcp_connect()
and udp_connect().
In in6_pcbconnect() initialization in6a = NULL is not necessary.
in6_pcbselsrc() sets the pointer, but does not read the value.
Pass a constant in6_addr pointer to in6_pcbselsrc() and in6_selectsrc().
It returns a reference to the address of some internal data structure.
We want to be sure that in6_addr is not modified this way. IPv4
in_pcbselsrc() solves this by passing a copy of the address.
OK kn@ sashan@ mvs@
|
|
Struct inpcb field inp_socket is initialized in in_pcballoc(). It
is not NULL and never changed.
OK mvs@
|
|
rip6_output() did modify inp_outputopts6 temporarily to provide
different ip6_pktopts to in6_embedscope(). Better pass inp_outputopts6
and inp_moptions6 as separate arguments to in6_embedscope().
Simplify the code that deals with these options in in6_embedscope().
Doucument inp_moptions and inp_moptions6 as protected by net lock.
OK kn@
|
|
ip_output() received inp as parameter. This is only used to lookup
the IPsec level of the socket. Reasoning about MP locking is much
easier if only relevant data is passed around. Convert ip_output()
to receive constant inp_seclevel as argument and mark it as protected
by net lock.
OK mvs@
|
|
For implementing MP safe route lookup, it helps to know which
function parameters are constant. Add some const declarations, so
that the compiler guarantees that sockaddr dst parameter of
rtable_match() does not change.
OK dlg@
|
|
Using a scratch buffer makes it possible to take a consistent snapshot of
per-CPU counters without having to allocate memory.
Makes ddb(4) show uvmexp command work in OOM situations.
ok kn@, mvs@, cheloha@
|
|
When called with NULL options, ip_output() and ip6_output() are MP
safe. Convert exclusive to shared net lock in send dispatch.
OK mpi@
|
|
More complete solution after tb@ pointed out what Coverity missed.
ok tb@
|
|
Coverity CID #1566406
ok phessler@
|
|
In in6_ifdetach() two struct rtentry were leaked. This was triggered
by regress/sbin/route and detected with btrace(8) refcnt. The
reference returned by rtalloc() must be freed with rtfree() in all
cases.
OK phessler@ mvs@
|
|
When doing LRO (Large Receive Offload), the drivers, currently ix(4)
and lo(4) only, record an upper bound of the size of the original
packets in ph_mss. When sending, either stack or hardware must
chop the packets with TSO (TCP Segmentation Offload) to that size.
That means we have to call tcp_if_output_tso() before ifp->if_output().
Put that logic into if_output_tso() to avoid code duplication. As
TCP packets on the wire do not get larger that way, path MTU discovery
should still work.
tested by and OK jan@
|
|
Replace hand-rolled reference counting with refcnt_init(9) and hook it up
with a new dt(4) probe.
OK bluhm mvs
|
|
Goal is to run UDP input in parallel. Btrace kstack analysis shows
that SIP hash for PCB lookup is quite expensive. When running in
parallel, there is also lock contention on the PCB table mutex.
It results in better performance to calculate the hash value before
taking the mutex. The hash secret has to be constant as hash
calculation must not depend on values protected by the table mutex.
Do not reseed anymore when hash table gets resized.
Analysis also shows that asserting a rw_lock while holding a mutex
is a bit expensive. Just remove the netlock assert.
OK dlg@ mvs@
|
|
First try to send with TSO. The goto senderr handles icmp6 redirect
and other errors. If TSO is not necessary and the interface MTU
fits, just send the packet. Again goto senderr handles icmp6.
Finally care about icmp6 packet too big.
tested and OK jan@
|
|
ok bluhm
|
|
with if_mtu and not the packet checksum flags. ph_mss contains the
size of the copped packets.
OK jan@
|
|
Also fix ip6_forwarding of TSO packets with tcp_if_output_tso().
With a lot of testing from Hrvoje Popovski
and a lot of tweaks from bluhm@
ok bluhm@
|
|
When sending TCP packets with software TSO to the local address of
a physical interface, the TCP checksum was miscalculated. As the
small MSS is taken from the physical interface, but the large MTU
of the loopback interface is used, large TSO packets are generated,
but sent directly to the loopback interface. There we need the
regular pseudo header checksum and not the modified without packet
length.
To avoid this confusion, use the same decision for checksum generation
in in_proto_cksum_out() as for using hardware TSO in tcp_if_output_tso().
bug reported and tested by robert@ bket@ Hrvoje Popovski
OK claudio@ jan@
|
|
If the driver of a network interface claims to support TSO, do not
chop the packet in software, but pass it down to the interface
layer.
Precalculate parts of the pseudo header checksum, but without the
packet length. The length of all generated smaller packets is not
known yet. Driver and hardware will use the mbuf packet header
field ph_mss to calculate it and update checksum.
Introduce separate flags IFCAP_TSOv4 and IFCAP_TSOv6 as hardware
might support ony one protocol family. The old flag IFXF_TSO is
only relevant for large receive offload. It is missnamed, but keep
that for now.
Note that drivers do not set TSO capabilites yet. Also the ifconfig
flags and pseudo interfaces capabilities will be done separately.
So this commit should not change behavior.
heavily based on the work from jan@; OK sashan@
|
|
entries in rt_llinfo are protected either by exclusive netlock or
the ND6 mutex. The performance critical lookup path in nd6_resolve()
uses shared netlock, but is not lockless. In contrast to ARP it
grabs the mutex also in the common case.
tested by Hrvoje Popovski; with and OK kn@
|
|
OK mvs@
|
|
meant as a fallback if network hardware does not support TSO. Driver
support is still work in progress. TCP output generates large
packets. In IP output the packet is chopped to TCP maximum segment
size. This reduces the CPU cycles used by pf. The regular output
could be assisted by hardware later, but pf route-to and IPsec needs
the software fallback in general.
For performance comparison or to workaround possible bugs, sysctl
net.inet.tcp.tso=0 disables the feature. netstat -s -p tcp shows
TSO counter with chopped and generated packets.
based on work from jan@
tested by jmc@ jan@ Hrvoje Popovski
OK jan@ claudio@
|
|
is passed to ifp->if_output(). The fragment code has its own
checksum calculation and the other paths end in goto bad.
OK claudio@
|
|
rt->rt_llinfo has to be guaranteed. Replace the complicated logic
in nd6_rtrequest() case RTM_ADD with what we have in ARP. This
avoids accessing ln here.
Digging through histroy shows a lot of refactoring that makes
rt_expire handling in RTM_ADD obsolete. Just initialize it to 0.
Cloning and local routes should never expire. If RTF_LLINFO is
set, ln should not be NULL. So nd6_llinfo_settimer() was not reached
in this case.
While there, remove obsolete comments and #if 0 code that never
worked.
OK kn@ claudio@
|
|
nd6_mtx must be held longer in nd6_rtrequest() case RTM_RESOLVE.
OK kn@
|
|
if_output_ml() to send mbuf lists to interfaces. This can be used
for TSO, fragments, ARP and ND6. Rename variable fml to ml. In
pf_route6() split the if else block. Put the safety check (hlen +
firstlen < tlen) into ip_fragment(). It makes the code correct in
case the packet is too short to be fragmented. This should not
happen, but other functions also have this logic.
No functional change. OK sashan@
|
|
only protects nd6_list. It does not unlock ND6 from kernel lock
yet.
OK kn@
|
|
code for ARP has been unlocked a while ago.
OK kn@
|
|
OK kn@
|
|
when receiving a valid Neighbor Advertisement.
OK florian@ kn@
|
|
doesn't know where to send traffic. This will send an unsolicited
neighbor advertisement, as described in RFC9131, to the all-routers
multicast address so all routers on the same link will learn the path
back to the address.
This is intended to speed up the first return packet on an IPv6 interface.
OK florian@
|
|
|
|
|
|
Tested as part of bigger unlock diffs, commit now as tiny first step.
OK bluhm
|
|
response. Implement analog sysctl net.inet6.icmp6.nd6_queued for
ND6 to reduce places where mbufs can hide within the kernel.
Atomic operations operate on unsigned int. Make the type of total
hold queue length consistent.
Use atomic load to read the value for the sysctl. This clarifies
why no lock around sysctl_rdint() is needed.
OK mvs@ kn@
|
|
ND6 did only hold a single packet. Unify the logic and add a mbuf
hold queue to struct llinfo_nd6. This is MP safe and queue limits
are tracked with atomic operations. New function if_mqoutput() has
common code for ARP and ND6. ln_saddr6 holds the source address
of the requesting packet. That is easier than fiddling with mbuf
queue in nd6_ns_output().
OK kn@
|
|
|
|
Feedback OK bluhm
|
|
checksum may be wrong. Locally generated packets diverted by pf
out rules may have no checksum due to to hardware offloading.
Calculate the checksum in that case.
OK mvs@ sashan@
|
|
|
|
Less diff between them; merging three returns into one also reduces
upcoming unlock diffs. OK bluhm
|
|
makes the two familiar functions look more alike; OK bluhm
|
|
and not some endless loop with some gotos.
OK kn@
|
|
receive buffer. As it was done for SS_CANTSENDMORE bit, the definition
kept as is, but now these bits belongs to the `sb_state' of receive
buffer. `sb_state' ored with `so_state' when socket data exporting to the
userland.
ok bluhm@
|
