On Mon, Jan 29, 2024 at 04:20:02PM +0300, Vitaliy Makkoveev wrote:
> So, I want to commit soassertlocked() right now. sofilt_lock() diff
> could be pushed to the snaps, so we could test them together.

Regress not finished, but here are the first findings.  I think it
is regress/sys/net/pf_divert which requires complicated setup with
two machines.

divert_packet runs with shared net lock, without rwlock socket lock,
but with inpcb mutex for receive socket buffer.  I think soassertlocked
has to take pru_lock() into account.

And for sorwakeup this is not sufficent, there I hope for your
sofilt_lock().

This is basically the same what my diff found a month ago.

bluhm

splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sbappendaddr(d7e499a0,d7e49a14,f58021f4,d8ceb200,0) at sbappendaddr+0x18
divert_packet(d8ceb200,1,9a02) at divert_packet+0x237
pf_test(2,1,d8366830,f58023a0) at pf_test+0x873
ip_input_if(f58023a0,f580238c,f58023b8,0,d8366830) at ip_input_if+0xab
ipv4_input(d8366830,d8ceb200) at ipv4_input+0x3a
ether_input(d8366830,d8ceb200) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sbappendaddr(d7e499a0,d7e49a14,f58021f4,d8ceb200,0) at sbappendaddr+0x45
divert_packet(d8ceb200,1,9a02) at divert_packet+0x237
pf_test(2,1,d8366830,f58023a0) at pf_test+0x873
ip_input_if(f58023a0,f580238c,f58023b8,0,d8366830) at ip_input_if+0xab
ipv4_input(d8366830,d8ceb200) at ipv4_input+0x3a
ether_input(d8366830,d8ceb200) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sorwakeup(d7e499a0) at sorwakeup+0xd
divert_packet(d8ceb200,1,9a02) at divert_packet+0x254
pf_test(2,1,d8366830,f58023a0) at pf_test+0x873
ip_input_if(f58023a0,f580238c,f58023b8,0,d8366830) at ip_input_if+0xab
ipv4_input(d8366830,d8ceb200) at ipv4_input+0x3a
ether_input(d8366830,d8ceb200) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sowakeup(d7e499a0,d7e49a14) at sowakeup+0x11
sorwakeup(d7e499a0) at sorwakeup+0x56
divert_packet(d8ceb200,1,9a02) at divert_packet+0x254
pf_test(2,1,d8366830,f58023a0) at pf_test+0x873
ip_input_if(f58023a0,f580238c,f58023b8,0,d8366830) at ip_input_if+0xab
ipv4_input(d8366830,d8ceb200) at ipv4_input+0x3a
ether_input(d8366830,d8ceb200) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
knote_locked(d7e49a48,0) at knote_locked+0x1a
sowakeup(d7e499a0,d7e49a14) at sowakeup+0x51
sorwakeup(d7e499a0) at sorwakeup+0x56
divert_packet(d8ceb200,1,9a02) at divert_packet+0x254
pf_test(2,1,d8366830,f58023a0) at pf_test+0x873
ip_input_if(f58023a0,f580238c,f58023b8,0,d8366830) at ip_input_if+0xab
ipv4_input(d8366830,d8ceb200) at ipv4_input+0x3a
ether_input(d8366830,d8ceb200) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sbappendaddr(d7e49560,d7e495d4,f580215c,d918e000,0) at sbappendaddr+0x18
divert6_packet(d918e000,1,9a02) at divert6_packet+0x24e
pf_test(18,1,d8366830,f58023a0) at pf_test+0x929
ip6_input_if(f58023a0,f580238c,29,0,d8366830) at ip6_input_if+0x1bd
ipv6_input(d8366830,d918e000) at ipv6_input+0x2b
ether_input(d8366830,d918e000) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sbappendaddr(d7e49560,d7e495d4,f580215c,d918e000,0) at sbappendaddr+0x45
divert6_packet(d918e000,1,9a02) at divert6_packet+0x24e
pf_test(18,1,d8366830,f58023a0) at pf_test+0x929
ip6_input_if(f58023a0,f580238c,29,0,d8366830) at ip6_input_if+0x1bd
ipv6_input(d8366830,d918e000) at ipv6_input+0x2b
ether_input(d8366830,d918e000) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sorwakeup(d7e49560) at sorwakeup+0xd
divert6_packet(d918e000,1,9a02) at divert6_packet+0x26b
pf_test(18,1,d8366830,f58023a0) at pf_test+0x929
ip6_input_if(f58023a0,f580238c,29,0,d8366830) at ip6_input_if+0x1bd
ipv6_input(d8366830,d918e000) at ipv6_input+0x2b
ether_input(d8366830,d918e000) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sowakeup(d7e49560,d7e495d4) at sowakeup+0x11
sorwakeup(d7e49560) at sorwakeup+0x56
divert6_packet(d918e000,1,9a02) at divert6_packet+0x26b
pf_test(18,1,d8366830,f58023a0) at pf_test+0x929
ip6_input_if(f58023a0,f580238c,29,0,d8366830) at ip6_input_if+0x1bd
ipv6_input(d8366830,d918e000) at ipv6_input+0x2b
ether_input(d8366830,d918e000) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
knote_locked(d7e49608,0) at knote_locked+0x1a
sowakeup(d7e49560,d7e495d4) at sowakeup+0x51
sorwakeup(d7e49560) at sorwakeup+0x56
divert6_packet(d918e000,1,9a02) at divert6_packet+0x26b
pf_test(18,1,d8366830,f58023a0) at pf_test+0x929
ip6_input_if(f58023a0,f580238c,29,0,d8366830) at ip6_input_if+0x1bd
ipv6_input(d8366830,d918e000) at ipv6_input+0x2b
ether_input(d8366830,d918e000) at ether_input+0x2c4
if_input_process(d8366830,f58023f4) at if_input_process+0x4d
ifiq_process(d8366b00) at ifiq_process+0x66
taskq_thread(d82f1000) at taskq_thread+0x84
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sbappendaddr(d7e499a0,d7e49a14,f5a5df18,d8e76d00,0) at sbappendaddr+0x18
divert_packet(d8e76d00,2,9a02) at divert_packet+0x237
pf_test(2,2,d8366830,f5a5e0d0) at pf_test+0x873
ip_output(d8e76d00,0,d7e4a154,0,0,d7e4a1d8,0) at ip_output+0x5aa
udp_output(d7e4a108,d8e76d00,0,0) at udp_output+0x37a
udp_send(d7e49560,d8e76d00,0,0) at udp_send+0xbe
sosend(d7e49560,0,f5a5e1dc,d8e76d00,0,0) at sosend+0x2d8
sendit(d81099d8,1,f5a5e230,0,f5a5e2a8) at sendit+0x2cc
sys_sendto(d81099d8,f5a5e2b0,f5a5e2a8) at sys_sendto+0x5c
syscall(f5a5e2f0) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sbappendaddr(d7e499a0,d7e49a14,f5a5df18,d8e76d00,0) at sbappendaddr+0x45
divert_packet(d8e76d00,2,9a02) at divert_packet+0x237
pf_test(2,2,d8366830,f5a5e0d0) at pf_test+0x873
ip_output(d8e76d00,0,d7e4a154,0,0,d7e4a1d8,0) at ip_output+0x5aa
udp_output(d7e4a108,d8e76d00,0,0) at udp_output+0x37a
udp_send(d7e49560,d8e76d00,0,0) at udp_send+0xbe
sosend(d7e49560,0,f5a5e1dc,d8e76d00,0,0) at sosend+0x2d8
sendit(d81099d8,1,f5a5e230,0,f5a5e2a8) at sendit+0x2cc
sys_sendto(d81099d8,f5a5e2b0,f5a5e2a8) at sys_sendto+0x5c
syscall(f5a5e2f0) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sorwakeup(d7e499a0) at sorwakeup+0xd
divert_packet(d8e76d00,2,9a02) at divert_packet+0x254
pf_test(2,2,d8366830,f5a5e0d0) at pf_test+0x873
ip_output(d8e76d00,0,d7e4a154,0,0,d7e4a1d8,0) at ip_output+0x5aa
udp_output(d7e4a108,d8e76d00,0,0) at udp_output+0x37a
udp_send(d7e49560,d8e76d00,0,0) at udp_send+0xbe
sosend(d7e49560,0,f5a5e1dc,d8e76d00,0,0) at sosend+0x2d8
sendit(d81099d8,1,f5a5e230,0,f5a5e2a8) at sendit+0x2cc
sys_sendto(d81099d8,f5a5e2b0,f5a5e2a8) at sys_sendto+0x5c
syscall(f5a5e2f0) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
sowakeup(d7e499a0,d7e49a14) at sowakeup+0x11
sorwakeup(d7e499a0) at sorwakeup+0x56
divert_packet(d8e76d00,2,9a02) at divert_packet+0x254
pf_test(2,2,d8366830,f5a5e0d0) at pf_test+0x873
ip_output(d8e76d00,0,d7e4a154,0,0,d7e4a1d8,0) at ip_output+0x5aa
udp_output(d7e4a108,d8e76d00,0,0) at udp_output+0x37a
udp_send(d7e49560,d8e76d00,0,0) at udp_send+0xbe
sosend(d7e49560,0,f5a5e1dc,d8e76d00,0,0) at sosend+0x2d8
sendit(d81099d8,1,f5a5e230,0,f5a5e2a8) at sendit+0x2cc
sys_sendto(d81099d8,f5a5e2b0,f5a5e2a8) at sys_sendto+0x5c
syscall(f5a5e2f0) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e499a0) at soassertlocked+0xaa
knote_locked(d7e49a48,0) at knote_locked+0x1a
sowakeup(d7e499a0,d7e49a14) at sowakeup+0x51
sorwakeup(d7e499a0) at sorwakeup+0x56
divert_packet(d8e76d00,2,9a02) at divert_packet+0x254
pf_test(2,2,d8366830,f5a5e0d0) at pf_test+0x873
ip_output(d8e76d00,0,d7e4a154,0,0,d7e4a1d8,0) at ip_output+0x5aa
udp_output(d7e4a108,d8e76d00,0,0) at udp_output+0x37a
udp_send(d7e49560,d8e76d00,0,0) at udp_send+0xbe
sosend(d7e49560,0,f5a5e1dc,d8e76d00,0,0) at sosend+0x2d8
sendit(d81099d8,1,f5a5e230,0,f5a5e2a8) at sendit+0x2cc
sys_sendto(d81099d8,f5a5e2b0,f5a5e2a8) at sys_sendto+0x5c
syscall(f5a5e2f0) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sbappendaddr(d7e49560,d7e495d4,f5a5dc64,d8ca4400,0) at sbappendaddr+0x18
divert6_packet(d8ca4400,2,9a02) at divert6_packet+0x24e
pf_test(18,2,d8366830,f5a5de80) at pf_test+0x929
ip6_output(d8ca4400,0,d7e4a43c,0,0,d7e4a4c0) at ip6_output+0xc07
udp6_output(d7e4a3f0,d8ca4400,0,0) at udp6_output+0x3fa
udp_send(d7e499a0,d8ca4400,0,0) at udp_send+0xbe
sosend(d7e499a0,0,f5a5dfec,d8ca4400,0,0) at sosend+0x2d8
sendit(d7e419e0,1,f5a5e040,0,f5a5e0b8) at sendit+0x2cc
sys_sendto(d7e419e0,f5a5e0c0,f5a5e0b8) at sys_sendto+0x5c
syscall(f5a5e100) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sbappendaddr(d7e49560,d7e495d4,f5a5dc64,d8ca4400,0) at sbappendaddr+0x45
divert6_packet(d8ca4400,2,9a02) at divert6_packet+0x24e
pf_test(18,2,d8366830,f5a5de80) at pf_test+0x929
ip6_output(d8ca4400,0,d7e4a43c,0,0,d7e4a4c0) at ip6_output+0xc07
udp6_output(d7e4a3f0,d8ca4400,0,0) at udp6_output+0x3fa
udp_send(d7e499a0,d8ca4400,0,0) at udp_send+0xbe
sosend(d7e499a0,0,f5a5dfec,d8ca4400,0,0) at sosend+0x2d8
sendit(d7e419e0,1,f5a5e040,0,f5a5e0b8) at sendit+0x2cc
sys_sendto(d7e419e0,f5a5e0c0,f5a5e0b8) at sys_sendto+0x5c
syscall(f5a5e100) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sorwakeup(d7e49560) at sorwakeup+0xd
divert6_packet(d8ca4400,2,9a02) at divert6_packet+0x26b
pf_test(18,2,d8366830,f5a5de80) at pf_test+0x929
ip6_output(d8ca4400,0,d7e4a43c,0,0,d7e4a4c0) at ip6_output+0xc07
udp6_output(d7e4a3f0,d8ca4400,0,0) at udp6_output+0x3fa
udp_send(d7e499a0,d8ca4400,0,0) at udp_send+0xbe
sosend(d7e499a0,0,f5a5dfec,d8ca4400,0,0) at sosend+0x2d8
sendit(d7e419e0,1,f5a5e040,0,f5a5e0b8) at sendit+0x2cc
sys_sendto(d7e419e0,f5a5e0c0,f5a5e0b8) at sys_sendto+0x5c
syscall(f5a5e100) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
sowakeup(d7e49560,d7e495d4) at sowakeup+0x11
sorwakeup(d7e49560) at sorwakeup+0x56
divert6_packet(d8ca4400,2,9a02) at divert6_packet+0x26b
pf_test(18,2,d8366830,f5a5de80) at pf_test+0x929
ip6_output(d8ca4400,0,d7e4a43c,0,0,d7e4a4c0) at ip6_output+0xc07
udp6_output(d7e4a3f0,d8ca4400,0,0) at udp6_output+0x3fa
udp_send(d7e499a0,d8ca4400,0,0) at udp_send+0xbe
sosend(d7e499a0,0,f5a5dfec,d8ca4400,0,0) at sosend+0x2d8
sendit(d7e419e0,1,f5a5e040,0,f5a5e0b8) at sendit+0x2cc
sys_sendto(d7e419e0,f5a5e0c0,f5a5e0b8) at sys_sendto+0x5c
syscall(f5a5e100) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.
splassert: soassertlocked: want 0 have 1
Starting stack trace...
soassertlocked(0) at soassertlocked+0xaa
soassertlocked(d7e49560) at soassertlocked+0xaa
knote_locked(d7e49608,0) at knote_locked+0x1a
sowakeup(d7e49560,d7e495d4) at sowakeup+0x51
sorwakeup(d7e49560) at sorwakeup+0x56
divert6_packet(d8ca4400,2,9a02) at divert6_packet+0x26b
pf_test(18,2,d8366830,f5a5de80) at pf_test+0x929
ip6_output(d8ca4400,0,d7e4a43c,0,0,d7e4a4c0) at ip6_output+0xc07
udp6_output(d7e4a3f0,d8ca4400,0,0) at udp6_output+0x3fa
udp_send(d7e499a0,d8ca4400,0,0) at udp_send+0xbe
sosend(d7e499a0,0,f5a5dfec,d8ca4400,0,0) at sosend+0x2d8
sendit(d7e419e0,1,f5a5e040,0,f5a5e0b8) at sendit+0x2cc
sys_sendto(d7e419e0,f5a5e0c0,f5a5e0b8) at sys_sendto+0x5c
syscall(f5a5e100) at syscall+0x55e
Xsyscall_untramp() at Xsyscall_untramp+0xa9
end of kernel
End of stack trace.