[OpenAFS] Client running out of thread with high connection number

[Ximeng Guan]

Hello,

We have a heavily-used OpenAFS client (a design server) that has repeatedly shown a message of "Connection timed out" during peak hours in the past week.

When probed with rxdebug at port 7001, we find a high number of calls that had waited for a thread. (90 calls have waited for a thread, noBuffers 11, 2892 client connections)

According to the users, the loss of connection happens randomly and can last up to 10min. When the loss of connection happens, the server seems to be unreachable from outside (ping or ssh). But existing sessions still seem to respond to keyboard strokes from remote users.

The client is running CentOS 6.6, kernel 2.6.32-696.3.2.el6.x86_64 and OpenAFS 1.6.20.

I am suspecting that the problem may not be related to the client, but to the network service in general. But the returned message by rxdebug seems odd too.

Can someone help interpret this message for us?

Trying 172.16.101.82 (port 7001):
Free packets: 424/531, packet reclaims: 0, calls: 184990, used FDs: 64
not waiting for packets.
0 calls waiting for a thread
1 threads are idle
90 calls have waited for a thread
rx stats: free packets 424, allocs 2067108999, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 11, selects 0, sendSelects 0
packets read: data 341104352 ack 2316391759 busy 216 abort 736892 ackall 0 challenge 49571 response 0 debug 3432 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 313908875, ack 2281732446, dup 7217124 spurious 61698033 dally 161427
packets sent: data 1985111690 ack 187255106 busy 0 abort 173 ackall 0 challenge 0 response 49567 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 187255106, data 1580369647 (not resends), resends 404742043, pushed 0, acked&ignored 2852065309
(these should be small) sendFailed 1470595, fatalErrors 153
Average rtt is 0.000, with 1163176554 samples
Minimum rtt is 0.000, maximum is 2.396
3 server connections, 2892 client connections, 77 peer structs, 1127 call structs, 39 free call structs

We looked at the file servers at port 7000 and didn't see the same congestion. However, we do see a log pattern that seems to be related to the high number of connections from that same client:

Wed Sep 5 13:02:45 2018 FindClient: stillborn client 00007FE9680116E0(1cedd31c); conn 00007FE77C011060 (host 172.16.101.82:7001) had client 00007FE968013AA0(1cedd31c)
Wed Sep 5 13:12:53 2018 FindClient: stillborn client 00007FE97001E400(1cedd370); conn 00007FE97401BB70 (host 172.16.101.82:7001) had client 00007FE97001E0C0(1cedd370)
Wed Sep 5 13:18:56 2018 FindClient: stillborn client 00007FE98C02A7C0(1cedd394); conn 00007FE974032180 (host 172.16.101.82:7001) had client 00007FE968013010(1cedd394)
Wed Sep 5 13:18:57 2018 FindClient: stillborn client 00007FE76401D840(1cedd39c); conn 0000000000C0A590 (host 172.16.101.82:7001) had client 00007FE98C02A7C0(1cedd39c)
Wed Sep 5 13:26:36 2018 FindClient: stillborn client 00007FE970020F10(1cedd3e4); conn 0000000000C07800 (host 172.16.101.82:7001) had client 00007FE968012240(1cedd3e4)
Wed Sep 5 13:37:09 2018 FindClient: stillborn client 00007FE98C029FA0(1cedd4c0); conn 00007FE77C0121B0 (host 172.16.101.82:7001) had client 00007FE968013690(1cedd4c0)


Thank you!

Best regards,
========================================
Ximeng (Simon) Guan, Ph.D.
Associate Principal Engineer
Royole Corporation
========================================

[Jonathan Billings]

Post by Ximeng Guan
The client is running CentOS 6.6, kernel 2.6.32-696.3.2.el6.x86_64 and OpenAFS 1.6.20.

Do you experience this issue on a version of CentOS that is up to date?
The version of CentOS 6 that is actively maintained is CentOS 6.10, with a
kernel 2.6.32-754.3.5.el6.x86_64. Also, the latest version of the OpenAFS
1.6.x release is 1.6.22.3.

[Ximeng Guan]

No, we only experience the issue on this particular machine, which is unique in its OS and OpenAFS client versions. It is the first server we put in a production environment for design purposes. Our IT is still struggling with bringing up new servers (OpenAFS clients) to shift the workload off this one before we can update its OS.

We are coordinating a re-boot of this machine over the upcoming weekend. If the same issue is observed after the re-boot, I will send an update to this mail group.

Thanks for the information. We greatly appreciate your effort in maintaining the latest compiled clients for CentOS/RHEL 7. They saved us time!

Best regards,
Simon

From: openafs-info-***@openafs.org <openafs-info-***@openafs.org> On Behalf Of Jonathan Billings
Sent: Thursday, September 6, 2018 5:06 AM
To: openafs-***@openafs.org
Subject: Re: [OpenAFS] Client running out of thread with high connection number

On Wed, Sep 5, 2018 at 5:50 PM, Ximeng Guan <***@royole.com<mailto:***@royole.com>> wrote:
The client is running CentOS 6.6, kernel 2.6.32-696.3.2.el6.x86_64 and OpenAFS 1.6.20.

Do you experience this issue on a version of CentOS that is up to date? The version of CentOS 6 that is actively maintained is CentOS 6.10, with a kernel 2.6.32-754.3.5.el6.x86_64. Also, the latest version of the OpenAFS 1.6.x release is 1.6.22.3.