Discussion:
[OpenAFS] Red Hat EL Support Customers - Please open a support case for kafs in RHEL8
Jeffrey Altman
2018-12-06 23:33:10 UTC
Permalink
To all AuriStorFS licensees and OpenAFS users,

After more than seventeen years of development led by David Howells, the
Linux kernel now includes a production ready AFS/AuriStorFS client
(kafs) and RX RPC protocol implementation (AF_RXRPC)[1]. These are not
add-ons. kafs and af_rxrpc are baked into the mainline kernel.

Jonathan Billings has been building kafs enabled Fedora Core kernels via
COPR[1] since July[3]. Beginning with the 4.19 kernel, Fedora Core
kernels now ship with kafs and af_rxrpc enabled.

It was my hope that Red Hat would include kafs and af_rxrpc in RHEL8
Beta as an experimental technology. Unfortunately, the RHEL8 Beta
announced on Nov 15th did not include either.

Thankfully it is not too late to for kafs and af_rxrpc to be added
before a final release of RHEL8 but RHEL support customers must make the
case by registering for the RHEL8 Beta[4] and opening a Support Case[5].

When opening a support case please specify:

Product: Red Hat Enterprise Linux
Version: 8.0 Beta
Case Type: Feature / Enhancement Request
Hostname: hostname of the system on which RHEL8 beta was installed
Problem
Statement: Request inclusion of kafs
Case
Description; Explain why your organization uses AFS/AuriStorFS in
addition to RHEL support storage systems and how the
inclusion of kafs and af_rxrpc in RHEL8 will benefit
your organization.

It is very important that the Case Description be specific to your
organization. Each organization's reasons for using AFS/AuriStorFS are
different just as each organization's use cases are different.

If you are eligible, please attempt to open a support request by
December 11th. Although this is not a hard deadline, many individuals
will begin taking end of year vacation time the middle of next week.

Frequently Asked Questions:

1. Is the kafs kernel module a replacement for the OpenAFS and
AuriStorFS kernel modules?

The best way to think of kafs is that it is an alternative to the
AFS/AuriStorFS and OpenAFS clients. When kafs is provided as part of a
Linux distribution and it is enabled, it is not necessary to install any
of the OpenAFS or AuriStorFS client packages.

2. Is kafs compatible with IBM AFS, OpenAFS and AuriStorFS servers?

Yes. kafs is compatible with IBM AFS 3.6 and all versions of OpenAFS
and AuriStorFS servers.

3. Does enabling kafs and af_rxrpc in RHEL8 prevent installation of
AuriStorFS or OpenAFS clients?

No. Not only can the AuriStorFS kernel module be installed on Linux
kernels built with kafs and af_rxrpc but both AuriStorFS and kafs can be
enabled at the same time. Runtime choices have to be made to decide
which will service the /afs mount point, which will use port 7001/udp
for the callback service, etc. However, there is nothing that prevents
co-existence.

OpenAFS developers will need to ensure compatibility with the latest
RHEL kernels and build configurations. It is likely that minor coding
adjustments will need to be implemented.

4. How does kafs/af_rxrpc performance compare to OpenAFS and AuriStorFS
clients?

In many workflows the kafs/af_rxrpc client is faster and more efficient
than either OpenAFS or AuriStorFS clients. kafs/af_rxrpc are tightly
integrated into the Linux network stack and virtual file system layer.
There is significantly less overhead in the network stack (no double
buffering) and no global locks. Building the Linux kernel from source
in /afs with kafs is at least 30% faster than the AuriStorFS cache
manager without encryption.

5. Are there features that OpenAFS has that kafs does not?

Yes. kafs does not split horizon caching, it does not have an
equivalent of cache bypass, it does not implement any of the rxdebug or
xstat_cm statistics collection. Nor does it provide pioctls and there is
no fs, vos, pts, bos command suite. kafs does not export afs2nfs.

6. Are there features that AuriStorFS has that kafs does not?

In addition to the items mentioned in the prior answer, kafs does not
yet support the yfs-rxgk security class and aes256-cts-hmac-sha1-96 or
aes256-cts-hmac-sha512-384 encryption. O_DIRECT support is not yet
complete. kafs is a fairly complete AuriStorFS client including support
for IPv6 and AuriStorFS RPC suites.

7. Does kafs have capabilities that are implemented in neither OpenAFS
nor AuriStorS?

kafs auto-mounts each afs volume as a separate device instead of
treating the entire /afs file namespace as a single device. kafs
implements speculative file status fetching from directory lookups.

Perhaps more important is what kafs will be able to implement that
OpenAFS and AuriStorFS cannot due to GPL vs IPL10 license conflicts:

* inotify notification event generation
* SELinux/Smack label storage
* better container namespacing

8. If my organization is happy with OpenAFS and/or AuriStorFS clients,
why should my organization care?

Out-of-the-box Linux distribution support for accessing the /afs file
namespace will significantly simplify the lives of end users not to
mention system administrators and help desk support staff. When kafs is
distributed as part of the Linux kernel, there can never be a conflict
between the kernel version and the AFS kernel module since they are one
and the same. There can never be a delay between the availability of a
new kernel version and the matching OpenAFS or AuriStorFS kernel module.
AuriStor promises a new kernel module release within 48 hours of the
release of a kernel for a supported Linux distribution. With OpenAFS
there have been many circumstances when the delay has been measured in
weeks or months.

Organizations that have support contracts with Linux vendors are often
told that support does not apply when the Linux kernel has been tainted
by a third-party kernel module. When using kafs, the Linux kernel is
never tainted.

As part of the Linux kernel source tree, kafs is indirectly supported by
the entire Linux kernel development community. All of the automated
testing performed against the mainline kernel is also performed against
kafs. All kernel interface changes impacting kafs or af_rxrpc must be
implemented in kafs and af_rxrpc by the developer(s) promoting the
change. All-in-all kafs and af_rxrpc will receive reviews by a much
larger community of developers.

Finally, as an out-of-the-box solution, /afs becomes a first class file
system namespace. As a result, AFS adoption will increase and /afs will
become accessible on systems that are managed by third-party such as
those in the cloud.

9. Is there anything I shouldn't say to Red Hat?

Red Hat is going to make a business decision based upon its evaluation
of customer needs and their impact on growth of RHEL licensing. If I
were in their shoes I would not find a request to add support for kafs
compelling if it were combined with a statement that the requesting
organization intends to discontinue use of /afs within the next three to
five years. RHEL8 will have a support lifetime of at least a decade and
there is little justification to commit new engineering resources to a
technology that customers believe has no future.

10. Will AuriStor stop developing its own Linux client?

No. AuriStor will always be able to ship new functionality in its own
clients first. AuriStor believes that kafs will be the AFS client for
99.9% of end users with Linux desktops and servers. The AuriStorFS
client for Linux will be used by organizations that have special needs
and highly managed environments.


Thanks for your assistance on behalf of the entire AFS/AuriStorFS community.

Jeffrey Altman


[1] https://www.infradead.org/~dhowells/kafs/
[2] https://copr.fedorainfracloud.org/coprs/jsbillings/kafs/
[3] https://lists.openafs.org/pipermail/openafs-info/2018-July/042481.html
[4] https://developers.redhat.com/rhel8/getrhel8/
[5]
https://access.redhat.com/support/cases/#/case/new?intcmp=hp%7Ca%7Ca3%7Ccase
Harald Barth
2018-12-07 09:00:53 UTC
Permalink
Hi Jeff, hi David!

Has it been 17 years? Well, we are all getting - mature ;-)

Obviously a file system is ready for use if it's old enough to buy
liquor (which difffers a little between countries).
Post by Jeffrey Altman
Product: Red Hat Enterprise Linux
Version: 8.0 Beta
Case Type: Feature / Enhancement Request
Hostname: hostname of the system on which RHEL8 beta was installed
We have a hen and egg problem here: Why would I install 8.0b on
<whatever hostname> if it does not have kafs? Install a Fedora
test, sure, but RHEL 8.0b?
Post by Jeffrey Altman
If you are eligible, please attempt to open a support request by
December 11th.
3 workdays. Optimistic.
Post by Jeffrey Altman
As part of the Linux kernel source tree, kafs is indirectly supported by
the entire Linux kernel development community. All of the automated
testing performed against the mainline kernel is also performed against
kafs.
But the automated testing does probably not (yet) fetch a single file
from an AFS server. (Compare to how the gssapi-key-exchange features
in ssh are never tested in the ssh/scp shipped with distributions as
the testing never fetched a single file with that feature - with known
results). Testing that requires infrastructure is a lot of work to
automate.

Sorry, I may sound much more pessimistic here than I am actually are.
This _might_ fly. I wish.... :-)

Season's greetings,
Harald.
Jeffrey Altman
2018-12-08 03:34:14 UTC
Permalink
Post by Harald Barth
Hi Jeff, hi David!
Has it been 17 years? Well, we are all getting - mature ;-)
Obviously a file system is ready for use if it's old enough to buy
liquor (which difffers a little between countries).
Post by Jeffrey Altman
Product: Red Hat Enterprise Linux
Version: 8.0 Beta
Case Type: Feature / Enhancement Request
Hostname: hostname of the system on which RHEL8 beta was installed
We have a hen and egg problem here: Why would I install 8.0b on
<whatever hostname> if it does not have kafs? Install a Fedora
test, sure, but RHEL 8.0b?
You need to register for and install the RHEL 8.0 Beta to be eligible to
submit a feature / enhancement request.

RHEL 8.0 Beta does not include kafs. Hence the need to request its
inclusion as a feature request.
Post by Harald Barth
Post by Jeffrey Altman
If you are eligible, please attempt to open a support request by
December 11th.
3 workdays. Optimistic.
If you personally are unable to do so, then you won't.

However, others have already submitted feature requests in response to
this thread. To them the community owes thanks. The more requests that
are received the better.
Post by Harald Barth
Post by Jeffrey Altman
As part of the Linux kernel source tree, kafs is indirectly supported by
the entire Linux kernel development community. All of the automated
testing performed against the mainline kernel is also performed against
kafs.
But the automated testing does probably not (yet) fetch a single file
from an AFS server.
The automated testing performed by Red Hat doesn't as yet perform such
testing but AuriStor's testing does.
Post by Harald Barth
Testing that requires infrastructure is a lot of work to
automate.
Not true. At the 2015 AFSBPW Marc Dionne presented on AuriStor's docker
based infrastructure for automated testing of multi-server cells
including clients.

http://workshop.openafs.org/afsbpw15/talks/friday/dionne-docker.pdf

Since that time AuriStor has added live testing of kernel modules.

Each run currently performs approximately 7000 individual tests.
Post by Harald Barth
Sorry, I may sound much more pessimistic here than I am actually are.
This _might_ fly. I wish.... :-)
There is ample reason to be pessimistic Its really easy for Fedora
Core and Debian to ship kernels built with kafs and AF_RXRPC enabled
because there are no guarantees. I believe that if Red Hat enables kafs
in Enterprise Linux there are substantial costs and commitments
associated with that decision:

* Documentation of AFS and AuriStorFS capabilities and limitations
not only as a filesystem but with regards to interactions with
other Red Hat supported components.

* Training for system engineers.

* Integration of AFS and AuriStorFS support into other Red Hat
supported technologies

* Development of Certification and Training programs for partners.

* A ten year commitment to develop, maintain and fix kafs and AF_RXRPC

* Development and maintenance of test infrastructure.

The truth is that the costs associated with code development is a small
component of the total costs. As such Red Hat must be convinced that
inclusion of kafs and AF_RXRPC will provide functional capabilities to
end users that cannot be achieved via other Red Hat supported storage
technologies; and that supporting kafs and AF_RXRPC will provide a
long-term benefit to their bottom line.

That said, I believe a case can be made by the members of this
community. In 2001 Red Hat couldn't support AFS because of GPL vs IPL10
conflicts. Now that kafs is available, it becomes possible for Red Hat
to do so. Its up to all OpenAFS and AuriStorFS end user organizations
to make the case.

Good luck to all.

Jeffrey Altman
Dirk Heinrichs
2018-12-07 16:26:09 UTC
Permalink
Post by Jeffrey Altman
5. Are there features that OpenAFS has that kafs does not?
Yes. kafs does not split horizon caching, it does not have an
equivalent of cache bypass, it does not implement any of the rxdebug or
xstat_cm statistics collection. Nor does it provide pioctls and there is
no fs, vos, pts, bos command suite. kafs does not export afs2nfs.
What about PAM integration? Does pam-afs-session also work with kafs? Or
is there any other way for users to get access to their $HOME in /afs?

From the documentation inside the kernel tree I take it that there's
currently only a klog program, which needs to be invoked explicitly (so
AFTER the user has logged in). Or can it be used by said PAM module by
using its "program=path" configuration option (see pam_afs_session(5))?

Bye...

    Dirk
--
Dirk Heinrichs <***@altum.de>
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de
Jonathan Billings
2018-12-07 16:36:01 UTC
Permalink
On my systems, I install the kafs-client package (currently in COPR, but
eventually to be in Fedora 29) that includes a kafs-aware aklog package,
and use pam_exec to have it run aklog as part of the PAM stack. Here's the
source: http://git.infradead.org/users/dhowells/kafs-client.git

I append this to my PAM config, where I use pam_sss to get kerberos tickets
for UMICH.EDU.
session optional pam_exec.so quiet seteuid /usr/bin/aklog umich.edu

I've not tried getting pam-afs-session to work with the kafs version of
aklog. It does look like program=/path/to/kafs-aklog would work.
Post by Dirk Heinrichs
Post by Jeffrey Altman
5. Are there features that OpenAFS has that kafs does not?
Yes. kafs does not split horizon caching, it does not have an
equivalent of cache bypass, it does not implement any of the rxdebug or
xstat_cm statistics collection. Nor does it provide pioctls and there is
no fs, vos, pts, bos command suite. kafs does not export afs2nfs.
What about PAM integration? Does pam-afs-session also work with kafs? Or
is there any other way for users to get access to their $HOME in /afs?
From the documentation inside the kernel tree I take it that there's
currently only a klog program, which needs to be invoked explicitly (so
AFTER the user has logged in). Or can it be used by said PAM module by
using its "program=path" configuration option (see pam_afs_session(5))?
Bye...
Dirk
--
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
--
Jonathan Billings <***@umich.edu>
College of Engineering - CAEN - Unix and Linux Support
Dirk Heinrichs
2018-12-07 17:39:00 UTC
Permalink
Post by Jonathan Billings
On my systems, I install the kafs-client package (currently in COPR, but
eventually to be in Fedora 29) that includes a kafs-aware aklog package,
and use pam_exec to have it run aklog as part of the PAM stack. Here's the
source: http://git.infradead.org/users/dhowells/kafs-client.git
Nice. Wasn't aware of this.
Post by Jonathan Billings
I append this to my PAM config, where I use pam_sss to get kerberos tickets
for UMICH.EDU.
session optional pam_exec.so quiet seteuid /usr/bin/aklog umich.edu
Did a quick test (on Debian, btw., which already ships kafs) and it
works fine.
Post by Jonathan Billings
I've not tried getting pam-afs-session to work with the kafs version of
aklog. It does look like program=/path/to/kafs-aklog would work.
Turns out this module checks for the "traditional" AFS client, so it
doesn't work with kafs. Anyway, the pam_exec method makes for a good
workaround ;-)

Bye...

Dirk
--
Dirk Heinrichs <***@altum.de>
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de
Dirk Heinrichs
2018-12-08 10:21:00 UTC
Permalink
Post by Dirk Heinrichs
Did a quick test (on Debian, btw., which already ships kafs) and it
works fine.
While getting tokens at login work with this setup, things start to fail
once the users $HOME is set to be in /afs. While simple scenarios like
pure shell/console logins work, graphical desktop environments have lots
of problems. XFCE4 doesn't even start, Plasma works to some degree after
presenting lots of error dialogs to the user.

Seems there's still some work to do until this becomes an alternative
for the standard OpenAFS client.

So I wonder why RH customers would want that?

Bye...

Dirk
--
Dirk Heinrichs <***@altum.de>
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de
Harald Barth
2018-12-08 12:32:08 UTC
Permalink
Post by Dirk Heinrichs
While getting tokens at login work with this setup, things start to fail
once the users $HOME is set to be in /afs. While simple scenarios like
pure shell/console logins work, graphical desktop environments have lots
of problems. XFCE4 doesn't even start, Plasma works to some degree after
presenting lots of error dialogs to the user.
Is this a problem due to AFS or due to the startup of the graphical
environment which nowadays may involve systemd --user services instead
of running all processes in the same session?
Post by Dirk Heinrichs
Seems there's still some work to do until this becomes an alternative
for the standard OpenAFS client.
It may make a "beta", so yes.
Post by Dirk Heinrichs
So I wonder why RH customers would want that?
RH decided that their customers wanted systemd ;->

Harald.
Jeffrey Altman
2018-12-08 19:08:22 UTC
Permalink
Post by Dirk Heinrichs
Post by Dirk Heinrichs
Did a quick test (on Debian, btw., which already ships kafs) and it
works fine.
While getting tokens at login work with this setup, things start to fail
once the users $HOME is set to be in /afs. While simple scenarios like
pure shell/console logins work, graphical desktop environments have lots
of problems. XFCE4 doesn't even start, Plasma works to some degree after
presenting lots of error dialogs to the user.
As Harald indicated, "systemd --user" services are a problem not just
for kafs but for openafs as well. There has been discussions on this
mailing list of the issues dating back more than a year. In summary,
"systemd --user" services are incompatible with "session keyrings" which
are used to represent AFS Process Authentication Groups.

You have no indicated which kernel version you are using nor am I aware
of the options used to build AF_RXRPC and KAFS on Debian. The Linux
kernel versions that are recommended are 4.19 with a couple of back port
patches from the forthcoming 4.20 and the 4.20 release candidate series.

Regardless, it would be useful for you to file bug reports with the
Linux distribution describing the issues you are experiencing.

Debian: https://wiki.debian.org/reportbug

Fedora: https://fedoraproject.org/wiki/Bugs_and_feature_requests
Post by Dirk Heinrichs
Seems there's still some work to do until this becomes an alternative
for the standard OpenAFS client.
All software including OpenAFS has work to do. The kafs to-do list of
known work items is here:

https://www.infradead.org/~dhowells/kafs/todo.html
Post by Dirk Heinrichs
So I wonder why RH customers would want that?
Obviously, no one wants bugs, but at the same time this community does want:

1. A solution to "systemd --user" service compatibility with AFS.
The required changes are going to require Linux distribution
intervention because systemd is integrated with differences
to each distribution. At the moment there is no interest among
the systemd developers to work to fix a behavior they consider
to be a bug in OpenAFS, an out of tree file system.

2. The RHEL AFS user community needs an end to the repeated breakage
of /afs access following each RHEL dot release. How many times
has getcwd() broken because RHEL kernels updates preserve the API
between releases but do not preserve the ABI. While this permits
third party kernel modules to load it does not ensure that they
will do the right thing. If the community is lucky the symptoms
are visible. If unlucky, the symptoms are hidden until someone
reports silent data corruption.

3. Day zero support for new kernel releases. It took OpenAFS a month
to support 7.4 and two months to fix the breakage from 7.5.
There were also extensive delays in fixing OpenAFS to work with
5.10 and 6.5.

The need for an in-tree Linux AFS client extends to all Linux
distributions not just Red Hat. Any OpenAFS Linux developer can attest
to the extensive effort that must be expended to maintain compatibility
with the mainline Linux kernel. Then multiply that effort by all of the
Linux distributions that ship modified kernels such as RHEL, SuSE,
Ubuntu, Oracle, ....

RHEL 8 is in beta. The next opportunity to argue for inclusion of the
in-tree AFS client will be RHEL 9. The clock is ticking ....

Jeffrey Altman

Loading...