[OpenAFS] problems with ubuntu 18.04 client

[m***@desy.de]

Hi !

In the last days we're getting more and more messages from users about
having problems accessing their home-directories or subfolder or the hole
user-tree /afs/desy.de/user is missing ...

Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and 4.15.0-34-generic kernel ...
Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ...

Are any problems known here with this combination ?

Solution seems to be only a reboot of the system ... Unfortunatley i've
got not time to analyze one of these Desktops, because the user report
this problem afterwards of the reboot ...

Thanks & Cheers


Martin

[Benjamin Kaduk]

Post by m***@desy.de
Hi !
In the last days we're getting more and more messages from users about
having problems accessing their home-directories or subfolder or the hole
user-tree /afs/desy.de/user is missing ...
Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and 4.15.0-34-generic kernel ...
Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ...
Are any problems known here with this combination ?
Solution seems to be only a reboot of the system ... Unfortunatley i've
got not time to analyze one of these Desktops, because the user report
this problem afterwards of the reboot ...

The only thing that comes to mind would be the fix for broken volume
callbacks from https://gerrit.openafs.org/13140 , since
https://gerrit.openafs.org/12851 (already in 1.8.0pre5) was supposed to
take care of the main source of spurious ENOENTs.

Any chance you could try a 1.8.1 or newer (e.g., the 1.8.2 available at
https://launchpad.net/~openafs/+archive/ubuntu/stable)?

Thanks,

Ben

[Andreas Ladanyi]

Hi Martin,

we had the same problems.

We are using the 1.6 release from ppa ( https://launchpad.net/~openafs/%2Barchive/ubuntu/stable ) on server and client now and there seems to be no problems anymore.

1.8 pre on client from default ubuntu 18.04 repo and if i remember correctly 1.8.0 from paa on the client was a problem.

And yes after rebooting a 1.8 client the problem has gone.


Andy

Post by m***@desy.de
Hi !
In the last days we're getting more and more messages from users about
having problems accessing their home-directories or subfolder or the
hole user-tree /afs/desy.de/user is missing ...
Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and
4.15.0-34-generic kernel ...
Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ...
Are any problems known here with this combination ?
Solution seems to be only a reboot of the system ... Unfortunatley
i've got not time to analyze one of these Desktops, because the user
report this problem afterwards of the reboot ...
Thanks & Cheers
       Martin
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info

--
Karlsruher Institut für Technologie (KIT)
Fakultät für Informatik
ATIS – Abteilung Technische Infrastruktur

Dipl.-Ing. Andreas Ladanyi
- Systemadministrator -

Am Fasanengarten 5, Gebäude 50.34, Raum 013
76131 Karlsruhe

Telefon: [phone] - 4 3663
Fax: [phone] - 4 6699
E-Mail: ***@kit.edu
www.atis.informatik.kit.edu

www.kit.edu

KIT - Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft

Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.

[Gaja Sophie Peters]

Post by Andreas Ladanyi
we had the same problems.
We are using the 1.6 release from ppa (
https://launchpad.net/~openafs/%2Barchive/ubuntu/stable ) on server
and client now and there seems to be no problems anymore.

We were probably just lucky, or the packages from the 1.8 ppa
http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem.

At the moment, we live with all clients and all servers at 1.8.2 and
(crossing fingers and knocking wood) everything seems to work.


The one problem we occasionally have (which has nothing to do with the
above) is that when people force a shutdown or reboot at an inopportune
moment during an OpenAFS update, the AFS is entirely lost (the old
DKMS-package deleted, the new not yet built), but the system thinks, "oh
the home-directory should be /afs/math.uni-hamburg.de/users/../... -
let's just create that directory and the entire tree".

In that case, a simple rebuild of the DKMS-module and "service
openafs-client force-start" fixes the problem however.


Greetings,
Gaja Peters

[Andreas Ladanyi]

Post by Gaja Sophie Peters
We were probably just lucky, or the packages from the 1.8 ppa
http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem.

Did you use 1.8.0 from ppa for the clients in the past or did you start
at 1.8.2 when switching from 1.6 release ?

Post by Gaja Sophie Peters
Greetings,
Gaja Peters

cheers,

Andy

[Martin Flemming]

Thanks all for input !

We will try at first upgrade to

https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2)

and see ... if there also problems then we'll report it and switch
maybe to an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23)

thanks & cheers

Martin

[Martin Flemming]

Hi, again !

Shortly after i send this mail to the list, one of the user report
back ... same problemes like before ... :-(

In an old terminal (where afs was running well) everyhing seems to be ok, create files,folder, pwd... etc)
but for every new one terminal or GUI-Application the AFS-Tree /afs/desy.de/user/ is not available anymore

Other directories in /afs/desy.de/ are available ...



My first idea: I've to delete the afs-cache, after the upgrade from 1.8.0 ~ pre5-1 to 1.8.2, haven't i ?


in dmesg we see follow messages

[42798.792549] audit: type=1400 audit(1538639477.910:42): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.792552] audit: type=1400 audit(1538639477.910:43): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.795332] audit: type=1400 audit(1538639477.914:44): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.795334] audit: type=1400 audit(1538639477.914:45): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42811.588120] audit: type=1400 audit(1538639490.705:46): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D71/V146768" pid=29664 comm="EvJobScheduler" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0


Any further suggestions for debugging ?

thanks & cheers,

martin

Post by Martin Flemming
Thanks all for input !
We will try at first upgrade to
https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2)
and see ... if there also problems then we'll report it and switch maybe to
an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23)
thanks & cheers
Martin

[Andreas Ladanyi]

Hi Martin,

Post by Martin Flemming
Hi, again !
Shortly after i send this mail to the list, one of the user report
back ... same problemes like before ... :-(
In an old  terminal (where afs was running well) everyhing seems to be
ok, create files,folder, pwd... etc) but for  every new one terminal
or GUI-Application the  AFS-Tree /afs/desy.de/user/  is not available
anymore

are the volumes from the users online or offline ?

If the volumes are offline could you salvage (bos solvage ...) the
volumes which result in the volumes go online and later the volumes go
offline again ?

Post by Martin Flemming
Other directories in /afs/desy.de/ are available ...

cheers,

Andy

[Martin Flemming]

Post by Andreas Ladanyi
Hi Martin,

Post by Martin Flemming
Hi, again !
Shortly after i send this mail to the list, one of the user report
back ... same problemes like before ... :-(
In an old  terminal (where afs was running well) everyhing seems to be
ok, create files,folder, pwd... etc) but for  every new one terminal
or GUI-Application the  AFS-Tree /afs/desy.de/user/  is not available
anymore

are the volumes from the users online or offline ?
If the volumes are offline could you salvage (bos solvage ...) the
volumes which result in the volumes go online and later the volumes go
offline again ?

Post by Martin Flemming
Other directories in /afs/desy.de/ are available ...

The Volumes are still online, from any other Computer the hole tree
/afs/desy.de/user/ is available and also the user-Volume itself ...

cheers,
martin

[Harald Barth]

In an old terminal (where afs was running well) everyhing seems to be
ok, create files,folder, pwd... etc)

Are you in the same or different PAG?

Harald.

[Martin Flemming]

Post by Harald Barth

In an old terminal (where afs was running well) everyhing seems to be
ok, create files,folder, pwd... etc)

Are you in the same or different PAG?

Hmm, i think that's not the reason,
if i login into the same Computer, the tree /afs/desy.de/user is also missing for me ...

martin

[Andreas Ladanyi]

Hi,

Post by Martin Flemming
if i login into the same Computer, the tree /afs/desy.de/user is also missing for me ...

Does a reboot solve the issue ?

Did you use ubuntu 18.04 and afs 1.6 before switching to afs 1.8 ?

Are there issues from volumes in the salvager log  ?


regards,

Andy

[Harald Barth]

Post by Martin Flemming

Post by Harald Barth
Are you in the same or different PAG?

Hmm, i think that's not the reason,
if i login into the same Computer, the tree /afs/desy.de/user is also missing for me ...

Yes, but a new login probably gives you a new pag and a new security
context. Question is if the old pag and security context still
continues to work and if the answer is yes why in that case that
differs from a new pag.

Any complaints in the file server log about the client in question?

Harald.

[Benjamin Kaduk]

You need to update your apparmor policy to allow rw access to
/var/cache/openafs/**; accesses are performed by the kernel cache manager
on behalf of all processes and apparmor's view of the credentials do not
line up. MIT's configuration does this as of
https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7

-Ben

Post by Martin Flemming
Hi, again !
Shortly after i send this mail to the list, one of the user report
back ... same problemes like before ... :-(
In an old terminal (where afs was running well) everyhing seems to be ok, create files,folder, pwd... etc)
but for every new one terminal or GUI-Application the AFS-Tree /afs/desy.de/user/ is not available anymore
Other directories in /afs/desy.de/ are available ...
My first idea: I've to delete the afs-cache, after the upgrade from 1.8.0 ~ pre5-1 to 1.8.2, haven't i ?
in dmesg we see follow messages
[42798.792549] audit: type=1400 audit(1538639477.910:42): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.792552] audit: type=1400 audit(1538639477.910:43): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.795332] audit: type=1400 audit(1538639477.914:44): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42798.795334] audit: type=1400 audit(1538639477.914:45): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
[42811.588120] audit: type=1400 audit(1538639490.705:46): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D71/V146768" pid=29664 comm="EvJobScheduler" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0
Any further suggestions for debugging ?
thanks & cheers,
martin

Post by Martin Flemming
Thanks all for input !
We will try at first upgrade to
https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2)
and see ... if there also problems then we'll report it and switch maybe to
an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23)
thanks & cheers
Martin

_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info

[Andreas Ladanyi]

Post by Benjamin Kaduk
You need to update your apparmor policy to allow rw access to
/var/cache/openafs/**; accesses are performed by the kernel cache manager
on behalf of all processes and apparmor's view of the credentials do not
line up. MIT's configuration does this as of
https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7
-Ben

Is the openafs bug which is workaround by the apparmor config since
release 1.8 or also in 1.6 ?

The date of the commit is from february this year. AFS 1.8 was released
at april this year.


Andy

[Benjamin Kaduk]

Post by Andreas Ladanyi

Post by Benjamin Kaduk
You need to update your apparmor policy to allow rw access to
/var/cache/openafs/**; accesses are performed by the kernel cache manager
on behalf of all processes and apparmor's view of the credentials do not
line up. MIT's configuration does this as of
https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7
-Ben

Is the openafs bug which is workaround by the apparmor config since
release 1.8 or also in 1.6 ?
The date of the commit is from february this year. AFS 1.8 was released
at april this year.

The details of the openafs bug are not known (otherwise it would be fixed
already). It may be new in 1.8 -- January/February was when we started
introducing 1.8.0 prereleases into Debian/Ubuntu, so the apparmor issues
may have been new at that time.

-Ben

[Martin Flemming]

Hi !

Thanks all for input and assistance !

For now i want to say all-clear ...

The situation is not so dramaticly as i thought,
We've got 3 of about 20 ubuntu 18.04 machines,
on which this problem appears ... but all 3 machines comes up with
this problem in 2 days and i want rather be active early than late ;-)

two of them got only once, and one,
has got this problem multiple times ....

In this case i've done the upgrade to 1.8.2 ...
After it appears again, i've deleted the cache total with "rm -rf" and
start the client again ... till now the user hasn't contacted us again .. :-)


Only short comment to the hint of Harald in terms of PAG

i think, it won't be a PAG problem, because the user couldn't see
/afs/desy.de/user and at the same time i'm as well, for me it seems to
be a client-problem not a PAG .... and nothing suspicious in the fileserver log ...

in terms to Andreas

Post by Andreas Ladanyi
Does a reboot solve the issue ?

Yes

Post by Andreas Ladanyi
Did you use ubuntu 18.04 and afs 1.6 before switching to afs 1.8 ?

No, we use the ubuntu openafs-client '1.8.0 ~ pre5-1' out of the box

Post by Andreas Ladanyi
Are there issues from volumes in the salvager log  ?

No messages in salvager logs


and in terms to Benjamin

Thanks for the hint with apparmor policy, maybe we need this sometime ;-)


All the best and i wll report again, if Problems comes again

thanks & cheers,

martin

Post by Andreas Ladanyi

Post by Andreas Ladanyi

Post by Benjamin Kaduk
You need to update your apparmor policy to allow rw access to
/var/cache/openafs/**; accesses are performed by the kernel cache manager
on behalf of all processes and apparmor's view of the credentials do not
line up. MIT's configuration does this as of
https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7
-Ben

Is the openafs bug which is workaround by the apparmor config since
release 1.8 or also in 1.6 ?
The date of the commit is from february this year. AFS 1.8 was released
at april this year.

The details of the openafs bug are not known (otherwise it would be fixed
already). It may be new in 1.8 -- January/February was when we started
introducing 1.8.0 prereleases into Debian/Ubuntu, so the apparmor issues
may have been new at that time.
-Ben
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info

[Gaja Sophie Peters]

Post by Andreas Ladanyi

Post by Gaja Sophie Peters
We were probably just lucky, or the packages from the 1.8 ppa
http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem.

Did you use 1.8.0 from ppa for the clients in the past or did you start
at 1.8.2 when switching from 1.6 release ?

Apologies for the late answer. We had to switch to the PPA ages ago
(since some time in 2013), since the regular Ubuntu DKMS-module made
occasionally troubles with Ubuntu kernel updates. So the switch for us
was at the same time when 1.8 appeared in the PPA. Looking back, it was
on the 25th of April 2018 - we had to change our "apt-get update" line
to something like this, before it worked, though

apt-get --allow-releaseinfo-change-label update

Without that, Ubuntu 18.04 wouldn't accept the name-change of 'OpenAFS
1.6.x stable releases' to 'OpenAFS stable releases (1.8.x)' (Ubuntu
14.04 and 16.04 accepted it without that)

Greetings,
Gaja Peters