Discussion:
a noobs question and problems on a new cell
(too old to reply)
Adnoh
2007-05-04 09:44:56 UTC
Permalink
hello @all
maybe stupid question but I don't know it :-)

i have two fileservers connected over a 2 mbit vpn (fs1 with rw-volume and
fs2 with a ro-instance) and mounted that on /afs/domain/vol
can I settup this that way:
my users on fs1 work with /afs/domain/vol - rw + ro - thats clear to me.
when my users on fs2 read a file it should be taken from the ro-instance on
the internal harddrive and if they change anything it is written to
fs1-rw-instance !? on my try i get only the error "readonly filesystem"

can this be done or do i have to setup a mountpoint /afs/domain/.vol (-rw)
and point the users to that directory !?

thanks for any responses to help me getting started
--
View this message in context: http://www.nabble.com/a-noobs-question-and-problems-on-a-new-cell-tf3691140.html#a10319740
Sent from the OpenAFS - General mailing list archive at Nabble.com.
Adnoh
2007-05-04 10:24:15 UTC
Permalink
hello @all
maybe stupid question but I don't know it :-)

i have two fileservers connected over a 2 mbit vpn (fs1 with rw-volume
and fs2 with a ro-instance) and mounted that on /afs/domain/vol
can I settup this that way:
my users on fs1 work with /afs/domain/vol - rw + ro - thats clear to me.
when my users on fs2 read a file it should be taken from the ro-instance
on the internal harddrive and if they change anything it is written to
fs1-rw-instance !?
on my try i get only the error "readonly filesystem"

can this be done or do i have to setup a mountpoint /afs/domain/.vol
(-rw) and point the users to that directory !?

thanks for any responses to help me getting started
Jason Edgecombe
2007-05-04 12:53:19 UTC
Permalink
Post by Adnoh
maybe stupid question but I don't know it :-)
i have two fileservers connected over a 2 mbit vpn (fs1 with rw-volume and
fs2 with a ro-instance) and mounted that on /afs/domain/vol
my users on fs1 work with /afs/domain/vol - rw + ro - thats clear to me.
when my users on fs2 read a file it should be taken from the ro-instance on
the internal harddrive and if they change anything it is written to
fs1-rw-instance !? on my try i get only the error "readonly filesystem"
can this be done or do i have to setup a mountpoint /afs/domain/.vol (-rw)
and point the users to that directory !?
thanks for any responses to help me getting started
The typical scenario is as follows:

/afs/domain/volume is the read-only copy if it exists.
/afs/.domain/volume is always the read-write copy. <-- note the
/afs/<dot>domain This is referred to at the "dotted" path.

writing to the read-only path of /afs/domain/volume will get the error
"read-only filesystem". To write, you must use the path /afs/.domain/volume

If you are doing lots of writes, then having the volume replicated may
not be the best thing.

What is the usage model for this volume?

Jason
Thomas Kula
2007-05-04 13:27:47 UTC
Permalink
Post by Jason Edgecombe
/afs/domain/volume is the read-only copy if it exists.
/afs/.domain/volume is always the read-write copy. <-- note the
/afs/<dot>domain This is referred to at the "dotted" path.
writing to the read-only path of /afs/domain/volume will get the error
"read-only filesystem". To write, you must use the path /afs/.domain/volume
Remember that this is just convention --- it's a very highly
recommended one, I think, and every place I've touched AFS
uses this or some variant of it. What makes this work is
the fact that /afs/.cellname is the mount point explicitly
for the R/W version of root.cell for <cellname>, and going
through that gets you the R/W versions of volumes --- the
semantics for when the cache manager prefers R/O volumes
and when it prefers R/W volumes has been sent out before
on this list.
--
Thomas L. Kula | ***@tproa.net | http://kula.tproa.net/
Adnoh
2007-05-04 18:34:36 UTC
Permalink
ok, I try to explain.

we have several districts connected to our headquarter over a vpn.
In every district we have ~50-150 windows-clients and a samba server ( all
authenticating against a Win2003 Server with ADS)
for their filestorage with an usb-hdd attached for backup purpose.
In our headquarter we would have enough space to hold all the company data
and backup.
so i would have something like
/afs/domain/dist1/data
/afs/domain/dist2/data
/afs/domain/dist3/data
/afs/domain/shared/data
where every district has it's own data folder and one volume where all the
districts work together in.
the backup should be done on a central place (our headquarter).

so if somebody can explain me how you would build this scenario it would be
really great for me to get started ;-)))

thanks
Post by Jason Edgecombe
Post by Adnoh
maybe stupid question but I don't know it :-)
i have two fileservers connected over a 2 mbit vpn (fs1 with rw-volume and
fs2 with a ro-instance) and mounted that on /afs/domain/vol
my users on fs1 work with /afs/domain/vol - rw + ro - thats clear to me.
when my users on fs2 read a file it should be taken from the ro-instance on
the internal harddrive and if they change anything it is written to
fs1-rw-instance !? on my try i get only the error "readonly filesystem"
can this be done or do i have to setup a mountpoint /afs/domain/.vol (-rw)
and point the users to that directory !?
thanks for any responses to help me getting started
/afs/domain/volume is the read-only copy if it exists.
/afs/.domain/volume is always the read-write copy. <-- note the
/afs/<dot>domain This is referred to at the "dotted" path.
writing to the read-only path of /afs/domain/volume will get the error
"read-only filesystem". To write, you must use the path
/afs/.domain/volume
If you are doing lots of writes, then having the volume replicated may
not be the best thing.
What is the usage model for this volume?
Jason
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
--
View this message in context: http://www.nabble.com/a-noobs-question-and-problems-on-a-new-cell-tf3691140.html#a10327886
Sent from the OpenAFS - General mailing list archive at Nabble.com.
Christopher D. Clausen
2007-05-04 18:55:37 UTC
Permalink
Post by Adnoh
ok, I try to explain.
we have several districts connected to our headquarter over a vpn.
In every district we have ~50-150 windows-clients and a samba server
( all authenticating against a Win2003 Server with ADS)
for their filestorage with an usb-hdd attached for backup purpose.
In our headquarter we would have enough space to hold all the company
data and backup.
so i would have something like
/afs/domain/dist1/data
/afs/domain/dist2/data
/afs/domain/dist3/data
/afs/domain/shared/data
where every district has it's own data folder and one volume where
all the districts work together in.
the backup should be done on a central place (our headquarter).
What do you mean by "backup" ?
Post by Adnoh
so if somebody can explain me how you would build this scenario it
would be really great for me to get started ;-)))
As you said, just create volumes, set quotas and mount them and backup
data to the appropriate central location.

If you want to actually have seperate fileservers at the external site
and use AFS volume replication, well, that would take some planning and
you'd need to provide more information to have a good solution defined.

-----

I suspect that you may want to actually pay someone to design a setup
for you if you do not have the expertise to design it yourself.

<<CDC
Adnoh
2007-05-07 06:23:14 UTC
Permalink
By "Backup" I mean the typical type of backup - one snapshot of the data at
the weekend and some
incremental backups every day. i don't think this is posible with backup
volumes - so i thought about volume dumps.

i don't want to have all the volumes in our headquarter. so every time a
user openes his word-doc or similar it would be completly transfered over
our VPN - and I can hear the people crying "our fileservers are too slow !"
so seperate fileservers in every district would be a good choice, I think -
would'nt they ?

i thought about every district his own fileserver with the special volume
for them and a readonly volume in our headquarter released every night - so
i could do the volume dump - i'm not very trusted with the "backup" command
yet.
but I don't know how I could set up the "shared" part where every user in
every district can read/write to it.

so if you tell me which information you need to know I can provide :-))
Post by Christopher D. Clausen
What do you mean by "backup" ?
As you said, just create volumes, set quotas and mount them and backup
data to the appropriate central location.
If you want to actually have seperate fileservers at the external site
and use AFS volume replication, well, that would take some planning and
you'd need to provide more information to have a good solution defined.
--
View this message in context: http://www.nabble.com/a-noobs-question-and-problems-on-a-new-cell-tf3691140.html#a10352620
Sent from the OpenAFS - General mailing list archive at Nabble.com.
Christopher D. Clausen
2007-05-07 07:29:41 UTC
Permalink
Post by Adnoh
By "Backup" I mean the typical type of backup - one snapshot of the
data at the weekend and some
incremental backups every day. i don't think this is posible with
backup volumes - so i thought about volume dumps.
Backup volumes must reside on the same server and partition as the RW
volume. You could use backup volumes, dump them, and copy the dump
files to a central location, even into a different AFS volume on another
server. That isn't the most efficient backup program though.
Post by Adnoh
i don't want to have all the volumes in our headquarter. so every
time a user openes his word-doc or similar it would be completly
transfered over our VPN - and I can hear the people crying "our
fileservers are too slow !" so seperate fileservers in every district
would be a good choice, I think - would'nt they ?
That is an option. There are of course problems with doing either.
Remember that the AFS clients themselves cache read-only data. So if
most of your data is only being read and not written back that often, it
might make sense to have only centrally located AFS servers.

It might also make sense to have AFS DB servers hosed locally at each
district, although I'm not sure what would happen if the network goes
out and quorum is lost. Otherwise certain information will still need
to come from the central DB server in order for AFS to function
properly.
Post by Adnoh
i thought about every district his own fileserver with the special
volume for them and a readonly volume in our headquarter released
every night - so i could do the volume dump - i'm not very trusted
with the "backup" command yet.
By default, the AFS client prefers to use readonly volumes, so if you
create a replica of a volume, the data will immediately become readonly.
You can however manualy force the mount point to be RW (-rw option to fs
mkm) and this way you can have an RW volume in each local district and
still be able to clone the data to other servers using vos release. All
volume rights must go to directly to the RW volume. The AFS client does
not detect when you want to make a write and find the proper RW volume.
You can modify the code to make it behave that way, but there are
reasons for not doing that.

You could also run an AFS cell in each district and vos dump
incrementals from the volumes and copy them to a central fileserver and
restore them there. Volumes do not have to be restored to the cell they
were dumped from.

However, you might simply be better off using a more common network
filesystem like NFS or samba and using something like rsync to backup
the data nightly. You mentioned a VPN. Since the network link is
already encrypted, you don't require filesystem encryption? Or do you?

I've not used the AFS "backup" command. Ever.
Post by Adnoh
but I don't know how I could set up the "shared" part where every
user in every district can read/write to it.
You just set ACLs with the fs setacl command. Unless I again am
misunderstanding what you mean by "shared."
Post by Adnoh
so if you tell me which information you need to know I can provide
You might want to read through:
http://www.dementia.org/twiki/pub/AFSLore/FurtherReading/NFS_on_steroids
and
http://reports-archive.adm.cs.cmu.edu/anon/home/anon/itc/CMU-ITC-053.pdf

Those are old, but are short and explain how volumes work. It seems as
though you are trying to use AFS like NFS or samba, creating a single
large share point and allowing everyone to write in it. This is not the
best way to use AFS, although it mostly works. Replicating single large
volumes can take a long time, especially over slow links.

Can you describe a "distrcit office" in more detail? How many users?
Is there technical staff there to diagnose problems with an AFS server,
if they occur? Are the offices always connected to the network? What
type of connection do they have? Bandwidth? Latency? Do users work
out of different offices at different times? How much data do you need
to store at each district? Do you use Kerberos 5 currently within your
organization? A single realm? Or a realm per district? What kind of
budget do you have for hardware and software for this project? How
reliable is the network link? Do you have any off-site backup or
disaster recovery requirements? Any specific features that the project
MUST do? Any features that the project SHOULD do? Anything else that
would be nice to do? How much data are we talking about here? Total
and at each district? What is the "change rate" of your data? How much
data is modified per day or per week as a percentage of the total data?
What are your projected storage requirements for 1 year? 2 years? 3
years? 5 years? 10 years? What are you using right now for file
sharing? What are the current problems that you are experiencing?

Why did you decide to look at AFS in the first place?

<<CDC
Adnoh
2007-05-09 08:55:11 UTC
Permalink
Hy Christopher,
first - thanks for your responses and your patience.
second - sorry for the long time I need to respond. I'm out of office this
week.
I write my answers direkt into your text below.
Post by Christopher D. Clausen
Post by Adnoh
i don't want to have all the volumes in our headquarter. so every
time a user openes his word-doc or similar it would be completly
transfered over our VPN - and I can hear the people crying "our
fileservers are too slow !" so seperate fileservers in every district
would be a good choice, I think - would'nt they ?
That is an option. There are of course problems with doing either.
Remember that the AFS clients themselves cache read-only data. So if
most of your data is only being read and not written back that often, it
might make sense to have only centrally located AFS servers.
thats right - but my problem at the moment is that we have only
windows-workstations. And I did'nt figure out how
I could customize the MSI-installation in that way, so I don't need to
travel to all our restricts and configure that client.
so I would like one afs "client" per district - the fileserver which is
already there (a linux gentoo machine) - some kind of afs->samba-gateway
Post by Christopher D. Clausen
By default, the AFS client prefers to use readonly volumes, so if you
create a replica of a volume, the data will immediately become readonly.
You can however manualy force the mount point to be RW (-rw option to fs
mkm) and this way you can have an RW volume in each local district and
still be able to clone the data to other servers using vos release. All
volume rights must go to directly to the RW volume. The AFS client does
not detect when you want to make a write and find the proper RW volume.
You can modify the code to make it behave that way, but there are
reasons for not doing that.
I tried that this way and didn't get it:
a volume called software (~1 Gig)
in our headquarter the rw-volume on the afs server.
in a district the (nightly) ro-snapshot of that volume.
mounted into afs like:
/afs/domain/.software (-rw)
/afs/domain/software (ro)
so if I understand that right i should now be able to access the data under
/afs/domain/.software on both sides.
in the headquarter it should use always the rw-instance and in the district
it should use the rw-instance (over vpn) on a write,
and on a read it should prefer the local ro-instance. but that doesn't work
for me.
everytime I accessed some software in the district it was transfered
completly over the vpn from our headquarter.
did I something missunderstood or have I done something wrong !?

the idea of this behaviour (take the lokal ro if available and just get what
you still need over vpn) was the coolest feature of the afs - i thougt. and
is the most case why I was looking on the whole afs thing - and not
something like nfs.
Post by Christopher D. Clausen
However, you might simply be better off using a more common network
filesystem like NFS or samba and using something like rsync to backup
the data nightly. You mentioned a VPN. Since the network link is
already encrypted, you don't require filesystem encryption? Or do you?
I'm not shure of the encryption ting. the vpn is a line from a large
provider in germany. so I think the line is secure, but I'm a little
bit paranoide ;-)
Post by Christopher D. Clausen
It seems as though you are trying to use AFS like NFS or samba, creating a
single
large share point and allowing everyone to write in it. This is not the
best way to use AFS, although it mostly works. Replicating single large
volumes can take a long time, especially over slow links.
yes and no. we have our samba-fileservers in every district completely
seperated from each other.
so if user a from district a wants to give a file to user b from district b
for working on it - he uses email. when
user b has his work completed on that file he uses that way to get the file
back to user a - and if someone in district
a has altered the file in that time - they have a problem...
so yes, i would like one big namespace - something like
/afs/domain/data/it
/controlling
/bookkeeping
and so on - so every user in a organisation unit can access his data from
each district he is at the moment and easilly share that to someone else who
is maybe not in the same district.
i thougt this is something afs wants me to give.

Can you describe a "distrcit office" in more detail? How many users?
->This differs - lets say 10 districts, 5 with ~100 users, 60 Gig of data
and a "data-change" of 100MB / Day
and the other 5 with the half of the above.

Is there technical staff there to diagnose problems with an AFS server, if
they occur? Are the offices always connected to the network? What type of
connection do they have? Bandwidth? Latency?
->no - the only technical staff is in our headquarter. we have a vpn from a
large provider which has a offline-time of maybe 10 Min / Year at all - so
it is very goot. The Bandwith differs - from 512k - 2Mbit. they are
connected 24h / day.

Do you use Kerberos 5 currently within your organization? A single realm?
Or a realm per district?
->We use a windows 2003 ADS for authentications of the windows workstations
and the samba-servers.

Do you have any off-site backup or disaster recovery requirements?
->I would like to have a backup on the local usb-hdd in each district and a
centraliced backup in our headquarter with a fullbackup/week and
diff-backup/day.

Any specific features that the project MUST do? Any features that the
project SHOULD do? Anything else that
would be nice to do?
-> yes - that what I have mentioned above ;-) - the "global" namespace
would be nice. maybe it is
interesting to tell you that we wanne migrate the workstations to linux in
the next 2-3 years.

How much data are we talking about here? Total and at each district? What
is the "change rate" of your data? How much
data is modified per day or per week as a percentage of the total data?
->mentioned above - all together, maybe ~ 500 Gig at the moment - but I
don't know how much duplicate data is there arround - you now that "i need
my files in every district, my local hdd and for best on my usb again" ;-)
--
View this message in context: http://www.nabble.com/a-noobs-question-and-problems-on-a-new-cell-tf3691140.html#a10390647
Sent from the OpenAFS - General mailing list archive at Nabble.com.
Klaas Hagemann
2007-05-09 11:39:02 UTC
Permalink
<snip>
Post by Adnoh
a volume called software (~1 Gig)
in our headquarter the rw-volume on the afs server.
in a district the (nightly) ro-snapshot of that volume.
/afs/domain/.software (-rw)
/afs/domain/software (ro)
so if I understand that right i should now be able to access the data under
/afs/domain/.software on both sides.
That is right, but you will always get the rw-instance in your headquarter.
Post by Adnoh
in the headquarter it should use always the rw-instance and in the district
it should use the rw-instance (over vpn) on a write,
and on a read it should prefer the local ro-instance. but that doesn't work
for me.
everytime I accessed some software in the district it was transfered
completly over the vpn from our headquarter.
did I something missunderstood or have I done something wrong !?
If you choose the rw-path (the "dotted" path) /afs/domain/.software, you
will always get the rw-path. OpenAFS do not bother about the location of
the volume at this point.

If you use the "normal" path /afs/domain/softare, you will preferable be
forwarded to an ro-Instance of that volume. In your case, users in the
headquarter would use a volume in one of your departments.

The decision, whether to use a RO or a RW instance of a volume is not
made by the location of the volume. the decision is based on:
- is it an explicit rw-mountpoint (.software)
- are ro instances available

If you do not make a rw-mountpoint, the afs client will contact
ro-volumes as long as you can access one. Only if no ro volume is
available, the rw instance is used.

Then there is another point to be aware of:
"Once RW, always RW"
So if you have in your afs path only on rw-volume, all the underlying
moint-points will be rw too. So if your root.cell volume (which is the
mount-point for /afs/domain) is only available as a rw-version, you will
never be able to access ro-volumes.
Post by Adnoh
the idea of this behaviour (take the lokal ro if available and just get what
you still need over vpn) was the coolest feature of the afs - i thougt. and
is the most case why I was looking on the whole afs thing - and not
something like nfs.
that is basically still true, but the decision is not made by accessing
a file. the decision is made by choosing the right mount-point for a volume.

Which volume you have access to is a manner of mount-points and ACLs,
NOT of the location of the volume. In an ideal world a user do not need
to know on which server his data is stored.


Klaas
A***@Dartmouth.EDU
2007-05-09 13:15:16 UTC
Permalink
i have been following this thread, and i haven't seen any description
of what you are actually trying to accomplish. are you trying to make
it so that only the users at "headquarters" can write to certain
files, and that the users at the "district sites" will only be able to
read the files, not make changes?

it sounds like you are heading for trouble trying to use replicated
volumes and possibly funky mounting schemes to accomplish something
you can do using acls.
why won't acls do what you want?

also, people have mentioned the issue of server quorum in a cell where
the database servers are geographically remote from each other. before
you design your cell, please consider the underlying network. with
AFS, unless you have a really stable network underneath, you will find
that putting fileservers or database servers out at remote sites is
not necessarily a good thing. example: one cell, three database
servers, one in sweden, one in italy, one in brazil, users in all
three places, administrators in sweden; say quorum drifts to brazil,
then network between brazil and sweden is down; the admins can't make
any normal changes until the network comes back. how inconvenient!

as with any construction project, it's worth putting the time into
planning. good luck!

anne
Post by Klaas Hagemann
<snip>
Post by Adnoh
a volume called software (~1 Gig)
in our headquarter the rw-volume on the afs server.
in a district the (nightly) ro-snapshot of that volume.
/afs/domain/.software (-rw)
/afs/domain/software (ro)
so if I understand that right i should now be able to access the data under
/afs/domain/.software on both sides.
That is right, but you will always get the rw-instance in your headquarter.
Post by Adnoh
in the headquarter it should use always the rw-instance and in the district
it should use the rw-instance (over vpn) on a write,
and on a read it should prefer the local ro-instance. but that doesn't work
for me.
everytime I accessed some software in the district it was transfered
completly over the vpn from our headquarter.
did I something missunderstood or have I done something wrong !?
If you choose the rw-path (the "dotted" path) /afs/domain/.software,
you will always get the rw-path. OpenAFS do not bother about the
location of the volume at this point.
If you use the "normal" path /afs/domain/softare, you will preferable
be forwarded to an ro-Instance of that volume. In your case, users in
the headquarter would use a volume in one of your departments.
The decision, whether to use a RO or a RW instance of a volume is not
- is it an explicit rw-mountpoint (.software)
- are ro instances available
If you do not make a rw-mountpoint, the afs client will contact
ro-volumes as long as you can access one. Only if no ro volume is
available, the rw instance is used.
"Once RW, always RW"
So if you have in your afs path only on rw-volume, all the underlying
moint-points will be rw too. So if your root.cell volume (which is the
mount-point for /afs/domain) is only available as a rw-version, you
will never be able to access ro-volumes.
Post by Adnoh
the idea of this behaviour (take the lokal ro if available and just get what
you still need over vpn) was the coolest feature of the afs - i thougt. and
is the most case why I was looking on the whole afs thing - and not
something like nfs.
that is basically still true, but the decision is not made by accessing
a file. the decision is made by choosing the right mount-point for a volume.
Which volume you have access to is a manner of mount-points and ACLs,
NOT of the location of the volume. In an ideal world a user do not need
to know on which server his data is stored.
Klaas
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
Kim Kimball
2007-05-16 17:33:18 UTC
Permalink
To play devil's advocate, unless the network is really unstable (or
pitifully slow) the worst that will happen is temporary inconvenience,
and mostly to administrators. User's will be unable to modify PTS
protection groups and will not be able to change their password, unless
the network isolates them from all other (up) DB servers, in which case
we have other concerns as well.

And of course there's no requirement to put the DB servers where the
file servers are.

I've used both approaches -- all DB servers in one location, 3 (and 5)
DB servers all in different locations.

I've also moved DB server functionality off of machines that were on
unstable (or pitifully slow) networks.

YMMV

If the goals are as Anne describes then ACLs and protection groups are
the way to go. The use of RW mount points does not help discriminate
access within a directory. ACLs do.

Have fun!

Kim
Post by A***@Dartmouth.EDU
i have been following this thread, and i haven't seen any description
of what you are actually trying to accomplish. are you trying to make
it so that only the users at "headquarters" can write to certain
files, and that the users at the "district sites" will only be able to
read the files, not make changes?
it sounds like you are heading for trouble trying to use replicated
volumes and possibly funky mounting schemes to accomplish something
you can do using acls.
why won't acls do what you want?
also, people have mentioned the issue of server quorum in a cell where
the database servers are geographically remote from each other. before
you design your cell, please consider the underlying network. with
AFS, unless you have a really stable network underneath, you will find
that putting fileservers or database servers out at remote sites is
not necessarily a good thing. example: one cell, three database
servers, one in sweden, one in italy, one in brazil, users in all
three places, administrators in sweden; say quorum drifts to brazil,
then network between brazil and sweden is down; the admins can't make
any normal changes until the network comes back. how inconvenient!
as with any construction project, it's worth putting the time into
planning. good luck!
anne
Post by Klaas Hagemann
<snip>
Post by Adnoh
a volume called software (~1 Gig)
in our headquarter the rw-volume on the afs server.
in a district the (nightly) ro-snapshot of that volume.
/afs/domain/.software (-rw)
/afs/domain/software (ro)
so if I understand that right i should now be able to access the data under
/afs/domain/.software on both sides.
That is right, but you will always get the rw-instance in your headquarter.
Post by Adnoh
in the headquarter it should use always the rw-instance and in the district
it should use the rw-instance (over vpn) on a write,
and on a read it should prefer the local ro-instance. but that doesn't work
for me.
everytime I accessed some software in the district it was transfered
completly over the vpn from our headquarter.
did I something missunderstood or have I done something wrong !?
If you choose the rw-path (the "dotted" path) /afs/domain/.software,
you will always get the rw-path. OpenAFS do not bother about the
location of the volume at this point.
If you use the "normal" path /afs/domain/softare, you will preferable
be forwarded to an ro-Instance of that volume. In your case, users in
the headquarter would use a volume in one of your departments.
The decision, whether to use a RO or a RW instance of a volume is not
- is it an explicit rw-mountpoint (.software)
- are ro instances available
If you do not make a rw-mountpoint, the afs client will contact
ro-volumes as long as you can access one. Only if no ro volume is
available, the rw instance is used.
"Once RW, always RW"
So if you have in your afs path only on rw-volume, all the underlying
moint-points will be rw too. So if your root.cell volume (which is the
mount-point for /afs/domain) is only available as a rw-version, you
will never be able to access ro-volumes.
Post by Adnoh
the idea of this behaviour (take the lokal ro if available and just get what
you still need over vpn) was the coolest feature of the afs - i thougt. and
is the most case why I was looking on the whole afs thing - and not
something like nfs.
that is basically still true, but the decision is not made by accessing
a file. the decision is made by choosing the right mount-point for a volume.
Which volume you have access to is a manner of mount-points and ACLs,
NOT of the location of the volume. In an ideal world a user do not need
to know on which server his data is stored.
Klaas
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
Christopher D. Clausen
2007-05-09 13:49:04 UTC
Permalink
Post by Adnoh
Post by Christopher D. Clausen
Post by Adnoh
i don't want to have all the volumes in our headquarter. so every
time a user openes his word-doc or similar it would be completly
transfered over our VPN - and I can hear the people crying "our
fileservers are too slow !" so seperate fileservers in every
district would be a good choice, I think - would'nt they ?
That is an option. There are of course problems with doing either.
Remember that the AFS clients themselves cache read-only data. So if
most of your data is only being read and not written back that
often, it might make sense to have only centrally located AFS
servers.
thats right - but my problem at the moment is that we have only
windows-workstations. And I did'nt figure out how
I could customize the MSI-installation in that way, so I don't need to
travel to all our restricts and configure that client.
so I would like one afs "client" per district - the fileserver which
is already there (a linux gentoo machine) - some kind of
afs->samba-gateway
While some people have done afs->samba gateways, I personally don't
think that is a good idea. You have all the problems of samba and AFS
combined and you miss a lot of the best features of either.
Post by Adnoh
Post by Christopher D. Clausen
By default, the AFS client prefers to use readonly volumes, so if you
create a replica of a volume, the data will immediately become
readonly. You can however manualy force the mount point to be RW
(-rw option to fs mkm) and this way you can have an RW volume in
each local district and still be able to clone the data to other
servers using vos release. All volume rights must go to directly to
the RW volume. The AFS client does not detect when you want to make
a write and find the proper RW volume. You can modify the code to
make it behave that way, but there are
reasons for not doing that.
a volume called software (~1 Gig)
in our headquarter the rw-volume on the afs server.
in a district the (nightly) ro-snapshot of that volume.
/afs/domain/.software (-rw)
/afs/domain/software (ro)
so if I understand that right i should now be able to access the data
under /afs/domain/.software on both sides.
in the headquarter it should use always the rw-instance and in the
district it should use the rw-instance (over vpn) on a write,
and on a read it should prefer the local ro-instance. but that
doesn't work for me.
everytime I accessed some software in the district it was transfered
completly over the vpn from our headquarter.
did I something missunderstood or have I done something wrong !?
What commands did you use to set this up? And physically where are the
servers that you used to do it? It should be possible to do something
that you want, but users will need to understand the difference between
the paths and open the appropriate folder for either writting or
reading. You can't have just writes go to the RW volume.
Post by Adnoh
the idea of this behaviour (take the lokal ro if available and just
get what you still need over vpn) was the coolest feature of the afs
- i thougt. and is the most case why I was looking on the whole afs
thing - and not something like nfs.
You might need to use fs setserverprefs to have the clients on each side
use the correct server. Also, note that the AFS client will NOT switch
between using the RO and RW automatically (well, if the RO goes down,
the RW will be used, but that isn't likely what you want to happen in
this case.) If you are using the "dot path" all reads and writes will
be to the RW volume.

Generally, its a "best practice" to have an RO clone on the same server
as the RW as well. Not sure if you did that or not.
Post by Adnoh
Post by Christopher D. Clausen
However, you might simply be better off using a more common network
filesystem like NFS or samba and using something like rsync to backup
the data nightly. You mentioned a VPN. Since the network link is
already encrypted, you don't require filesystem encryption? Or do you?
I'm not shure of the encryption ting. the vpn is a line from a large
provider in germany. so I think the line is secure, but I'm a little
bit paranoide ;-)
AFS has built-in encryption. Its not the best, but its better than
nothing. Since you already have a secured VPN, that is not an issue for
you though.
Post by Adnoh
Post by Christopher D. Clausen
It seems as though you are trying to use AFS like NFS or samba,
creating a single
large share point and allowing everyone to write in it. This is not
the best way to use AFS, although it mostly works. Replicating
single large volumes can take a long time, especially over slow
links.
yes and no. we have our samba-fileservers in every district completely
seperated from each other.
so if user a from district a wants to give a file to user b from
district b for working on it - he uses email. when
user b has his work completed on that file he uses that way to get
the file back to user a - and if someone in district
a has altered the file in that time - they have a problem...
so yes, i would like one big namespace - something like
/afs/domain/data/it
/controlling
/bookkeeping
and so on - so every user in a organisation unit can access his data
from each district he is at the moment and easilly share that to
someone else who is maybe not in the same district.
i thougt this is something afs wants me to give.
AFS can do what you want, but the performance over the WAN links is
likely going to be poor. And since the RW volume can only be a single
server, someone is going to be stuck with the slow connection.
Post by Adnoh
Can you describe a "distrcit office" in more detail? How many users?
->This differs - lets say 10 districts, 5 with ~100 users, 60 Gig of
data and a "data-change" of 100MB / Day
and the other 5 with the half of the above.
If you data change rate is only 100MBs, that should be okay to just use
a client from each district. Yes, opening and closing files will be
slow, but try to use large client caches to minimize the impact.
Post by Adnoh
Is there technical staff there to diagnose problems with an AFS
server, if they occur? Are the offices always connected to the
network? What type of connection do they have? Bandwidth? Latency?
->no - the only technical staff is in our headquarter. we have a vpn
from a large provider which has a offline-time of maybe 10 Min / Year
at all - so it is very goot. The Bandwith differs - from 512k -
2Mbit. they are connected 24h / day.
You do not likely want to try and have AFS servers in each remote
location if you do have the staff there to fix problems.
Post by Adnoh
Do you use Kerberos 5 currently within your organization? A single
realm? Or a realm per district?
->We use a windows 2003 ADS for authentications of the windows
workstations and the samba-servers.
Ah, ok.

Have you looked into using Microsoft's Dfs? It might provide the
namespace that you want, but not require you to completely switch your
infrastructure to use AFS.
Post by Adnoh
Do you have any off-site backup or disaster recovery requirements?
->I would like to have a backup on the local usb-hdd in each district
and a centraliced backup in our headquarter with a fullbackup/week and
diff-backup/day.
Okay. Its pretty easy to clone or copy volumes with AFS. The exact
details would depend upon lots of factors and should probably be
addressed in a seperate thread.
Post by Adnoh
Any specific features that the project MUST do? Any features that the
project SHOULD do? Anything else that
would be nice to do?
-> yes - that what I have mentioned above ;-) - the "global"
namespace would be nice. maybe it is
interesting to tell you that we wanne migrate the workstations to
linux in the next 2-3 years.
You can do a similar "global" type namespace using Dfs and Windows AD.
I strongly suggest you look at it first, especially for a mostly Windows
environment.
Post by Adnoh
How much data are we talking about here? Total and at each district?
What is the "change rate" of your data? How much
data is modified per day or per week as a percentage of the total
data? ->mentioned above - all together, maybe ~ 500 Gig at the moment
- but I don't know how much duplicate data is there arround - you now
that "i need my files in every district, my local hdd and for best on
my usb again" ;-)
Yeah, getting accurate disk space counts across lots of different
machines isn't easy.

-----

If you want some specific help with trying out AFS, it might be worth
asking the good folks on the #openafs IRC channel on the Freenode
network. For instance, the RO vs RW stuff isn't easy to fully grasp at
first.

<<CDC
Kim Kimball
2007-05-10 15:06:51 UTC
Permalink
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<br>
Christopher D. Clausen wrote:
<blockquote cite="***@m11"
type="cite"> <pre wrap="">Adnoh <a class="moz-txt-link-rfc2396E" href="mailto:***@users.sourceforge.net">&lt;***@users.sourceforge.net&gt;</a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Christopher D. Clausen wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">i don't want to have all the volumes in our headquarter. so every
time a user openes his word-doc or similar it would be completly
transfered over our VPN - and I can hear the people crying "our
fileservers are too slow !" so seperate fileservers in every
district would be a good choice, I think - would'nt they ?
</pre>
</blockquote>
<pre wrap="">That is an option. There are of course problems with doing either.
Remember that the AFS clients themselves cache read-only data. So if
most of your data is only being read and not written back that
often, it might make sense to have only centrally located AFS
servers.
</pre>
</blockquote>
</blockquote>
</blockquote>
Read/write data is also cached.&nbsp; If not changed it remains cached.&nbsp; If
changed and pushed back to the file server it is marked stale in the
other client caches and fetched again by other clients only if it is
requested again.<br>
<br>
Caching is whole-file, so if a large file is changed on client A and
was previously cached on client B, when client B requests the file
again it notices it is marked as stale and fetches the entire file from
the file server by client B.<br>
<br>
(Since client A made the changes the file is already cached on client
A.&nbsp; Writes to cached AFS files go first to the AFS cache and then back
to the AFS file server on flushes or closes or syncs.)<br>
<blockquote cite="***@m11"
type="cite">[]
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">By default, the AFS client prefers to use readonly volumes, so if you
create a replica of a volume, the data will immediately become
readonly. </pre>
</blockquote>
</blockquote>
</blockquote>
Two other factors affect whether a given replicated volume will be
accessed as RO or RW.<br>
<br>
Given mount points /a/b/c/d, the volume mounted at d/ will not be
accessed as RO if <br>
<br>
a) any of the mount points above it are RW mount points.<br>
b) any of the volumes mounted above it are not replicated.&nbsp; (This is
determined by the VLDB entry for the volume.&nbsp; If I have five replicas
on five different file servers but the VLDB entry for the volume does
not show any RO sites, then the volume is treated as unreplicated.&nbsp;
It's important that the VLDB be in the correct state.)<br>
<br>
It's easy to check -- assume that /afs mounts root.afs and root.afs is
replicated, and that cellname mounts root.cell and root.cell is
replicated, and that &lt;next&gt; is in a volume that is also
replicated, where &lt;next&gt; might be a directory in /afs/cellname or
perhaps a mount point to a volume called '&lt;next&gt;'<br>
<br>
cd /afs<br>
fs lq&nbsp; <br>
cd cellname<br>
fs lq<br>
cd &lt;next&gt;<br>
fs lq<br>
<br>
Each 'fs lq' command should return a .readonly volume name .&nbsp; Example:<br>
<br>
[***@satchmo afs]$ cd /afs<br>
[***@satchmo afs]$ fs lq<br>
Volume Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Quota&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Used %Used&nbsp;&nbsp; Partition<br>
root.afs.readonly&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5000&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 7&nbsp;&nbsp;&nbsp; 0%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 48%<br>
[***@satchmo afs]$ cd ccre.com<br>
[***@satchmo ccre.com]$ fs lq<br>
Volume Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Quota&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Used %Used&nbsp;&nbsp; Partition<br>
root.cell.readonly&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5000&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 24&nbsp;&nbsp;&nbsp; 0%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 48%<br>
[***@satchmo ccre.com]$ cd user<br>
[***@satchmo user]$ fs lq<br>
Volume Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Quota&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Used %Used&nbsp;&nbsp; Partition<br>
root.cell.readonly&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5000&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 24&nbsp;&nbsp;&nbsp; 0%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 48%<br>
[***@satchmo user]$ cd k<br>
[***@satchmo k]$ fs lq<br>
Volume Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Quota&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Used %Used&nbsp;&nbsp; Partition<br>
root.cell.readonly&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5000&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 24&nbsp;&nbsp;&nbsp; 0%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 48%<br>
<br>
-- Up to here all directory nodes are in a replicated volume and all
mount points are 'regular' (not RW) mount points.<br>
-- The next volume is not replicated, and no .readonly suffix is
returned by 'fs lq'<br>
[***@satchmo k]$ cd kim<br>
[***@satchmo kim]$ fs lq<br>
Volume Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Quota&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Used %Used&nbsp;&nbsp; Partition<br>
user.k.kim&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no limit&nbsp;&nbsp; 1074945&nbsp;&nbsp;&nbsp; 0%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4%<br>
[***@satchmo kim]$<br>
<br>
-- To further illustrate, we know that root.afs is replicated (fs lq
returned .readonly at /afs.)&nbsp; However, the rule is "
Jeffrey Altman
2007-05-10 15:34:29 UTC
Permalink
Read/write data is also cached. If not changed it remains cached. If
changed and pushed back to the file server it is marked stale in the
other client caches and fetched again by other clients only if it is
requested again.
Caching is whole-file, so if a large file is changed on client A and was
previously cached on client B, when client B requests the file again it
notices it is marked as stale and fetches the entire file from the file
server by client B.
Caching in OpenAFS is not whole-file. Caching is performed in chunks.
Only the chunks that are actually requested by the client will be placed
into the cache. Chunks are stored in a least-recently-used list and are
recycled last use first.

If a change to a file occurs by another client, all of the previously
cached chunks for a client will be discarded. Only the chunks that are
requested in the future will be re-read from the file server.
(Since client A made the changes the file is already cached on client
A. Writes to cached AFS files go first to the AFS cache and then back
to the AFS file server on flushes or closes or syncs.)
On Windows, the writes are written back to the file server in real time.
The Windows client does not implement the write-on-close semantics of
the UNIX client.

Jeffrey Altman
Secure Endpoints Inc.
Kim Kimball
2007-05-10 17:20:14 UTC
Permalink
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
True.&nbsp; Whole-file if smaller than chunk size.<br>
<br>
Jeffrey Altman wrote:
<blockquote cite="***@m11"
type="cite">
<pre wrap="">Kim Kimball wrote:

</pre>
<blockquote type="cite">
<pre wrap="">Read/write data is also cached. If not changed it remains cached. If
changed and pushed back to the file server it is marked stale in the
other client caches and fetched again by other clients only if it is
requested again.

Caching is whole-file, so if a large file is changed on client A and was
previously cached on client B, when client B requests the file again it
notices it is marked as stale and fetches the entire file from the file
server by client B.
</pre>
</blockquote>
<pre wrap=""><!---->
Caching in OpenAFS is not whole-file. Caching is performed in chunks.
Only the chunks that are actually requested by the client will be placed
into the cache. Chunks are stored in a least-recently-used list and are
recycled last use first.

If a change to a file occurs by another client, all of the previously
cached chunks for a client will be discarded. Only the chunks that are
requested in the future will be re-read from the file server.

</pre>
<blockquote type="cite">
<pre wrap="">(Since client A made the changes the file is already cached on client
A. Writes to cached AFS files go first to the AFS cache and then back
to the AFS file server on flushes or closes or syncs.)
</pre>
</blockquote>
<pre wrap=""><!---->
On Windows, the writes are written back to the file server in real time.
The Windows client does not implement the write-on-close semantics of
the UNIX client.

Jeffrey Altman
Secure Endpoints Inc.




</pre>
</blockquote>
</body>
</html>
Adnoh
2007-05-16 06:54:49 UTC
Permalink
Hello everyone, I'm back again :-)

very big thanks to every one who tried to help me to get some light into my
afs knowledge-darkness.
so i think afs can do amazing stuff but it also needs the end users to have
some computing skills which my users didn't have. they will never realize
the need for accessing some files in /afs/domain/mydir and others in
/afs/domain/.mydir
I need to think about the design of my afs namespace again - maybe i get it
work as I need it.
maybe I fall back to a rsync solution or setup afs-fileservers in every
district which distributes their own RW-instance of that locations data.
/afs/domain/distric1/oux /afs/domain/distric1/ouy /afs/domain/district2/oux
... and so on.

by the way, yes I know DFS - but I'm not willing to spend all that money
needed for a Windows server in every district just for DFS. some of my goals
for the next few years is to get the whole redmond-stuff away from our
organisation... it is everywhere, and it is even getting more - just like a
virus - on our company pcs, on our telephone system, on my mobile-phone, yes
- in our cars it is right know, too ... but that's my personal dislike.

again, thanks for everyone have so much patience with me - didn't see that
very often on a mailing list.

Cheers
--
View this message in context: http://www.nabble.com/a-noobs-question-and-problems-on-a-new-cell-tf3691140.html#a10636547
Sent from the OpenAFS - General mailing list archive at Nabble.com.
anne salemme
2007-05-04 18:43:40 UTC
Permalink
where are the districts physically located relative to the headquarters,
and where do you plan to locate the afs servers for each "domain"?

typically, for cells that span big geographical distances, you want to
be sure that the cell database servers always stay up and in contact
with each other.

i would imagine that you could put all the afs servers at the
headquarters, and basically just use it as back end storage, with no
access from clients outside the headquarters....is that what you have in
mind?

anne
Post by Adnoh
ok, I try to explain.
we have several districts connected to our headquarter over a vpn.
In every district we have ~50-150 windows-clients and a samba server ( all
authenticating against a Win2003 Server with ADS)
for their filestorage with an usb-hdd attached for backup purpose.
In our headquarter we would have enough space to hold all the company data
and backup.
so i would have something like
/afs/domain/dist1/data
/afs/domain/dist2/data
/afs/domain/dist3/data
/afs/domain/shared/data
where every district has it's own data folder and one volume where all the
districts work together in.
the backup should be done on a central place (our headquarter).
so if somebody can explain me how you would build this scenario it would be
really great for me to get started ;-)))
thanks
Post by Jason Edgecombe
Post by Adnoh
maybe stupid question but I don't know it :-)
i have two fileservers connected over a 2 mbit vpn (fs1 with rw-volume and
fs2 with a ro-instance) and mounted that on /afs/domain/vol
my users on fs1 work with /afs/domain/vol - rw + ro - thats clear to me.
when my users on fs2 read a file it should be taken from the ro-instance on
the internal harddrive and if they change anything it is written to
fs1-rw-instance !? on my try i get only the error "readonly filesystem"
can this be done or do i have to setup a mountpoint /afs/domain/.vol (-rw)
and point the users to that directory !?
thanks for any responses to help me getting started
/afs/domain/volume is the read-only copy if it exists.
/afs/.domain/volume is always the read-write copy. <-- note the
/afs/<dot>domain This is referred to at the "dotted" path.
writing to the read-only path of /afs/domain/volume will get the error
"read-only filesystem". To write, you must use the path
/afs/.domain/volume
If you are doing lots of writes, then having the volume replicated may
not be the best thing.
What is the usage model for this volume?
Jason
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
Loading...