Discussion:
how?: Distribute /home to n Terminal Servers
(too old to reply)
Steve Wright
2002-12-06 19:26:43 UTC
Permalink
Howdy Folks.

My Project is to distribute /home + UIDs + GIDs to multiple
installations of the K12LTSP thin-client Terminal Server (RedHat 7.3
based). http://k12ltsp.org

The problem to be solved is - One Terminal Server does not handle more
than 20-30 thin-clients.

So I will load-balance with XDMCP, and distribute /home across an
arbitrary number of Terminal Servers - 2 to probably 6 machines. and
consequently 60-1000 terminals.

I have spent two days trying to install OpenAFS-1.2.6 and have found the
documentation extensive, but convoluted, and I still do not have a
running OpenAFS Server.

(clients cannot connect to volume server..)

# tcpdump
08:20:15.737879 ws004.ltsp.afs3-callback > server.ltsp.afs3-vlserver:
rx data vldb call get-entry-by-name-u "root.afs.readonly" (56) (DF)
08:20:15.738017 server.ltsp > ws004.ltsp: icmp: server.ltsp udp port
afs3-vlserver unreachable [tos 0xc0]


I also do not understand what machine should be the OpenAFS server, and
what machine the OpenAFS clients - or should there be more than one server..

I am hoping List Members can offer an overall perspective and starting
point for my project, and some assistance with the odd technical detail
thereafter.


Thank you for your assistance.

kind regards,
Steve
Dan Pritts
2002-12-06 21:15:04 UTC
Permalink
Post by Steve Wright
I also do not understand what machine should be the OpenAFS server, and
what machine the OpenAFS clients - or should there be more than one server..
A typical configuration for a small AFS cell is to have 1 or 3 dedicated
AFS file & db servers (generally, you want to have an odd number of
database servers, so that in the event of a server going down, the servers
that are left can elect a new master server).

Larger cells will split the db functionality off to separate machines
from the file servers, and/or have more servers.


*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.


For your configuration, I woudl recommend a single AFS file & db server,
which may or may not also be one of your XDMCP servers (i would tend
to have it on a separate machine).

Depending on what it is that you are trying to get by using AFS (eg,
access across multiple sites/client side caching, or security, or
what), you may find that NFS is sufficient for your needs.

You will definitely find that NFS is easier to set up and administer.

danno
--
dan pritts ***@internet2.edu
systems administrator 734/352-4953 office
internet2 734/546-4423 mobile
Steve Wright
2002-12-06 22:29:33 UTC
Permalink
Dan Pritts wrote:

Hi Dan,

I am subscribed to the list, so replying to the list only will be fine. 8-)

For multiple Terminal Servers, we have been NFS mounting /home on a
dedicated machine. That is, none of the terminal servers actually
*have* a /home partition. All Servers `mount /home -t nfs
<remote-machine>/home/`.


We are looking at an alternative because ;

a. A backend /home NFS Server is a single point of failure, and it is
not easy to have this 'mount' failover in case this machine faults.

b. The NFS Server is an unnecessarily 'extra' box, when the same
functionality could be provided by a distributed filesystem between
existing Terminal Servers.

c. It is simpler. The installed, complete system will be distributed
as a ISO Set, so there will be very few AFS configuration for the
sysadmin to deal with.

The load-balancer will direct a login attempt to any one of the desktop
servers and users would expect to have their account act as normal, with
all their files intact.

At the next login attempt, the same user may end up logged-in to an
entirely different machine, with different DFS cache contents, but we
need the /home/<user>/* contents to be ready-synced for them to use.

I have examined all sorts of solutions over the last two months,
including `rsync`ing /home/<user>/* to all other machines on "user
logout", Intermezzo DFS, CODA DFS, and now I wish to test and examine a
solution with OpenAFS.

Please do tell me right away if I am entirely mis-directed. 8-)
Post by Dan Pritts
*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.
ok. Since we have a small number of machines, what would be the
situation running *all* (max 10) machines as AFS Servers, and then
acting as their own clients,
Post by Dan Pritts
For your configuration, I woudl recommend a single AFS file & db server,
which may or may not also be one of your XDMCP servers (i would tend
to have it on a separate machine).
ok. Will the clients still sync if the server is down ? Will we still
have our single point of failure ? Can more, or all of the Terminal
Servers act as OpenAFS Servers, as well as a client for the cell ? Can
we then `mount` this cell as /home ?
Post by Dan Pritts
Depending on what it is that you are trying to get by using AFS (eg,
access across multiple sites/client side caching, or security, or
what), you may find that NFS is sufficient for your needs.
We simply want 2-10 Terminal/Application Servers that keep /home/* +
UIDs + GIDs sync'ed at all times. All these Terminal/App Servers are on
the same physical Ethernet Segment, firewalled in, on one site, and
probably in the same rack - although it would be nice to distribute
across the complex/campus near their associated set of Terminals.
Post by Dan Pritts
You will definitely find that NFS is easier to set up and administer.
Oh Yes. I agree. But that functionality is limiting us now, and we
wish to move on.



Any and all comments would be most welcome.


TIA,
Steve
Dan Pritts
2002-12-06 22:51:26 UTC
Permalink
AFS won't do what you want it to do. If you went with AFS, you would
still essentially have a single point of failure for any given user's
home directory.

While AFS does give you a unified namespace with multiple file
servers, that does not mean that any given piece of data can necessarily,
at the backend, live in more than one place.

For an AFS volume (a volume is a logical unit of storage, such as a a
user's home directory or the directory for a particular software package)
to be replicated across multiple AFS file servers, that volume needs to
be set read-only. This is obviously unworkable for a home directory.

So, each read-write volume lives on a particular server, and if that
server is down, the user is hosed.

a situation like you describe would be doable, and would take you away
from the situation of "one NFS server dies, all users are dead", but
would leave you with a significantly harder to manage cluster,
and would still leave you with "any one server dies, some set of users
are dead".

If it were me, I would probably just stick with one NFS server. The cost
of the extra hardware is not that high, and you will more than make it
up in manageability. Obviously, it would be worth your while for any
sizeable installation to give some effort/$$ to making sure that that
NFS server did not go down. You might invest in a hardware RAID with
multiple host SCSI controllers, and use some sort of clustering software
to ensure that if the primary NFS server died, that a secondary machine
would take over.

Such hardware raids are not too terribly expensive - I just bought a
system with 8x80G IDE disks (host attachment via Ultra160 scsi), dual
PSU, single controller (but i am pretty sure that dual-controller configs
are available for a few hundred more) for under US$5000.
Post by Steve Wright
We are looking at an alternative because ;
a. A backend /home NFS Server is a single point of failure, and it is
not easy to have this 'mount' failover in case this machine faults.
b. The NFS Server is an unnecessarily 'extra' box, when the same
functionality could be provided by a distributed filesystem between
existing Terminal Servers.
c. It is simpler. The installed, complete system will be distributed
as a ISO Set, so there will be very few AFS configuration for the
sysadmin to deal with.
The load-balancer will direct a login attempt to any one of the desktop
servers and users would expect to have their account act as normal, with
all their files intact.
At the next login attempt, the same user may end up logged-in to an
entirely different machine, with different DFS cache contents, but we
need the /home/<user>/* contents to be ready-synced for them to use.
I have examined all sorts of solutions over the last two months,
including `rsync`ing /home/<user>/* to all other machines on "user
logout", Intermezzo DFS, CODA DFS, and now I wish to test and examine a
solution with OpenAFS.
Please do tell me right away if I am entirely mis-directed. 8-)
Post by Dan Pritts
*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.
ok. Since we have a small number of machines, what would be the
situation running *all* (max 10) machines as AFS Servers, and then
acting as their own clients,
Post by Dan Pritts
For your configuration, I woudl recommend a single AFS file & db server,
which may or may not also be one of your XDMCP servers (i would tend
to have it on a separate machine).
ok. Will the clients still sync if the server is down ? Will we still
have our single point of failure ? Can more, or all of the Terminal
Servers act as OpenAFS Servers, as well as a client for the cell ? Can
we then `mount` this cell as /home ?
Post by Dan Pritts
Depending on what it is that you are trying to get by using AFS (eg,
access across multiple sites/client side caching, or security, or
what), you may find that NFS is sufficient for your needs.
We simply want 2-10 Terminal/Application Servers that keep /home/* +
UIDs + GIDs sync'ed at all times. All these Terminal/App Servers are on
the same physical Ethernet Segment, firewalled in, on one site, and
probably in the same rack - although it would be nice to distribute
across the complex/campus near their associated set of Terminals.
Post by Dan Pritts
You will definitely find that NFS is easier to set up and administer.
Oh Yes. I agree. But that functionality is limiting us now, and we
wish to move on.
Any and all comments would be most welcome.
TIA,
Steve
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
danno
--
dan pritts ***@internet2.edu
systems administrator 734/352-4953 office
internet2 734/546-4423 mobile
Nathan Davis
2002-12-07 02:03:59 UTC
Permalink
Well, if you use at least three db + file servers, you can at least get
a more robust system that a single NFS server can offer. Sure, if one
of the servers goes down you lose access to all home volumes on it.
But, you can distribute the volumes across the servers so at least most
users can access their home directories if a single server goes down.
Also, in theory, you should be able to use fail-over with AFS servers
just as easily and reliably as with NFS servers.

AFS also offers several benefits over NFS, including a richer
authorization model and higher security. And, of course, you can move
volumes around with a single command, so its easy to keep your loads
balanced and expand as necessary.

About "manageability": I personally don't think AFS is much (if any)
more complicated than NFS. Sure, if things do wrong, it's not fun.
But, I haven't had any more trouble (so far) with AFS than I've had
with my experience with NFS (which, granted, isn't much for either one).
The only thing I can think of that makes AFS administration a bit of a
hassle is creating volumes, but then that pays off with more flexibility
in the long-run.

--Nathan Davis
Post by Dan Pritts
AFS won't do what you want it to do. If you went with AFS, you would
still essentially have a single point of failure for any given user's
home directory.
While AFS does give you a unified namespace with multiple file
servers, that does not mean that any given piece of data can necessarily,
at the backend, live in more than one place.
For an AFS volume (a volume is a logical unit of storage, such as a a
user's home directory or the directory for a particular software package)
to be replicated across multiple AFS file servers, that volume needs to
be set read-only. This is obviously unworkable for a home directory.
So, each read-write volume lives on a particular server, and if that
server is down, the user is hosed.
a situation like you describe would be doable, and would take you away
from the situation of "one NFS server dies, all users are dead", but
would leave you with a significantly harder to manage cluster,
and would still leave you with "any one server dies, some set of users
are dead".
If it were me, I would probably just stick with one NFS server. The cost
of the extra hardware is not that high, and you will more than make it
up in manageability. Obviously, it would be worth your while for any
sizeable installation to give some effort/$$ to making sure that that
NFS server did not go down. You might invest in a hardware RAID with
multiple host SCSI controllers, and use some sort of clustering software
to ensure that if the primary NFS server died, that a secondary machine
would take over.
Such hardware raids are not too terribly expensive - I just bought a
system with 8x80G IDE disks (host attachment via Ultra160 scsi), dual
PSU, single controller (but i am pretty sure that dual-controller configs
are available for a few hundred more) for under US$5000.
Post by Steve Wright
We are looking at an alternative because ;
a. A backend /home NFS Server is a single point of failure, and it is
not easy to have this 'mount' failover in case this machine faults.
b. The NFS Server is an unnecessarily 'extra' box, when the same
functionality could be provided by a distributed filesystem between
existing Terminal Servers.
c. It is simpler. The installed, complete system will be distributed
as a ISO Set, so there will be very few AFS configuration for the
sysadmin to deal with.
The load-balancer will direct a login attempt to any one of the desktop
servers and users would expect to have their account act as normal, with
all their files intact.
At the next login attempt, the same user may end up logged-in to an
entirely different machine, with different DFS cache contents, but we
need the /home/<user>/* contents to be ready-synced for them to use.
I have examined all sorts of solutions over the last two months,
including `rsync`ing /home/<user>/* to all other machines on "user
logout", Intermezzo DFS, CODA DFS, and now I wish to test and examine a
solution with OpenAFS.
Please do tell me right away if I am entirely mis-directed. 8-)
Post by Dan Pritts
*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.
ok. Since we have a small number of machines, what would be the
situation running *all* (max 10) machines as AFS Servers, and then
acting as their own clients,
Post by Dan Pritts
For your configuration, I woudl recommend a single AFS file & db server,
which may or may not also be one of your XDMCP servers (i would tend
to have it on a separate machine).
ok. Will the clients still sync if the server is down ? Will we still
have our single point of failure ? Can more, or all of the Terminal
Servers act as OpenAFS Servers, as well as a client for the cell ? Can
we then `mount` this cell as /home ?
Post by Dan Pritts
Depending on what it is that you are trying to get by using AFS (eg,
access across multiple sites/client side caching, or security, or
what), you may find that NFS is sufficient for your needs.
We simply want 2-10 Terminal/Application Servers that keep /home/* +
UIDs + GIDs sync'ed at all times. All these Terminal/App Servers are on
the same physical Ethernet Segment, firewalled in, on one site, and
probably in the same rack - although it would be nice to distribute
across the complex/campus near their associated set of Terminals.
Post by Dan Pritts
You will definitely find that NFS is easier to set up and administer.
Oh Yes. I agree. But that functionality is limiting us now, and we
wish to move on.
Any and all comments would be most welcome.
TIA,
Steve
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
danno
--
systems administrator 734/352-4953 office
internet2 734/546-4423 mobile
_______________________________________________
OpenAFS-info mailing list
https://lists.openafs.org/mailman/listinfo/openafs-info
Nathan Rawling
2002-12-07 02:05:41 UTC
Permalink
Dan,
Post by Dan Pritts
a situation like you describe would be doable, and would take you away
from the situation of "one NFS server dies, all users are dead", but
would leave you with a significantly harder to manage cluster,
and would still leave you with "any one server dies, some set of users
are dead".
I believe it depends a great deal on what sort of administrative hassle
you prefer. I concede that creating a user on a UNIX system with home
directories stored in AFS can be significantly more complicated than a
similar NFS system.

However, other common tasks are greatly simplified. Instead of the endless
tar-copy-tar that ensues when trying to move users from one NFS partition
(or server) to another, you get a great set of tools and infrastructure to
make it easier.

If you choose to take advantage of the single-sign-on opportunities that
exist with AFS you can almost avoid password files at all.
Post by Dan Pritts
If it were me, I would probably just stick with one NFS server. The cost
of the extra hardware is not that high, and you will more than make it
up in manageability. Obviously, it would be worth your while for any
sizeable installation to give some effort/$$ to making sure that that
NFS server did not go down. You might invest in a hardware RAID with
multiple host SCSI controllers, and use some sort of clustering software
to ensure that if the primary NFS server died, that a secondary machine
would take over.
Such hardware raids are not too terribly expensive - I just bought a
system with 8x80G IDE disks (host attachment via Ultra160 scsi), dual
PSU, single controller (but i am pretty sure that dual-controller
configs are available for a few hundred more) for under US$5000
However, you could also build a cell of 10 AFS servers using mirrored IDE
drives on some Linux variant for about $5000 too.

Personally, I like the distributed aspect of AFS. You can put one server
in a rack in one datacenter or closet, and another anywhere else, and
still seamlessly move and access your files. You can add and remove
servers without anyone becoming aware, or even noticing that their home
directory just moved halfway across the country.

I'm sorry to say that no matter how many times I dealt with a single
server that was redundant, clustered, or otherwise believed to be "safe",
something terrible happened. Sometimes the server crashed anyway, because
both RAID controllers failed at the same time, or perhaps rainwater
streamed from the ceiling onto the shared disk array of the cluster.

My preference is to have as much of my data left after a failure or
natural disaster takes out a single server (or a single site, for that
matter). I'm willing to invest a few minutes a day for the additional
peace of mind.

Nathan
Nathan Rawling
2002-12-07 01:52:16 UTC
Permalink
Steve,
Post by Steve Wright
Post by Dan Pritts
*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.
ok. Since we have a small number of machines, what would be the
situation running *all* (max 10) machines as AFS Servers, and then
acting as their own clients,
My understanding is that you want to keep the number of DB servers down
because of performance issues when you have too many. I suspect that
probably three DB servers would be sufficient for your application. You
want more than one to eliminate your risk of a single point of failure,
two is not recommended for technical reasons, so three is a good bet.

With that said, you could run the file server portion on each of your
terminal servers if you so desire. If your prime goal is to distribute
load, that might be the way to go.
Post by Steve Wright
Post by Dan Pritts
For your configuration, I woudl recommend a single AFS file & db server,
which may or may not also be one of your XDMCP servers (i would tend
to have it on a separate machine).
ok. Will the clients still sync if the server is down ? Will we still
have our single point of failure ? Can more, or all of the Terminal
Servers act as OpenAFS Servers, as well as a client for the cell ? Can
we then `mount` this cell as /home ?
Well, with one server, you would definitely have a single point of
failure. As Dan Pritts mentioned, even with a multiple server solution as
I talk about above, a server outage would result in some user home
directories becoming unavailable.

As you spread the user volumes across more fileservers, you reduce the
impact of the loss of one of them. If you use every one of your terminal
servers as a file server, and you have 10 servers, you only lose 10% of
your user volumes if/when a server dies.

I have heard that Netapp has an implementation of NFS with working
read-write failover, and I know that Veritas has a solution. However, I'm
sure that either would charge big bucks.
Post by Steve Wright
We simply want 2-10 Terminal/Application Servers that keep /home/* +
UIDs + GIDs sync'ed at all times. All these Terminal/App Servers are on
the same physical Ethernet Segment, firewalled in, on one site, and
probably in the same rack - although it would be nice to distribute
across the complex/campus near their associated set of Terminals.
You should watch out for the technical limitations of AFS as a solution.
If you have users with files larger than 2gig, AFS is not probably the
best solution. Byte-range file locking is another common area of concern.

The largest administration hassle with AFS, in my experience, has been
managing the backups. The bundled backup system is cumbersome (although it
has probably been greatly improved since I last used it), so frequently
you have to work out a solution that matches your environment.

All of my experiences with AFS, both for large cells and tiny ones, have
been positive, possibly excepting the backup system.

Nathan
Steve Wright
2002-12-07 07:57:11 UTC
Permalink
Post by Nathan Rawling
Steve,
Post by Steve Wright
Post by Dan Pritts
*every* machine that wants to access files that live in /afs needs to
be an afs client. This includes AFS server machines.
ok. Since we have a small number of machines, what would be the
situation running *all* (max 10) machines as AFS Servers, and then
acting as their own clients,
My understanding is that you want to keep the number of DB servers down
because of performance issues when you have too many. I suspect that
probably three DB servers would be sufficient for your application. You
want more than one to eliminate your risk of a single point of failure,
two is not recommended for technical reasons, so three is a good bet.
Thanks for that Nathan.

Are the 'performance issues' ethernet related, or processor related ?
We can easily place this traffic on a private network segment.
Post by Nathan Rawling
With that said, you could run the file server portion on each of your
terminal servers if you so desire. If your prime goal is to distribute
load, that might be the way to go.
The prime goal is redundancy and simplicity. I understand AFS is lots
more complex than an NFS solution, but we can mostly package the system
pre-configured, and the hardware/networking pre-specified.
Post by Nathan Rawling
Post by Steve Wright
ok. Will the clients still sync if the server is down ? Will we still
have our single point of failure ? Can more, or all of the Terminal
Servers act as OpenAFS Servers, as well as a client for the cell ? Can
we then `mount` this cell as /home ?
Well, with one server, you would definitely have a single point of
failure. As Dan Pritts mentioned, even with a multiple server solution as
I talk about above, a server outage would result in some user home
directories becoming unavailable.
As you spread the user volumes across more fileservers, you reduce the
impact of the loss of one of them. If you use every one of your terminal
servers as a file server, and you have 10 servers, you only lose 10% of
your user volumes if/when a server dies.
Ok. I would have mounted /home/ as *one* volume.. Can we have each
server maintain it's own sync'ed (cached) copy of the same (/home/)
fileset ?

I only need all the directories under /home/
Post by Nathan Rawling
I have heard that Netapp has an implementation of NFS with working
read-write failover, and I know that Veritas has a solution. However, I'm
sure that either would charge big bucks.
eeeh, ah well.
Post by Nathan Rawling
Post by Steve Wright
We simply want 2-10 Terminal/Application Servers that keep /home/* +
UIDs + GIDs sync'ed at all times. All these Terminal/App Servers are on
the same physical Ethernet Segment, firewalled in, on one site, and
probably in the same rack - although it would be nice to distribute
across the complex/campus near their associated set of Terminals.
You should watch out for the technical limitations of AFS as a solution.
If you have users with files larger than 2gig, AFS is not probably the
best solution.
2 gig ? shivers.. there would be no hdd space left ! nah, seriously,
1 700MB CD ISO at the most..
Post by Nathan Rawling
Byte-range file locking is another common area of concern.
I'm not up on that at this stage.. I trust I can safely ignore it for
now ? (famous last words...)
Post by Nathan Rawling
The largest administration hassle with AFS, in my experience, has been
managing the backups. The bundled backup system is cumbersome (although it
has probably been greatly improved since I last used it), so frequently
you have to work out a solution that matches your environment.
<shrug> Why can't I just `init 3 && cp -a /home /somewhere/else && init
5` ??

Sorry if that's an ignorant question, but it seems logical to me..

<spacer between questions>

I *thought* I heard that there is no such thing as UIDS + GIDS on an
AFS filesystem ? (I expect I'm really showing my ignorance now..)



Thank you people. Please, anything that anyone would like to add, feel
free...

regards,
Steve
Dan Pritts
2002-12-07 16:29:37 UTC
Permalink
Post by Steve Wright
Are the 'performance issues' ethernet related, or processor related ?
We can easily place this traffic on a private network segment.
I have never really paid much attention to this since i have not had a
large cell to administer but but i can guarantee you that the network
traffic is not significant.
Post by Steve Wright
The prime goal is redundancy and simplicity. I understand AFS is lots
more complex than an NFS solution, but we can mostly package the system
pre-configured, and the hardware/networking pre-specified.
Ok. I would have mounted /home/ as *one* volume..
I only need all the directories under /home/
Typically each user will get a volume for their home directory. For
example, quotas are enforced at the volume level.

You would probably have another volume for the /home directory itself,
which contained afs "mount points" for all the home directories.
*this* volume would hardly ever change, and could be replicated across
all of your file servers.

To implement what is suggested here, you would have volumes for each
user, and have the volumes spread across your server farm in roughly equal
proportions.

Should one server run short on disk space it is simple to
move volumes about from one server to another, but you would need to
have some scripts that monitored this situation and issued the commands
to move from one server to the other.
Post by Steve Wright
Can we have each
server maintain it's own sync'ed (cached) copy of the same (/home/)
fileset ?
This part will "just work". The AFS client software will automagically
grab whatever you need from the appropriate server when you need it,
assuming that said server is online.

It will *not* cache stuff forever. If you lose a file server, you lose
the data on it (other than your backups, which you will of course have).
Post by Steve Wright
I'm not up on that at this stage.. I trust I can safely ignore it for
now ? (famous last words...)
Probably, but there can be issues. the AFS FAQ at
http://grand.central.org/twiki/bin/view/AFSLore/UsageFAQ

byte-range file locking: [ Programmer ]

AFS does not support byte-range locking within a file, although lockf()
and fcntl() calls will return 0 (success). The first time a byte-range
lock is attempted, AFS will display:

"afs: byte-range lock/unlock ignored; make sure no one else else is
running this program."

whole file locking: [ Programmer ]

AFS does support advisory locking an entire file with flock(). Processes
on the same client workstation that attempt to lock a file obey the
proper locking semantics.

Processes on different AFS clients requesting a lock on the same file
would get EWOULDBLOCK returned.
Post by Steve Wright
<shrug> Why can't I just `init 3 && cp -a /home /somewhere/else && init
5` ??
Sorry if that's an ignorant question, but it seems logical to me..
The afs file server does not just serve out files that live on a directory
on the server - each volume is a single file on the file server partition.

Typically people back up on a volume-by-volume basis.

When you run AFS, you really run AFS - it is not just a file sharing
protocol, it is a way of life :).
Post by Steve Wright
I *thought* I heard that there is no such thing as UIDS + GIDS on an
AFS filesystem ? (I expect I'm really showing my ignorance now..)
UIDs: sort of, but not really. There is certainly the concept of
a user identifier.

GIDs: pretty much not. There is the concept of groups, though,
and the ACL mechanism is very flexible.



danno
--
dan pritts ***@internet2.edu
systems administrator 734/352-4953 office
internet2 734/546-4423 mobile
Dan Pritts
2002-12-07 16:06:45 UTC
Permalink
Several people commented on my "administrative hassle" assertion.

Primarily, this was based on the fact that with Steve's desired
configuration, he would end up with each machine in his cluster being
a critical point of failure (albeit a less critical point of failure
than a single NFS server), and thus each machine will require some
significant management, and if a single machine has a hardware failure,
then something has to be done about it sooner rather than later.

Compare this to a system with ten terminal servers and a pair of
clustered file servers. When one of the ten terminal servers dies,
the load balancer steve mentioned will notice, and distribute the
load to the other nine machines.

since these are all identical machines, this is easy, and the machine
can be repaired at the convenience of the administrator.

If we look at the likelihood of any single machine having a hardware
failure, we realize that by increasing our critical points of failure,
we are significantly increasing the likelihood of a failure that the
administrator will have to actually do something about in real-time.

We can protect against that by buying more expensive server-class
hardware for *all* the machines, at significant cost (say $3000 for
a decent server vs. $1500 for something with consumer-grade parts).

Or, we could take that extra money, and spend it on a robust central
server cluster. If we are talking two terminal server machines
it may not make sense. If we are talking ten terminal server machines,
it almost certainly does.

Additionally, distributing user volumes across a dozen file servers
would require administration - juggling volumes across file servers when
one gets too full, etc. Sticking with a big server or would require
significantly less.

Nathan Rawling commented on the ease of managing AFS vs. NFS - he is
certainly right that once it is running AFS has a lot of manageability
plusses. However, that comes with some significant overhead that
we in the AFS admin community tend to accept, and forget about.

I get the impression that Steve is intending to send out a preconfigured
CD set for admins at schools to load on their clusters. This suggests
that the end admin needs to have the simplest possible configuration.
I posit that a single (or clustered) (probably NFS) file server would
more likely provide that.

steve, I would love to hear more about who the consumers of your project
will be. It sounds interesting.

danno

dan pritts ***@internet2.edu
systems administrator 734/352-4953 office
internet2 734/546-4423 mobile
Derrick J Brashear
2002-12-07 16:38:06 UTC
Permalink
Post by Dan Pritts
Post by Steve Wright
Are the 'performance issues' ethernet related, or processor related ?
We can easily place this traffic on a private network segment.
I have never really paid much attention to this since i have not had a
large cell to administer but but i can guarantee you that the network
traffic is not significant.
if you're doing backups over your production network in a large cell, it
can be large. for everyone else i have to agree with Dan.
Post by Dan Pritts
Should one server run short on disk space it is simple to
move volumes about from one server to another, but you would need to
have some scripts that monitored this situation and issued the commands
to move from one server to the other.
run the volume balancer automatically every night.
http://ftp.andrew.cmu.edu/pub/AFS-Tools/balance-1.1b.tar.gz

it's probably due for another release and some updating.
Post by Dan Pritts
When you run AFS, you really run AFS - it is not just a file sharing
protocol, it is a way of life :).
for the moment. a fileserver which exports a real filesystem but treats it
as "sealed" (all modifications come in through afs, which makes noticing
modifications and breaking callbacks much simpler) is within the realm of
possible. as long as you only break the "seal" for backups (but you lose
the clone ability) or when you're doing maintenance it all works.
Continue reading on narkive:
Loading...