[OpenAFS] maximum number of files per directory??

[TIARA System Man]

hi guys,

i have one directory which contents 67,434 files in ext3 directory. i
can't copy all files into an afs directory. do you have any suggestion
of this issue? what is the maximum number of files per directory? the
version of openafs-server we are using is 1.4.11-76. thank you for
your help!!!

best, sam

[Jörg Herzinger]

See CAUTIONS in fileserver documentation [1].

The maximum number of directory entries is 64,000 if all of the
entries have names that are 15 octets or less in length. A name that
is 15 octets long requires the use of only one block in the directory.
Additional sequential blocks are required to store entries with names
that are longer than 15 octets. Each additional block provides an
additional length of 32 octets for the name of the entry.
Note that if file names use an encoding like UTF-8, a single
character may be encoded into multiple octets. In real world use, the
maximum number of objects in an AFS directory is usually between
16,000 and 25,000, depending on the average name length.

so long,
    Jörg Herzinger

[Simon Wilkinson]

Post by TIARA System Man
i have one directory which contents 67,434 files in ext3 directory. i
can't copy all files into an afs directory. do you have any suggestion
of this issue? what is the maximum number of files per directory?

man fileserver says:

The maximum number of directory entries is 64,000 if all of the entries
have names that are 15 octets or less in length. A name that is 15
octets long requires the use of only one block in the directory.
Additional sequential blocks are required to store entries with names
that are longer than 15 octets. Each additional block provides an
additional length of 32 octets for the name of the entry. Note that if
file names use an encoding like UTF-8, a single character may be
encoded into multiple octets.

It also goes on to say:
In real world use, the maximum number of objects in an AFS directory is
usually between 16,000 and 25,000, depending on the average name
length.

But that really depends on your "real world" - we have a number of directories here that have 60,000 or so objects in them. In fact, to hit a limit as low as 16,000 entires, every filename would need to take up 4 slots, which means every name would have to be more than 80 characters long.

Cheers,

Simon.

[Todd Lewis]

Post by Simon Wilkinson
But that really depends on your "real world" - we have a number of directories here that have 60,000 or so objects in them. In fact, to hit a limit as low as 16,000 entires, every filename would need to take up 4 slots, which means every name would have to be more than 80 characters long.

I once worked on a project that had ~800,000 files with names between 120
and 199 characters each. (Not in AFS, and not in the same directory!) The
first 8 characters of the files' names were held in common between large
groups of files, and performance was horrific. It wasn't until after the
project was over that we discovered that the hierarchical storage system
we were using employed a hash table to manage file names. It didn't hash
the entire file name though; to improve performance of the hashing
operation itself, it only hashed the first 6 characters. That gave us
about 200 hash buckets for 800,000 files.

Not that you should care about any of this. It's just to illustrate that
knowing (or not knowing in our case) the characteristics of your storage
system may have huge impacts on how effectively you use it.

Good luck with that.