[UFO Chicago] NFS problems

[Rory L. O'Connor]

having some issues with NFS on a new server arrangement we have.  we 
have a storage server, database server, and two web application servers 
with their home directories mounted to the storage server via NFS.  All 
fronted by a load balancer.  Anyway, this stuff is all super fast 
top-of-the line stuff with 64-bit processors and tons of memory.  But it 
appears NFS is bottlenecking.  I noticed this when I ran a script on the 
webserver that writes a large file to the storage server.  It took a 
very long time to run (MUCH longer than our current webserver 
arrangement which the new hardware is replacing).  I tailed the file 
while the script was running and noticed that it would write, pause, 
write, pause, etc.  Like it was reaching a threshold and pausing until 
it cleared out.

tried increasing rsize and wsize, and memory limits on the input queue 
and it made an initial difference - from 2 minutes down to 10 secs - but 
on the old server it runs in 4 seconds (and the old server is old and 
slow!).  so it's almost like NFS is making our fancy new hardware 
arrangement actually a step BACKWARDS.  our sessions are managed with 
text files that will be written over this NFS connection, and though I 
don't have a way to test how it will work under heavy load, if writing 
is slow for one big file I'm gonna go ahead and guess it's going to be 
slow for tons of little ones.

Has anyone used NFS for heavy-load arrangements, or used another 
solution like OpenAFS with alot of load?  any ideas are apprciated!

Thanks!