Operating System - HP-UX
1748209 Members
2887 Online
108759 Solutions
New Discussion юеВ

Re: Need Technical Answer to FTP Problem with too many files

 
SOLVED
Go to solution
Laurie_2
Advisor

Need Technical Answer to FTP Problem with too many files

Hi All,

I fixed this end-users FTP problem by deleting
a lot of old big files. She had more files
in her directory than I thought possible.

We use QVT ver. 5.1.1 and telnet into our
server. I need to give my boss a technical
answer why she should clean up her files.

She also has way too many files and I am
trying to push her to create Directories and
sort all her files, but I am not having
much luck.She has several hundred files and
not one directory.

And she orginally told me that deleting the
files wouldn't fix the problem and it did...

So, having a technical boss...can I have
a good simple technical reason why folks
should not have too many files and why
using directories is good.

TIA,
Laurie
How can you make the world a better place
8 REPLIES 8
Tom Maloy
Respected Contributor

Re: Need Technical Answer to FTP Problem with too many files

There are two issues you can bring up:
1) software limitations on the system
2) effectiveness/efficiency of the person

From a software point of view, there are limitations on the sheer number of files that can be handled in a directory. Some software will stop working when too many files are referenced at once. Other software will just get extremely slooooow, trying to handle the volume.

Second, the effectiveness of the person. If the person took all of their documents out of the file cabinet and folders, and put them in one giant pile, the information is still accessible. It just takes a hideously long time to *find* that information. Which is exactly what the user is doing in that one directory.

Good luck.

Tom
Carpe diem!
Martin Johnson
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

Commands like "ls" take longer. Backups take longer and use more tape. Directories help organize the files, allowing you to find what you are looking for more quickly. Not deleting old files means you eventually need to buy more disk space.

How's that for starters?
HTH
Marty
Sandip Ghosh
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

The only problem I could see that it wil take long time to search. Whenever you are FTPing some file from the server, it will first search whether it maches with any of the filenames or not and then it starts transferring the data. If it finds the same file it asks for changing the name of the file. And for each and every file it does that. So if the no of files in that particular directory is very high then it will take much longer time to match & transfer the data.

Sandip
Good Luck!!!
S.K. Chan
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

You didn't really mention what ftp problem was fixed by doing the housekeeping. I was just wondering what has that got to do with ftp (unless the user is ftp'ing and using wildcards to match a long list of files) ?Anyway .. in theory .. if you're using JFS filesystem, there is no limit (except disk space) to how many files can be put in a directory. For HFS its limitation depends on the number of inodes made available when the FS is created. Some technical/common-sense reasons I can think of ...
1- Neatness and manageable structure.
2- Reduce some localized limitation (eg: "arg list too long" error)
3- Depending on the limitation of the 3rd party software when it comes to handling these number of files.
Dave Chamberlin
Trusted Contributor
Solution

Re: Need Technical Answer to FTP Problem with too many files

My wifes boss did the same on a PC - she saved everything in the My Documents folder and eventually had problems finding documents. Certainly the ls will take longer. Programs that run interactively will take longer when they read that directory for file related operations - causing excessive resource consumption for the whole system. Another issue is that a mistake - such as using the rm command would be costly in such a directory. Ask her if she uses folders in her file cabinet...
MANOJ SRIVASTAVA
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

Hi Laurie

I did a find in yahoo for your question here are some nice links :


Using Subject Directories - A Tutorial
... Because humans organize the websites in subject directories, you can
often find a good starting point if your topic is included. ...
http://home.sprintmail.com/~debflanagan/subject.html



and
http://google.yahoo.com/bin/query?p=why+using+directories+is+good.&hc=0&hs=0



Manoj Srivastava

Wodisch_1
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

Hello Laurie,

a very simple reason ist that UN*X (all flavours of!) reads and writes directories always at once, i.e. in one piece. Hence every file she creates, renames, or deletes in that huge directory forces your kernel ro read and write the whole directory. If that happens often, your system will waste megabytes of diskbandwidth just for writing that directory, leaving no bandwidth for the rest (like your applications, your users, and such).
So, move her files into multiple (and hence smaller) subdirectories, then make her log off, "cd" to her home-dir's parent-dir, rename her home-dir to something else, create a new (and small) one and "mv" everything from her *old* home-dir to the new one. Care about her new home-dirs ownership and permissions, of course...

HTH,
Wodisch
George Petrides_1
Honored Contributor

Re: Need Technical Answer to FTP Problem with too many files

You might want to try explaining using a basic data structures book (Any basic college book will do) and show them how tree structures work. If a directory has too many files (even 1000 files) then it takes very long to access a file in that directory. The directory is basically a node of a tree structured db and we are talking about serial search of a file. This affects seriously the performance of a system and especially backups that setup the archive bit on files (fbackup, omniback and other elegant packages... except from tar). What might have been the problem with FTP is timing out due to long access times, but this is just a guess. I hope this helps. George