ls is slow

[Muhammad Asghar_1]

Dears,
its a unique case.
machine is dl380 g3.
fc hba is qla2312 (rev 2). a slice is shared from XP with size 125 GB. OS is Linux EL4 kernel 2.4.

my problem is regarding its response time. when i do ls, it goes in coma and come back after 1 to 5 minutes. slice has oracle file system and system is in production. your help in this regard will be appreciated.


Regards,

[Matti_Kurkela]

What's the filesystem type on that slice?

Does the problem happen in one directory only, or in all directories on that filesystem?

Is there a huge number of files in a single directory?

Is there anything special about those files? For example, are some of them links to a NFS share?

Are there any related messages in syslog or in dmesg output? (e.g. scary-looking disk error messages appearing while the ls command is in coma?)

MK

[Ciro Iriarte]

Any relevant entries in /var/log/messages?. Are you using multipath?, check on both ends that the fibre is not loose.

[Muhammad Asghar_1]

filesystem is OCFS on that slice.
problem is directly related to the number of files in a dirctor on that slice. more the number of files, more time it will take to ls
total slice size is 122 GB and 84% is full.
no NFS involvment.
syslog (var/log/messages) shows everything normal. i take fc cable out; syslog shows that loop is no more there and then loop is back after cable is back.

Single FC is used to share multiple slices on same disk. no such problem on other slices.

disks are not multi-path.

by the way, i m think now that OCFS is culprit. i dont know to what extent you people will agree with me. :(


Regards,
Asghar

[Rick Beldin]

I am not sure what distro 'Linux EL4 kernel2.4' is, but if you suspect the problem is with OCFS, and this is a Red Hat installation, then you need to be talking to the folks at Oracle.

If the platform is SLES9 or SLES10, then Novell is responsible for the OCFS implementation.

In general, temporary pauses like this could be coming from a number of places, but I would suspect that some kernel thread has taken out the big kernel lock (BKL) and everyone is stacked up behind it. In RHEL3 (2.4 kernel) there was a situation where a kernel thread would run periodically (kscand)
and try to find free pages. While doing so, it would lock the page tables.

You should:

- examine sar for excessive swapping
- see if the 'coma' is periodic
- examine Oracle Metalink database

[Ciro Iriarte]

I've seen this kind of "hang" with a loose FC cable on the switch side.

How did you create the filesystem?, probably some tuning is needed.

Apart of the obvious block size parameter, there are some "templates" that you can specify at creation time to improve performance.

From mkfs.ocfs2 man:

-T filesystem-type
Specify how the filesystem is going to be used, so that mkfs.ocfs2 can chose optimal filesystem parameters for that
use. The supported filesystem types are:

mail Appropriate for file systems which will have many meta data updates. Creates a larger journal.

datafiles
Appropriate for file systems which will host a relatively small number of very large files. A small jourâ
nal is selected. Cluster size will be at least 128K.

[Muhammad Asghar_1]

dears,
i have checked backup end , interconnect and OS. have asked oracle vendor to come-in and play. will update soon. any new idea in this regard will be much appreciated.