topic Re: ls is slow in Operating System - Linux

ls is slow

Muhammad Asghar_1 — Tue, 27 Jan 2009 06:44:41 GMT

Dears,
its a unique case.
machine is dl380 g3.
fc hba is qla2312 (rev 2). a slice is shared from XP with size 125 GB. OS is Linux EL4 kernel 2.4.

my problem is regarding its response time. when i do ls, it goes in coma and come back after 1 to 5 minutes. slice has oracle file system and system is in production. your help in this regard will be appreciated.

Regards,

Re: ls is slow

Matti_Kurkela — Tue, 27 Jan 2009 08:18:34 GMT

What's the filesystem type on that slice?

Does the problem happen in one directory only, or in all directories on that filesystem?

Is there a huge number of files in a single directory?

Is there anything special about those files? For example, are some of them links to a NFS share?

Are there any related messages in syslog or in dmesg output? (e.g. scary-looking disk error messages appearing while the ls command is in coma?)

MK

Re: ls is slow

Ciro Iriarte — Tue, 27 Jan 2009 21:56:09 GMT

Any relevant entries in /var/log/messages?. Are you using multipath?, check on both ends that the fibre is not loose.

Re: ls is slow

Muhammad Asghar_1 — Wed, 28 Jan 2009 06:13:05 GMT

filesystem is OCFS on that slice.
problem is directly related to the number of files in a dirctor on that slice. more the number of files, more time it will take to ls
total slice size is 122 GB and 84% is full.
no NFS involvment.
syslog (var/log/messages) shows everything normal. i take fc cable out; syslog shows that loop is no more there and then loop is back after cable is back.

Single FC is used to share multiple slices on same disk. no such problem on other slices.

disks are not multi-path.

by the way, i m think now that OCFS is culprit. i dont know to what extent you people will agree with me. :(

Regards,
Asghar

Re: ls is slow

Rick Beldin — Wed, 28 Jan 2009 16:17:17 GMT

I am not sure what distro 'Linux EL4 kernel2.4' is, but if you suspect the problem is with OCFS, and this is a Red Hat installation, then you need to be talking to the folks at Oracle.

If the platform is SLES9 or SLES10, then Novell is responsible for the OCFS implementation.

In general, temporary pauses like this could be coming from a number of places, but I would suspect that some kernel thread has taken out the big kernel lock (BKL) and everyone is stacked up behind it. In RHEL3 (2.4 kernel) there was a situation where a kernel thread would run periodically (kscand)
and try to find free pages. While doing so, it would lock the page tables.

You should:

- examine sar for excessive swapping
- see if the 'coma' is periodic
- examine Oracle Metalink database

Re: ls is slow

Ciro Iriarte — Wed, 28 Jan 2009 21:37:58 GMT

I've seen this kind of "hang" with a loose FC cable on the switch side.

How did you create the filesystem?, probably some tuning is needed.

Apart of the obvious block size parameter, there are some "templates" that you can specify at creation time to improve performance.

From mkfs.ocfs2 man:

-T filesystem-type
Specify how the filesystem is going to be used, so that mkfs.ocfs2 can chose optimal filesystem parameters for that
use. The supported filesystem types are:

mail Appropriate for file systems which will have many meta data updates. Creates a larger journal.

datafiles
Appropriate for file systems which will host a relatively small number of very large files. A small jourÃ¢
nal is selected. Cluster size will be at least 128K.

Re: ls is slow

Muhammad Asghar_1 — Fri, 30 Jan 2009 07:03:02 GMT

dears,
i have checked backup end , interconnect and OS. have asked oracle vendor to come-in and play. will update soon. any new idea in this regard will be much appreciated.