topic Re: SAM Needs Disk and File System Input! in Operating System - HP-UX

SAM Needs Disk and File System Input!

SAM TEAM — Mon, 17 Nov 2003 16:36:07 GMT

Hello! Many thanks to those who have responded in the past. Your feedback is being put to good use and has contributed towards enhancements such as superior performance, command preview, and better command support. It is extremely helpful to receive so much input!

Below are some new questions concerning file system diagnosis and disk management. We are interested in improvements we could make to either SAM or the command line. Please take a few minutes and help us understand how YOU do things and how we could work to enhance your experience. Your time and input, like always, is greatly appreciated!

The SAM team

===============

1. Concerning file system diagnosis, what are the issues that arise and how do you go about resolving them (i.e. check log files when thereâ s a full file system, check I/O performance of slow file system, etc.)?

2. We are looking into better ways to list file system information.

A. How would you like to see file systems listed?

B. What data do you like to see listed? Indicate all that apply but try to stick to those most important (top ~7):

a. mount point
b. source (device file)
c. mount options
d. file system type
e. total size
f. free space
g. used space
h. percent used
i. percent free
j. mounted/unmounted status (fstab vs. mnttab)
k. current mount options differ from fstab entry
l. date last written
m. other(s) (please specify)

C. Should this data be part of a new command, an enhancement to an existing command (which one), or is there no need for anything new or different than what is available today?

D. Should a listing include file systems that are not currently mounted?

3. SAM lists disk devices and logical volumes with their â usageâ (i.e. LVM, file system, swap, business copy) and allows you to manually â markâ them as used.

A. What information would you like to see in a listing of disks/LVs? Indicate all that apply but try to stick to those most important (top ~5):

a. device file or name
b. h/w path
c. number of paths
d. description (from ioscan)
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
g. driver
h. volume group name or mount point (with respect to what it is used for)
i. device type (i.e. cdrom, disk, optical, etc.)
j. instance or minor number
k. other(s) (please specify)

B. Do you use SAMâ s disk/LV marking facility (to mark it as in use)?

C. Would you value the extra checking for commands like newfs and swapon to refuse to overwrite a marked device, or would you consider this a nuisance (assume thereâ s an option to override)?

4. What could SAM or the HP-UX commands do to make disk array LUN management, and their configuration tools easier to use? Do you have any problems determining how a device file maps to the representati

Re: SAM Needs Disk and File System Input!

SAM TEAM — Mon, 17 Nov 2003 16:44:26 GMT

Sorry for the "Ã¢Â Â " throughout the original submission, thanks to a copy and paste from Word. Below is another submission of the questions with this fixed:

===============

1. Concerning file system diagnosis, what are the issues that arise and how do you go about resolving them (i.e. check log files when there's a full file system, check I/O performance of slow file system, etc.)?

2. We are looking into better ways to list file system information.

A. How would you like to see file systems listed?

B. What data do you like to see listed? Indicate all that apply but try to stick to those most important (top ~7):

a. mount point
b. source (device file)
c. mount options
d. file system type
e. total size
f. free space
g. used space
h. percent used
i. percent free
j. mounted/unmounted status (fstab vs. mnttab)
k. current mount options differ from fstab entry
l. date last written
m. other(s) (please specify)

C. Should this data be part of a new command, an enhancement to an existing command (which one), or is there no need for anything new or different than what is available today?

D. Should a listing include file systems that are not currently mounted?

3. SAM lists disk devices and logical volumes with their "usage" (i.e. LVM, file system, swap, business copy) and allows you to manually "mark" them as used.

A. What information would you like to see in a listing of disks/LVs? Indicate all that apply but try to stick to those most important (top ~5):

a. device file or name
b. h/w path
c. number of paths
d. description (from ioscan)
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
g. driver
h. volume group name or mount point (with respect to what it is used for)
i. device type (i.e. cdrom, disk, optical, etc.)
j. instance or minor number
k. other(s) (please specify)

B. Do you use SAM's disk/LV marking facility (to mark it as in use)?

C. Would you value the extra checking for commands like newfs and swapon to refuse to overwrite a marked device, or would you consider this a nuisance (assume there's an option to override)?

4. What could SAM or the HP-UX commands do to make disk array LUN management, and their configuration tools easier to use? Do you have any problems determining how a device file maps to the representation within the a

Re: SAM Needs Disk and File System Input!

SAM TEAM — Mon, 17 Nov 2003 16:47:20 GMT

One more correction; thanks for your patience. Looks like the last question was cut off:

4. What could SAM or the HP-UX commands do to make disk array LUN management, and their configuration tools easier to use? Do you have any problems determining how a device file maps to the representation within the array?

Re: SAM Needs Disk and File System Input!

Steven E. Protter — Mon, 17 Nov 2003 17:17:29 GMT

I don't use sam all that much accpet for a few tasks that I haven't learned on the command line or can't be done on the command line. My backup could use some enhancements, he doesn't do enough admin to do a lot of command line stuff.

1. Should be flags or field indicating fs status being full. You should be able to initiate a fsck -F from sam. That would be nice.

You should be ablet to display log files and current I/O rates on an fs or a disk or a logical volume.

2.
A. How would you like to see file systems listed?

lvol name mount point %full

B. What data do you like to see listed? Indicate all that apply but try to stick to those most important (top ~7):

a. mount point
b. source (device file)
c. mount options This should be available on more information click
d. file system type
e. total size
f. free space
g. used space
h. percent used
i. percent free
j. mounted/unmounted status (fstab vs. mnttab)
m. other(s) (please specify)

C. Current commands should remain the same. Perhaps these should be new command line options on the underlying commands.

D. Should a listing include file systems that are not currently mounted?

Yes

3. ?????

A. What information would you like to see in a listing of disks/LVs? Indicate all that apply but try to stick to those most important (top ~5):

a. device file or name
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
g. driver
h. volume group name or mount point (with respect to what it is used for)
i. device type (i.e. cdrom, disk, optical, etc.)

B. Do you use SAMÃ¢Â Â s disk/LV marking facility (to mark it as in use)?

No.

C. Would you value the extra checking for commands like newfs and swapon to refuse to overwrite a marked device, or would you consider this a nuisance (assume thereÃ¢Â Â s an option to override)?

Overrides are good for newbies. Perhaps give the admin the right to stop it if it gets annoying. A check box profile....

4. What could SAM or the HP-UX commands do to make disk array LUN management, and their configuration tools easier to use? Do you have any problems determining how a device file maps to the representation within the array?

Yes. fcmsutil is a pretty basic utility. It needs

Re: SAM Needs Disk and File System Input!

John Payne_2 — Mon, 17 Nov 2003 17:26:27 GMT

1) I use 'bdf' or 'df -k' to see filesystem utilitization. Log files are checked manually. (Not with SAM) They are also dealt with manually. (Not with SAM) Performance questions are looked at generally with Glance.

2) A. It's ok now.

B. a,e,j,c,m -> How about "Is filesystem size the same as logical volume size?

C. I think you are doing a pretty good job, if you make a change, please make an enhansement to an existing command, unless it is something drastic. (Post the new options here on the forums...)

D. definitely list non-mounted filesytems! (I assume you are talking bdf or df -k) It would be really nice to have it warn you something is unmounted, and I think would help out people that are always having problems with this...

3. A. e,b,a,c,h and i, if you want a 6th.

B. We do not use SAM's marking facility

C. If there is an option to override, go for it. I am all for checks and balances for things that can potentially destroy data...

4. We have the Compaq EVA on our HPUX machines and use the secure path drivers. Once the secure path drivers are understood and a user understands the 'spmgr' command, I think that is sufficient there, but that's just my opinion... (What we get now on the HPUX side is pretty much sufficient past that, but some of the changes you asked about in 2 and 3 will be nice there also.

Hope it helps

John

Re: SAM Needs Disk and File System Input!

Vijaya Kumar_3 — Mon, 17 Nov 2003 18:34:59 GMT

okay, i am with Steve too on this case. I wont use sam much ;) i used SAM last year (Sep 2002, to be exact) to export/import a VG...

Q1:
I definitely need a monitoring stuff with file systems. I use some shell scripts data to collect file system performance, usage and disk usage. I need to know whether "largefiles" enabled or disabled.
Some other info like:
-how many users/processes using the FS from lsof
-how busy the FS is...like I/O usage read and write info, info from iostat
-superblock info for HFS, which superblock it is using right now
-Automatically removing unnecessary files in a file system ('core' files, for example), what files are removed etc

Q2-A&B:
File system data:
I really like linux in putting the file system info with `df -ah` with some add ons...
-File System Device
-Size
-Used (in megs/gigs)
-Available (in megs/gigs)
-Available(in %)
-Mount Point
-Mounted (Yes/No)
-Mount Options

Q2-C:
I am not sure abt this...

Q2-D:
Yep, There should be... like Mounted - Yes/No
column.

Q3-A:
Logical Volume listing can be like:
-LV Name
-Size
-Used (in megs/gigs)
-Avail (in megs/gigs)
-Avail (%)
-FS Type
-HW/Path (primary/alternate)

Q3-B:
No, but it will be useful.

Q3-C:
No override. This can also be used from damaging it...

Q4:
I really like the this question. because i am facing it. I will face an issue most of the time looking for my device->LVM mapping. So there should be an easy way to find this. It can be like this:
LV's -> Disk/LUN Mapping
VG's -> Disk/LUN Mapping
I would be happy if u make it very clear to understand. i know it is hard.

Thanks
Vijay

Re: SAM Needs Disk and File System Input!

H.Merijn Brand (procura — Tue, 18 Nov 2003 05:13:13 GMT

1. Seldom have disk performance problems (wheew). But when we have them
a. Find the offending process(es)
b. Decrease disk usage (if possible)
c. Check file size of the offending access and decrease if possible
d. spread IO accross disks (RAID0 is an option)

And yes, I woul *L*O*V*E* to have a *FREE* defragmentation tool like AIX has in smit

2.
A. Easy answer: http://www.gentoo.com/di/
important is the ability to spot (nearly) full disks, and filter on local file systems vs. remote (NFS) file systems and soft shares like samba mount points.

B. b,a,d,g,h,i,m
m: number of remote locks (samba, cifs, nfs)

mount options are only interesting if it would prevent my to unmount a FS for whatever reason.

C. Just replace hp's bdf and df with di :)
and extend with extra options, like locks
Honoustly I don't care. I'm used to putting a (sym)link to the (b)df command that best fits my needs to a local bin folder that will be searched before the default path. I've always thought that shipping bdf AND df sucks. Just drop df and only ship bdf, but now call it df, and make it have di's functionality.

D. Only optional. Nice option, but my actual list already has 40 entries, so definitely optional.

3.
A. % is too difficult. I can (with pain) cut down to 6: a,b,d,e,f,i

B. no

C. since I don't use the 'marked' feature, I don't care

4. n/a

And a new utility (we're in SAM, right?) that would resamble TreeSize (http://www.jam-software.com/freeware/index.shtml) which would be like 'du -k' but with a graphical representation, ordering options, expand/collapse part of the tree and such would be nice

Enjoy, have FUN! H.Merijn

Re: SAM Needs Disk and File System Input!

Pete Randall — Tue, 18 Nov 2003 06:20:41 GMT

1. I agree with others that fs full vs lv full would be useful.

2A. By LV withing VG.
2B.
a. mount point
b. source (device file)
e. total size
i. percent free
j. mounted/unmounted status (fstab vs. mnttab)
m. other(s) (please specify): usage (swap/fs,etc.)
2C. I don't care
2D. Yes

3A. All of the Below:
a. device file or name
b. h/w path
c. number of paths
d. description (from ioscan)
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
g. driver
h. volume group name or mount point (with respect to what it is used for)
i. device type (i.e. cdrom, disk, optical, etc.)
j. instance or minor number
k. other(s) (please specify)
3B. No
3C. Sounds valuable to me.

4. It can be problematic to understand but I don't have any specific recommendation. Anything you could come up with would help, I think.

Pete

Re: SAM Needs Disk and File System Input!

Pete Randall — Tue, 18 Nov 2003 06:22:38 GMT

Oh! One other thing:

Don't even think about doing *anything* web based. kcweb is a disaster not worth repeating.

Pete

Re: SAM Needs Disk and File System Input!

Stefan Farrelly — Tue, 18 Nov 2003 07:10:01 GMT

1. Usual issues arrive - file systems filling up, not enough free space, slow performance etc. Resolving them; dmesg - output via cron regularly goes to email accounts and phones, we run a cron fscheck script also, once were notified by phone message or email we login and investigate and resolve manually. For performance always use glance/perfview.

2A. listed how they look in bdf, with a total for all filesystems at the bottom.

2B. Top 7;

mount point
source (device file)
mount options
file system type
total size
free space
used space

2C. an enhancement to an existing command, eg; bdf

2D. an option/switch to include those not mounted.

3A. Top 5;

a. device file or name
b. h/w path
d. description (from ioscan)
f. usage (i.e. vxfs, swap, LVM, etc.)
h. volume group name or mount point (with respect to what it is used for)

3B. No

3C. Yes, the extra checking would be good.

4. Yes, have problems sometimes determining how a device file maps with disk arrays. Not sure of the solution except some very sophisticated tool which does lots of checking and displays a nice visual map showing what is where and what can be used.

Re: SAM Needs Disk and File System Input!

Jon Mattatall — Tue, 18 Nov 2003 07:50:58 GMT

1. Full filesystems and vanishing nfs mounts.
Check with du -kx and find, manually manipulate any files or resize mountpoints/filesystems.

2a. mount/bdf are fine.
2b. a. mount point
b. source (device file)
f. free space
g. used space
d. file system type
e. total size
j. mounted/unmounted status
2c. Current commands are fine for me.
2d. Yes.

3a. e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
a. device file or name
h. volume group name or mount point
3b. No.
3c. Yes, and probably yes. Any check would be nice, and I'd also find it annoying. Doesn't mean it wouldn't be a good idea, though.

4. Mapping could be clearer. I stick with Grid Manager. (Model 10/20/30)

Re: SAM Needs Disk and File System Input!

H.Merijn Brand (procura — Tue, 18 Nov 2003 08:04:26 GMT

And of course we *all* want *F*R*E*E* OnLine JFS :):)

Where can we sign the petition?

/pro/local/share/elvis/elvis.ini

Re: SAM Needs Disk and File System Input!

Bill Hassell — Tue, 18 Nov 2003 08:39:13 GMT

1. There are several dozen postings in the ITRC Forums about full filesystems and most of them start out with "How do I find big files to remove?" which is the wrong question. The question should be: what are the biggest directories? The reason is that while an occasional big file may show up, a large directory may be the problem but contains thousands of small files. A very useful SAM enhancement:

du -kx /some_mountpoint | sort -rn

and display the first 10-20 lines. And for 'standard' HP-UX mountpoints, SAM's help on context could provide some guidelines on 'normal' sizes for common directories like / /dev /sbin /etc / and then some guidelines on /var /uss /tmp and /var.

2. bdfmegs
bdf is leftover from the days of megabyte disks so the Kbyte values are very hard to use. (bdfmegs has been attached to this posting) Being able to display the largefile flag and the JFS version is quite useful, along with the variable length format that ensures no split lines. A new command is probably needed since bdf has a Berkeley heritage, perhaps like nslookup and nsquery.

It would be useful in a new command to see umnounted disks that have VG identifers much in the way that SAM shows importable VGs. And unmounted filesystems would be quite useful as long as long disk hangs due to hardware problems can be avoided. Additionally, there should be a special entry in fstab called nomount (like the undocumented noauto) that can document raw filesystems. The new command would then report on these raw disks as 'used' rather than assume 'unused'.

3. All the details would be useful, some as part of 1-liner summaries, but by picking one lvol, all info can be shown. See Xvg as an example of detailed info. As far as using SAM's mark facility, I don't know anyone who uses it, perhaps due to lack of publicity. Is the mark visible/useful outside of SAM? Certainly having swapon, newfs, mkfs, even dd refuse to write over marked devices would be VERY useful as many Sybase and Informix users will attest. An override (like pvcreate -f) is of course required, with a second choice to unmark the disk/lvol then the command would work. In all cases, marking controls should be available on the command line as well as SAM.

4. LUN management is quite cumbersome, requiring the sysadmin to merge knowledge about HP-UX device files and ioscan info with the vendor's cryptic (and often PC-oriented) LUN definitions.

Re: SAM Needs Disk and File System Input!

SAM TEAM — Tue, 18 Nov 2003 12:58:56 GMT

Thought I would clarify a bit on a couple things.

When looking into new features, we are not only concerned with SAM but are also really looking to improve the command line experience. So, if you do not use SAM, you can still help with the end goal. First and foremost, we would like to see a complete command line.

A bit more info concerning question 3. When entering SAM's Disk Devices or Logical Volumes sub areas, all disks/LVs are retrieved and then SAM collects various system information about LVM/VxVM, file systems, swap, basically any way a disk can be "used" and then indicates this in the "Usage" column. In some cases (i.e. databases) a usage cannot be easily determined so SAM allows a user to "mark" a disk as used to essentially indicate the device is in use and do not use it. This way, when a user goes to something like the file systems area and tries to add a new one, the user is not presented with the marked device in the list of disks to choose from (list of unused disks).

Thanks, SAM Team

Re: SAM Needs Disk and File System Input!

Victor BERRIDGE — Tue, 18 Nov 2003 14:25:42 GMT

Hi,
since you can extend a logical volume online (if you have optionnal JFS-online) it would be nice to have the reduce also...
Just thoughts...

For the rest, I agree with the previous answers...(Yes bdf should display MBs)
In sam/VG it would be nice to see the pvs with I/O timeout values set (with max in ref.) and queue depth values...

All the best
Victor

Re: SAM Needs Disk and File System Input!

Geoff Wild — Tue, 18 Nov 2003 14:31:22 GMT

I mainly use the command line for disk activities.

1. Concerning file system diagnosis:

I check log files, sar, monitoring disk space via OVO.

2. We are looking into better ways to list file system information.

A. How would you like to see file systems listed?

I would like to see a nicely formatted bdf.
Currently, we do this with scripts.

B. What data do you like to see listed? Indicate all that apply but try to stick to those most important (top ~7):

a. mount point
b. source (device file)
e. total size
f. free space
g. used space
h. percent used
i. percent free
l. date last written

C. Should this data be part of a new command, an enhancement to an existing command (which one), or is there no need for anything new or different than what is available today?

Doesn't matter

D. Should a listing include file systems that are not currently mounted?

Yes, but should be optional.

3. SAM lists disk devices and logical volumes with their "usage" (i.e. LVM, file system, swap, business copy) and allows you to manually "mark" them as used

A:
a. device file or name
c. number of paths
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
h. volume group name or mount point (with respect to what it is used for)
i. device type (i.e. cdrom, disk, optical, etc.)

B: I do not use marking facility

C: Don't care

4. What could SAM or the HP-UX commands do to make disk array LUN management, and their configuration tools easier to use?

We use EMC so I don't think this would work for us - unless you could tie into them?
What would be nice is better reporting/capacity planning.

Rgds...Geoff

Re: SAM Needs Disk and File System Input!

Michael Tully — Tue, 18 Nov 2003 15:52:29 GMT

1. Size checking. As rightly suggested by Bill, we should be looking at directory sizes as well.

2 (a) Well my first thought would be reporting order ... I'd like to see options to report in volume group order, alphabetical without wrapping a script around it.
2 (b) all of them, but perhaps using options on h, j & k
2 (c) options for existing command
2 (d) option
3 (a) I don't use SAM for any type of disk/vg/lv management
3 (b) no
3 (c) yes
4 We use EMC
## One further thought. When software such as MirrorDisk/UX is not installed on a system and you try to mirror a volume, perhaps an error message like "MirrorDisk/UX is not installed" as opposed to a usage message of the lvextend command. I too think that MirrorDisk/UX and OnLineJFS should be free.

Re: SAM Needs Disk and File System Input!

David Ritchie — Tue, 18 Nov 2003 16:02:59 GMT

1) have ioscan ignore phantom LUN's from XP512 (and sam, and all the other utilities). By phanton LUN's, I referring to those marked as "HP DISK SUBSYSTEM" when in fact there is no disk there.

2) A quick way to map vg's into CU:ldev's on the XP512. (Similar things for Symmetrix would be useful also).

Here is a script that I wrote to do this in a basic sort of way...
------------------cut here ----------------
#!/usr/bin/sh

xpinfo -il | tail +6 >/tmp/xpinfo$$.out

for i in `echo /dev/*/group`
do
echo ::::::::::::::::
echo $i
echo ::::::::::::::::
for j in `vgdisplay -v \`dirname $i\` 2>/dev/null | grep "PV Name" | awk ' { print $3 } '`
do
grep `basename $j` /tmp/xpinfo$$.out | awk ' { print $1," ",$5," ",$6 } '
done
done
rm /tmp/xpinfo$$.out
---------------cut here----------------

3) A way other than 'strings /etc/lvmtab' to
display (and perhaps even interpret)
/etc/lvmtab's data structures. Conversely,
at least document the format via an .h file so that we can read and interpret it.

4) A way to both stripe and mirror simultaneously under LVM (without having to pay $30K for the priviledge). This is not possible currently under LVM, and would make my life much simpler. It would also allow for conversions between the two without bringing filesystems off line.

5) A way to pvmove a unstriped lvol out of a volume group that contains a striped lvol (this is not possbile currently.)

6) Fix sam so that when building striped lvols, you can pick the drive order to pull the stripes from. This may be possible by using pvg's, but it should not be necessary
to create a striped lvol.

7)document physical volume groups and how they are useful for striping...

8) document extent bases striping and why you would use this vs. lvm striping. When would you use this?

9) What are the optimal stripe sizes for various LVM volumes?

10) Write routines to manipulate the allocation and performance reporting on the
XP512... this is extremely disjoint currently.

11) Have enough knowledge of XP512 to recommend against striping within an array group (this is very bad from a performance standpoint).

Best regards,
Dave Ritchie

Re: SAM Needs Disk and File System Input!

Adisuria Wangsadinata_1 — Tue, 18 Nov 2003 23:28:44 GMT

A1. In the file system size checking, especially when the usage of the file system reach the limit. I used find command to resolve this problem. It will be nice if this can be happen also on the sam plus indication if the file system reach the limit.

A2A. bdf and if possible with MBytes or there's a converter for that.

A2B. here is the list :
a. mount point
b. source (device file)
d. file system type
e. total size
f. free space
g. used space
i. percent free

A2C. if the data just an additional option, you can use the old command & only put the new option in there. But if totally new, better put as the new command 8-).

A2D. Yes, it would be nice

A3A. here is the list :
a. device file or name
b. h/w path
e. size
f. usage (i.e. vxfs, swap, LVM, etc.)
g. driver

A3B. No

A3C. extra checking is nice as long as based on request, means the extra checking will be run if we need it. Cause if all checking plus extra checking comes in 1 display, it will makes headache 8-).

A4. Very difficult and complicated I guess, but any positive changes will help us to make it easier 8-).

Re: SAM Needs Disk and File System Input!

David Ritchie — Wed, 19 Nov 2003 11:35:16 GMT

a terse version of the verbose output for lvdisplay - so if you perform a lvdisplay -t /dev/vg01, it would look something like:

--- Logical Extents ---
LE's PV1 PV's
status
00000 - 00004 /dev/dsk/c4t0d0 00000 - 00004 current

instead of
--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t0d0 00000 current
00001 /dev/dsk/c4t0d0 00001 current
00002 /dev/dsk/c4t0d0 00002 current
00003 /dev/dsk/c4t0d0 00003 current

This would make it more readable by humans.