strange thing with disk utilization

[Srini Penumuchu]

This morning I am copying data from on mount point to a new one and one of the cp failed saying not enough space even when I create the mount point with the same space. I am on 11i.

Target mount point:
/alt_u59/oradata # find .
.
./ps1f
./ps1f/undotbs_02.dbf
./ps1f/undotbs_03.dbf
./ps1f/pstemp_02.dbf
./ps1f/pstemp_03.dbf
./ps1u
./ps1u/archive_11.dbf
./ps1o
./ps1r
eckpsdev:root:/alt_u59/oradata # du -k .
8183873 ./ps1f
4899 ./ps1u
0 ./ps1o
0 ./ps1r
8188772 .

Source mount point:

spenumuc:/u59/oradata:du -k .
6889497 ./ps1f
1024016 ./ps1u
0 ./ps1o
0 ./ps1r
7913513 .
u59/oradata:cd ps1f
u59/oradata/ps1f:ll
total 13778992
-rw-r----- 1 oracle dba 2095063040 Jul 10 14:15 pstemp_02.dbf
-rw-r----- 1 oracle dba 2095063040 Sep 12 13:37 pstemp_03.dbf
-rw-r----- 1 oracle dba 2095063040 Oct 2 10:03 undotbs_02.dbf
-rw-r----- 1 oracle dba 2095063040 Oct 2 10:03 undotbs_03.dbf
:spenumuc:/u59/oradata/ps1f:bdf|grep u59
/dev/data7/u59 8192000 7916621 258173 97% /u59
/dev/vg_data7/lv_u59
8192000 8191880 117 100% /alt_u59


Any clues?

[Michael Steele_2]

Need to see the following:

bdf /mount_point (* of old and new *)

lvdisplal /dev/vg##/lvol# (* refer to bdf for the logical volume or /etc/fstab *)

grep -i mount_point /etc/fstab

Also need to make sure you didn't mount to / (root). So also include:

bdf /

grep -i "oct 04" /var/adm/syslog/syslog.log | grep -i vmunix

dmesg | tail -10

[Michael Steele_2]

Need to see the following:

bdf /mount_point (* of old and new *)

lvdisplay /dev/vg##/lvol# (* refer to bdf for the logical volume or /etc/fstab *)

grep -i mount_point /etc/fstab

Also need to make sure you didn't mount to / (root). So also include:

bdf /

grep -i "oct 04" /var/adm/syslog/syslog.log | grep -i vmunix

dmesg | tail -10

[Hari Kumar]

If i correctly understood,
U please compare u r ps1f of the TARGET and SORCE as TARGET shows here 8183873 ./ps1f
but when u see the output of SOURCE is only
6889497 ./ps1f.
Plese check with " ll " and u can also use "diff" for the comparision ,bcoz the bdf output u have given shows KBYTES coloumn the mounting is OK with same size ,
May be this is due to recursive copy into subdirectories or any Application using it for Temporary files on that mount point.
If this does not work, please post ll output of ps1 of TARGET and SOURCE
Thanks,

[Michael Steele_2]

Also possible that you're creating 'sparse' files so run a 'cksum' on old and new.

cksum /old_mount_pt/file

cksum /new_mount_pt/file

Should see identical numbers.

[Srini Penumuchu]

Thanks for Ur quick response. In my posting,
please see the ll of source directory /u59.
There are 4 2 gig dbf files and du -k of this
mount point shows only around 6.8 Gig.

Please let me know if I still did not make it clear.

Thanks,
Srini

[Srini Penumuchu]

All gurus,
Please help! Respond to my issue...

Lambu

[Michael Steele_2]

Gee, still waiting for the 'lvdisplay' big guy.

[Alzhy]

Is it possible you have different Filesystem block sizes of /u59 and /alt_u59?
.
It is also possible that since you're not ROOT, you're unable to completely fill up your /alt_u59 as the 4 dbf files are already about 8183840 Kilobytes /and alt_u59 has 8192000 Kilobytes Capacity --- thereby giving you 8183840/8192000 = 99.9% Utilization...

[Srini Penumuchu]

SOURCE LVDISPLAY
/etc/lvdisplay /dev/data7/u59
--- Logical volumes ---
LV Name /dev/data7/u59
VG Name /dev/data7
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 8000
Current LE 2000
Allocated PE 2000
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default


TARGET LVDISPLAY
/etc/lvdisplay /dev/vg_data7/lv_u59
--- Logical volumes ---
LV Name /dev/vg_data7/lv_u59
VG Name /dev/vg_data7
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 10000
Current LE 1250
Allocated PE 1250
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict
IO Timeout (Seconds) default

/dev/data7/u59 /u59 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg_data7/lv_u59 /u59 vxfs rw,suid,largefiles,delaylog,datainlog 0 2

[Michael Steele_2]

Refer to the lvdisplay data posted above, and what I'v pasted below. VG 'data7' has a PE size of 4 mb while VG 'vg_data7' use 8 mb.

8000 / 2000 = 4

10000 / 1250 = 8

Confirm this with 'vgdisplay'.

vgdisplay data7
vgdispaly vg_data7

##########################################
##
##########################################

"Bad block on" for /dev/data7/u59 but "Bad block off" for /dev/vg_data7/lv_u59.

##########################################
##
##########################################

nolargefiles for /dev/data7/u59
largefiles for /dev/vg_data7/lv_u59

##########################################
##
##########################################

LV Name /dev/data7/u59
LV Size (Mbytes) 8000
Current LE 2000
Allocated PE 2000
Bad block on
nolargefiles

LV Name /dev/vg_data7/lv_u59
LV Size (Mbytes) 10000
Current LE 1250
Allocated PE 1250
Bad block off
largefiles

##########################################
##
##########################################

Are you on a disk array for
/dev/vg_data7/lv_u59?

[Bill Hassell]

The problem is definitely sparse files. cksum will produce the same result but the target will be larger. What is happening is that these are probably database files and they were created using lseek ten write. For example, the file is created, then one record is written to position 1 and another written to position 1,000,000 then the file is closed. The directory is updated that there are one million records but only two occupy any space. When you use cp (or tar, cpio, pax, ftp, etc), the file will be read serially and the unwritten records will be returned as a string of zeros (conveniently supplied by the filesystem code). But the target is being written serially and the stream of zeros becomes real data, so the copy is nearly ne million times bigger than the source. NOTE: Both files are equivalent and produce the same result. The missing million records in the source will always be zeros until more data is written.

fbackup is the only tool that can recover a sparse file (more accurately, see streams of zeros and lseek around them to conserve space). Note that the source file will continue to grow in occupied space as middle records are written. It is VERY possible that the million record file will be too large to fit in the current lvol as middle records are written.

[Bill Hassell]

I said that fbackup can recover sparse files but really, it is the fbackup/frecover pair. fbackuo creates a data stream compatible with frecover and the -s option for frecover will skip over long streams of zero data (man frecover).

[Alzhy]

Srini,

Give us outputs of:
.
fstyp -v /dev/data/ru59
fstyp -v /dev/vg_data/lv_u59
.
There might be a clue in there....

[Srini Penumuchu]

SOURCE:

fstyp -v /dev/data7/u59
vxfs
version: 4
f_bsize: 8192
f_frsize: 1024
f_blocks: 8192000
f_bfree: 275379
f_bavail: 258168
f_files: 68876
f_ffree: 68844
f_favail: 68844
f_fsid: 1074200580
f_basetype: vxfs
f_namemax: 254
f_magic: a501fcf5
f_featurebits: 0
f_flag: 0
f_fsindex: 7
f_size: 8192000


TARGET:

fstyp -v /dev/vg_data7/lv_u59
f_bsize: 8192
f_frsize: 1024
f_blocks: 10240000
f_bfree: 1028497
f_bavail: 96421600
f_files: 2571567
f_ffree: 2571246
f_favail: 257124
f_fsid: 1074331652
f_basetype: vxfs
f_namemax: 2541652
f_magic: a501fcf5
f_featurebits: 0
f_flag: 16501fcf5
f_fsindex: 7s: 0
f_size: 10240000

I did not quite understand sparse files phenomenon. I will spend more time grasping this but my basic question was,

du -k .
6889497 ./ps1f
1024016 ./ps1u

Then how come ll shows,

ll
total 13778992
-rw-r----- 1 oracle dba 2095063040 Jul 10 14:15 pstemp_02.dbf
-rw-r----- 1 oracle dba 2095063040 Sep 12 13:37 pstemp_03.dbf
-rw-r----- 1 oracle dba 2095063040 Oct 2 10:03 undotbs_02.dbf
-rw-r----- 1 oracle dba 2095063040 Oct 2 10:03 undotbs_03.dbf
:spenumuc:/u59/oradata/ps1f:bdf|grep u59

I hope I am not bothering you by repeating the same question....

Lambu

[Michael Steele_2]

I don't suppose NFS is involved?

Latest patches for 'du': PHNE_28983 and PHNE_28103 if NFS related.

Regarding different PE_SIZE's this is similar to fragmentation.

Regarding 'largefile' parameter, affects files > 2 GB. Could be the reason for 'sparse' files.

Did you try 'cksum' yet?