Software Raid issue

[Shannon Petry]

I recently had to rebuild a system. In the process I tar'd up 2 20GB files to my RH8.0 server onto a 2x64gb disk stripe of 128GB.

After deleting these files, I am now not able to add any data to the stripe. I get an error that the disk is full, even though it is only 6GB used out of 130GB. THis was verified with both du -sk, and df -k.

Since these were plain files (tar archives) I know that nothing would have had them open. Just to be sure, I rebooted the Linux server, and still can not create data in the stripe with the error "no space left on device"

[root@masterc 1b]# df -k .
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/md1 143347736 5902848 130276160 5% /1b

[root@masterc 1b]# touch test
touch: creating `test': No space left on device

I have looked with lsof, and see no open files on the device.



Any ideas?

Thanks!
Shannon

[Sachin Patel]

Hi Shanon,

Check for quota
#repquota

If you don't want it just turn it off
#quotaoff


how is your raid configuration look like

#lsraid

[Steven E. Protter]

Greetings Shannon,


Here is my output on a working raid 1/0 system.

[root@jerusalem root]# lsraid -A -g -a /dev/md0
[dev 9, 0] /dev/md0 E21817F0.29578AF2.8ED33C38.7B44A0A9 online
[dev 3, 2] /dev/hda2 E21817F0.29578AF2.8ED33C38.7B44A0A9 good
[dev 3, 66] /dev/hdb2 E21817F0.29578AF2.8ED33C38.7B44A0A9 good


Actual use of the lsraid command.

In your context I'd change it to.

lsraid -A -g -a /dev/md1

Hope this helps.

SEP

[Steven E. Protter]

Shannon,

Would you be so kind as to post your results from the command. Just curious as to what can cause your results.

Thanks.

Steve

[Shannon Petry]

quota's are not enabled

There is no such tool as lsraid on Redhat 8, I'll have to check 9. But here is the contents of /etc/raidtab.

raiddev /dev/md0
raid-level 0
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/sdc
raid-disk 0
device /dev/sdd
raid-disk 1

raiddev /dev/md1
raid-level 0
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/sda
raid-disk 0
device /dev/sdb
raid-disk 1

There is no issue that I can see with the raid device, the drivers are loaded, it's on line.

Regards,
Shannon

[Steven E. Protter]

Here is my /etc/raidtab for comparison's sake.

[root@jerusalem root]# more /etc/raidtab
raiddev /dev/md7
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda1
raid-disk 0
device /dev/hdb1
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda5
raid-disk 0
device /dev/hdb5
raid-disk 1
raiddev /dev/md0
raid-level 1
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda2
raid-disk 0
device /dev/hdb2
raid-disk 1
raiddev /dev/md4
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda7
raid-disk 0
device /dev/hdb7
raid-disk 1
raiddev /dev/md6
raid-level 1
nr-raid-disks 2
chunk-size 64k
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda8
raid-disk 0
device /dev/hdb8
raid-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda3
raid-disk 0
device /dev/hdb3
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
raid-disk 0
device /dev/hdb3
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda6
raid-disk 0
device /dev/hdb6
raid-disk 1
raiddev /dev/md5
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda9
raid-disk 0
device /dev/hdb9
raid-disk 1



Are you sure about the lsraid command? I can find no desupport notice, though my resuls are admittedly off a red hat 7.3 server.

[root@jerusalem root]# find / -name lsraid
/sbin/lsraid
[root@jerusalem root]# ll /sbin/lsraid
-rwxr-xr-x 1 root root 67272 Apr 15 2002 /sbin/lsraid

Note that its not on the PATH of non-root users.

SEP

[Steven E. Protter]

Red hat search on lsraid.

http://www.redhat.com/whitepapers/rhel/OracleonLinux.pdf

I'm not running oracle on this server, wonder where the darned thing came from.

Will do further research and try and find you an alternate command for checking raid status.

My guess is based on data, the response you posted and prior fun with Red Hat Raid is that there is a problem with raid.

Alternate solution: use fdisk, cfdisk or the gui graphical tool to blow away and re-create the filesystem, see what happens.

SEP

[Sachin Patel]

Hi Shannon,

There is a command lsraid in 8.0.

Try raidstart. what is /proc/mdstat?

Is raidtool rpm install?

Sachin

[Steven E. Protter]

Obviously the lsraid command is not installed on Shannon's system.

Here is where to get it.

http://rpmfind.net/linux/RPM/redhat/8.0/i386/raidtools-1.00.2-3.3.i386.html

Download it, install it, run the command and post the output, it should lead us to a cause.

A pleasure working with you. The answer would have been faster, but itrc was a dog today.

For you Shannon, I'd be happy to arrange a phone call to work this through(your dime)

stevenprotter@juf.org

SEP

[Steven E. Protter]

Sorry, these tools are probably also on your CD's for red hat. That's where mine came from.

SEP

:-)

[Shannon Petry]

I had the raidtools installed on the CD, but am trying to download the linked software and install this one.

I know there is no lsraid as I run slocate -u every day, and locate lsraid produced nadda ;)

Running raidstart is done at boot, note I have 2 separate Raid stripes, and both are mounted and working and active.

The active and working is verified by using "cat /proc/mdstat"

[root@masterc root]# cat /proc/mdstat
Personalities : [raid0]
read_ahead 1024 sectors
md1 : active raid0 sdb[1] sda[0]
143374592 blocks 64k chunks

md0 : active raid0 sdd[1] sdc[0]
71687168 blocks 64k chunks

unused devices:

Regards,
Shannon

[Shannon Petry]

Installed the raidtools package, and kind of like cat /proc/mdstat here are the results.

[root@masterc tmp]# lsraid -A -g -a /dev/md1
[dev 9, 1] /dev/md1 E475ADD5.0CB7B814.11161BB8.D84F5860 online
[dev 8, 0] /dev/sda E475ADD5.0CB7B814.11161BB8.D84F5860 good
[dev 8, 16] /dev/sdb E475ADD5.0CB7B814.11161BB8.D84F5860 good

Regards,
Shannon

[Steven E. Protter]

Okay, I'm stumped.

Perhaps remove and rebuild the filesystem, using the newly installed raid tools. If its empty, when its rebuilt, it should be fine.

I hate messing with Linux disk layout after its running, I'm an LVM man.

If I think of anything actually useful, I'll post.


SEP

[Shannon Petry]

Yeah, Im stumped too. All looks okay, and I had no problems until I made those tar files. I was trying to avoid rebuilding the file system as this raid device has some production data on it.

fsck reveals no errors, so it's very strange.

I'll let you know if I resolve it ;/

Shannon

[Shannon Petry]

Looks like it was the raidtools I was using that caused the problem. After updating, I unmounted and fsck'd. The fsck found a ton of errors and fixed everything up. Since the raidtools didnt mess with fsck, must be something else in the package.


Thanks for the nfo, and at least I dont have to rebuild the fs ;)

Shannon