topic Re: Software Raid issue in Operating System - Linux

Software Raid issue

Shannon Petry — Wed, 07 May 2003 15:17:28 GMT

I recently had to rebuild a system. In the process I tar'd up 2 20GB files to my RH8.0 server onto a 2x64gb disk stripe of 128GB.

After deleting these files, I am now not able to add any data to the stripe. I get an error that the disk is full, even though it is only 6GB used out of 130GB. THis was verified with both du -sk, and df -k.

Since these were plain files (tar archives) I know that nothing would have had them open. Just to be sure, I rebooted the Linux server, and still can not create data in the stripe with the error "no space left on device"

[root@masterc 1b]# df -k .
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/md1 143347736 5902848 130276160 5% /1b

[root@masterc 1b]# touch test
touch: creating `test': No space left on device

I have looked with lsof, and see no open files on the device.

Any ideas?

Thanks!
Shannon

Re: Software Raid issue

Sachin Patel — Wed, 07 May 2003 16:27:39 GMT

Hi Shanon,

Check for quota
#repquota

If you don't want it just turn it off
#quotaoff

how is your raid configuration look like

#lsraid

Re: Software Raid issue

Steven E. Protter — Wed, 07 May 2003 17:52:08 GMT

Greetings Shannon,

Here is my output on a working raid 1/0 system.

[root@jerusalem root]# lsraid -A -g -a /dev/md0
[dev 9, 0] /dev/md0 E21817F0.29578AF2.8ED33C38.7B44A0A9 online
[dev 3, 2] /dev/hda2 E21817F0.29578AF2.8ED33C38.7B44A0A9 good
[dev 3, 66] /dev/hdb2 E21817F0.29578AF2.8ED33C38.7B44A0A9 good

Actual use of the lsraid command.

In your context I'd change it to.

lsraid -A -g -a /dev/md1

Hope this helps.

SEP

Re: Software Raid issue

Steven E. Protter — Wed, 07 May 2003 18:31:18 GMT

Shannon,

Would you be so kind as to post your results from the command. Just curious as to what can cause your results.

Thanks.

Steve

Re: Software Raid issue

Shannon Petry — Wed, 07 May 2003 18:32:37 GMT

quota's are not enabled

There is no such tool as lsraid on Redhat 8, I'll have to check 9. But here is the contents of /etc/raidtab.

raiddev /dev/md0
raid-level 0
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/sdc
raid-disk 0
device /dev/sdd
raid-disk 1

raiddev /dev/md1
raid-level 0
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/sda
raid-disk 0
device /dev/sdb
raid-disk 1

There is no issue that I can see with the raid device, the drivers are loaded, it's on line.

Regards,
Shannon

Re: Software Raid issue

Steven E. Protter — Wed, 07 May 2003 18:41:19 GMT

Here is my /etc/raidtab for comparison's sake.

[root@jerusalem root]# more /etc/raidtab
raiddev /dev/md7
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda1
raid-disk 0
device /dev/hdb1
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda5
raid-disk 0
device /dev/hdb5
raid-disk 1
raiddev /dev/md0
raid-level 1
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda2
raid-disk 0
device /dev/hdb2
raid-disk 1
raiddev /dev/md4
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda7
raid-disk 0
device /dev/hdb7
raid-disk 1
raiddev /dev/md6
raid-level 1
nr-raid-disks 2
chunk-size 64k
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda8
raid-disk 0
device /dev/hdb8
raid-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda3
raid-disk 0
device /dev/hdb3
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
raid-disk 0
device /dev/hdb3
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda6
raid-disk 0
device /dev/hdb6
raid-disk 1
raiddev /dev/md5
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda9
raid-disk 0
device /dev/hdb9
raid-disk 1

Are you sure about the lsraid command? I can find no desupport notice, though my resuls are admittedly off a red hat 7.3 server.

[root@jerusalem root]# find / -name lsraid
/sbin/lsraid
[root@jerusalem root]# ll /sbin/lsraid
-rwxr-xr-x 1 root root 67272 Apr 15 2002 /sbin/lsraid

Note that its not on the PATH of non-root users.

SEP

Re: Software Raid issue

Steven E. Protter — Wed, 07 May 2003 18:45:55 GMT

Red hat search on lsraid.

http://www.redhat.com/whitepapers/rhel/OracleonLinux.pdf

I'm not running oracle on this server, wonder where the darned thing came from.

Will do further research and try and find you an alternate command for checking raid status.

My guess is based on data, the response you posted and prior fun with Red Hat Raid is that there is a problem with raid.

Alternate solution: use fdisk, cfdisk or the gui graphical tool to blow away and re-create the filesystem, see what happens.

SEP

Re: Software Raid issue

Sachin Patel — Wed, 07 May 2003 19:38:41 GMT

Hi Shannon,

There is a command lsraid in 8.0.

Try raidstart. what is /proc/mdstat?

Is raidtool rpm install?

Sachin

Re: Software Raid issue

Steven E. Protter — Thu, 08 May 2003 00:47:29 GMT

Obviously the lsraid command is not installed on Shannon's system.

Here is where to get it.

http://rpmfind.net/linux/RPM/redhat/8.0/i386/raidtools-1.00.2-3.3.i386.html

Download it, install it, run the command and post the output, it should lead us to a cause.

A pleasure working with you. The answer would have been faster, but itrc was a dog today.

For you Shannon, I'd be happy to arrange a phone call to work this through(your dime)

stevenprotter@juf.org

SEP

Re: Software Raid issue

Steven E. Protter — Thu, 08 May 2003 00:49:13 GMT

Sorry, these tools are probably also on your CD's for red hat. That's where mine came from.

SEP

:-)

Re: Software Raid issue

Shannon Petry — Thu, 08 May 2003 11:46:10 GMT

I had the raidtools installed on the CD, but am trying to download the linked software and install this one.

I know there is no lsraid as I run slocate -u every day, and locate lsraid produced nadda ;)

Running raidstart is done at boot, note I have 2 separate Raid stripes, and both are mounted and working and active.

The active and working is verified by using "cat /proc/mdstat"

[root@masterc root]# cat /proc/mdstat
Personalities : [raid0]
read_ahead 1024 sectors
md1 : active raid0 sdb[1] sda[0]
143374592 blocks 64k chunks

md0 : active raid0 sdd[1] sdc[0]
71687168 blocks 64k chunks

unused devices:

Regards,
Shannon

Re: Software Raid issue

Shannon Petry — Thu, 08 May 2003 12:12:39 GMT

Installed the raidtools package, and kind of like cat /proc/mdstat here are the results.

[root@masterc tmp]# lsraid -A -g -a /dev/md1
[dev 9, 1] /dev/md1 E475ADD5.0CB7B814.11161BB8.D84F5860 online
[dev 8, 0] /dev/sda E475ADD5.0CB7B814.11161BB8.D84F5860 good
[dev 8, 16] /dev/sdb E475ADD5.0CB7B814.11161BB8.D84F5860 good

Regards,
Shannon

Re: Software Raid issue

Steven E. Protter — Thu, 08 May 2003 12:40:53 GMT

Okay, I'm stumped.

Perhaps remove and rebuild the filesystem, using the newly installed raid tools. If its empty, when its rebuilt, it should be fine.

I hate messing with Linux disk layout after its running, I'm an LVM man.

If I think of anything actually useful, I'll post.

SEP

Re: Software Raid issue

Shannon Petry — Thu, 08 May 2003 14:06:26 GMT

Yeah, Im stumped too. All looks okay, and I had no problems until I made those tar files. I was trying to avoid rebuilding the file system as this raid device has some production data on it.

fsck reveals no errors, so it's very strange.

I'll let you know if I resolve it ;/

Shannon

Re: Software Raid issue

Shannon Petry — Thu, 08 May 2003 14:50:47 GMT

Looks like it was the raidtools I was using that caused the problem. After updating, I unmounted and fsck'd. The fsck found a ton of errors and fixed everything up. Since the raidtools didnt mess with fsck, must be something else in the package.

Thanks for the nfo, and at least I dont have to rebuild the fs ;)

Shannon