Really Big RAMDisk?

[Richard Munn]

We have an application where there are about a dozen HP DL380's all hooked together with a GE network.

One of the nodes is a collector. I aquires data from a monitoring array in real time. The problem is, the data in rate is so fast, the 2.5 GB of data is collects in one hit can only be stored on a RAMdisk, magnetic disks are just too slow. The collector has 4GB RAM.

What we wanted to do then was NFS share the RAMDisk out to the other nodes who then pour over the data reducing it to the required state. The speed of this network link is not an issue as the 12 or so machines doing this a flat out compute bound. In fact we may even need more to improve the computation rate.

But I have a problem. With RAMfs ramdisks I can get the size OK but NFS will not share them. I guess this makes sense since both NFS and RAMfs sit as a VFS. The old RAM disks using /dev/ramN do work find on NFS if you pass mke2fs over them. I guess NFS likes the fs structure a bit better. But if you tell the kernel to use 3000000KB for ramdisks (it's OK with upto about 500MB) and it just will not play ball. This is too small. We also found even then you can do a mke2fs -v -F -m 0 /dev/ram1 300000 and it works but then a mount -t ext2 /dev/ram1 /rd1 may or may not. Sometimes it does, but others it will just bomb out with "mount: wrong fs type, bad option, bad superblock on /dev/ram1, or too many mounted file systems". Try it again and it may work.

BTW there are no other memory consumers running when all of this is going on apart from the odd ls command.

There has got to be a better way to do this???
Is there anyway to get RAMfs to work over NFS, can I get a big /dev/ramN device. Someone did suggest trying LVM or RAID to combine the limited /dev/ramN disks into a big volume. Is that the best way to go?

This is all very frustrating.

[Steven E. Protter]

NFS needs an actual filestructure.

Better off getting the fastest highest rotation speed storage you can get your hand on and set the buffers very high so a lot of the filesystem data is in memory.

Raid 1 on two independent controllers will limit bottlenecks.

SEP

[Richard Munn]

There is no mechanical disk fast enough to do this. Disks used to be used but the DMA capability of the array cards is so high now that the disks can't keep up and being able to get to the processing point faster is high on the required list. Not to mention, if the disks and the array cards are on the same bus, the array collectors will stomp all over the SCSI cards.

It could also be said that we could just point the collectors to a hugh DMA buffer then let it go. But, there is still a latency while the buffer is flushed to disk (again it's too slow) and the RAMdisk solution on a small size shows it's performance is adequate (although only just) to avoid undesirable latency.

[Claudio Cilloni]

A lot of linux distributions use to mount a tmpfs filesystem on /dev/shm.
tmpfs keeps all its data in ram, and it is usually big as half the ram installed in your machine (so it is 2G for you).
This filesystem is _really_fast_, but keep an eye on the swap activity because the kernel is happy to swap out tmpfs when applications need memory (you have other 2G to use for all the other things... maybe it's enough).

I don't know if nfs could export a tmpfs filesystem... If it doesn't, you can try to create a 2GB file into /dev/shm and mount it as a block device (using a loop device) and format it as ext3/reiser filesystem. This should like to nfs.

Ciao
Claudio

[Wim Van den Wyngaert]

Don't have a machine to test it but can't you export / (which is a real fs), mount it and then use your directory as a subdirectory of the mounted / ?

Wim

[Steven E. Protter]

I agree that regular disk can't do it. However a buffer cache hit ratio of say 99% will probably provide acceptable peformance.

There is also really expensive, solid state disk that I was once told about at an oracle seminar.

NFS can't do what you ask of it, so perhaps explore other ideas.

SEP