copy a filesystem best solution (local host or remote host)

[Dennis Handly]

>I tested it, every error is "exit status" 1

Yes, that is the Unix standard, except in a few rare cases where it is documented.

>better to grep stderr  but  when the output maybe will change

I would think "directory busy" wouldn't change.  You can always look for just "busy".

[James R. Ferguson]

Hi:

If you want to take static copies of a filesystem while allowing users to continue to modify and/or read it, consider JFS snapshots.  This works on all recent releases and is documented here (for example on page-101):

http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c02023734/c02023734.pdf

Regards!

...JRF...

[support_billa]

thx for the information.

we will think about static copies of a filesystem.

but when we copy service guard filesystems, we cannot lock the server login's because other service guard package maybe want to login to the server .

i spoke about this issue with colleagues and we want to use following workflow:

 after umount filesystem
-  protect a directory "read only " : change permission to "000" (  save permissions before changing to "000")

before mount filesystem
- check with "fuser" , if directory before "mounting" is busy .

- set "save" permission

regards

[support_billa]

hello,

i put allmost all the input's of this thread in my copy script and at the end i want to make a litte compare statistics (named statistics_local_filesystems), i ask a lot here, how i can compare two filesystems , the best one will be a tool like "diff" for filesystems ....

i use : :

du -s <filesystem>
df -i  <filesystem>

count files and directories with "find"

md5sum

when i copy a empy filesystem ( we make a ORACLE RMAN ARCHIVE-LOG Backup before we start to copy, so i don't know , if the filesystem is empty ) with "cpio", then i have different statistics with "du" and "df"

the filesystems have the same LVOL SIZE !

in the attachment is the report  ( new filesystem is mounted with "/new/filesystem" )

why i don't use "dd" ..... we improve our "copy" plan because when you copy with "dd" a empty filesystem with about 100 GB, then i takes a little time ....

regards,

[James R. Ferguson]

---

@support_billa wrote:

i put allmost all the input's of this thread in my copy script and at the end i want to make a litte compare statistics (named statistics_local_filesystems), i ask a lot here, how i can compare two filesystems , the best one will be a tool like "diff" for filesystems ....

i use : :

du -s <filesystem>
df -i  <filesystem>

count files and directories with "find"

md5sum

...

---

A count of files and directories for the source and destination filesystems is probably adequate to confirm success.   Running and comparing checksums for equality ('cksum' or 'md5sum') is fine, and gives yet another level of assurance.  Running 'diff' works too.

You will *not* be able to compare filesystem or directory sizes.  Several factors affect this.  The first is the JFS block (fragment) size of the filesystem as set during its creation.  This may differ especially if the source filesystem was created on one VxFS (JFS) release and the destination filesystem was created on another (later) release.  Some of the rules have changed.

The second factor is that directory blocks allocated for files are never deleted when a file to which it points is removed.  These empty blocks are not copied and the destination directory will be smaller than the source directory for this reason.  This is probably the most common reason for differences.

A third reason involves *how* you copy and if you have sparse files.  A sparse file will be "inflated" if you use 'cp' to copy it, and hence comparing two filesystems with 'du' will yield different values.  Using tools like 'fbackup' or 'pax' circumvent this.

Regards!

...JRF...

[support_billa]

thanks a lot about the useful detail informations.

we use maybe different filesystem version , this will be one reason.

a explanation of "sparse" is also part of this thread. so it is better to use "pax" than "cpio".

i also will change my "statistics" filesystem report ....

regards

[support_billa]

hello,

a little question to compare statistics (named statistics_local_filesystems),

when i want to use diff" for directories it easy to compare on a local server.

but how i use "diff", when i copy a filesystem to a remote server ?

regards

[Dennis Handly]

>but how I use "diff", when I copy a filesystem to a remote server?

If it was over NFS it wouldn't be a problem.

[support_billa]

no nfs .... :-(((

[Dennis Handly]

>no nfs

You could use find and cksum and sort and compare those two.

[James R. Ferguson]

Hi:

As Dennis noted, generate two files to 'diff()'.  For the source and the destination directory, do the following:

# cd /srcpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/srcpath

# cd /dstpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/dstpath

# diff /tmp/srcpath /tmp/dstpath

By using the current working directory with 'find()' the ouput will be a relative name and you won't have to diddle with deriving the basename of files from different pathes.  The sort orders the files (necessarily) in name order, and the 'diff()' exposes any missing or differing files.

Regards!

...JRF...

[Dennis Handly]

>diff /tmp/srcpath /tmp/dstpath

If you want to use comm(1) you'll need to put the name column first.

[support_billa]

cd /srcpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/srcpath

this sounds so great , but i have following problem :

example :

cd /srcpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/srcpath

produces errors like

cksum: can't open ./subdir1/subdir2/subdir3/subdir4/filename.DAT: No such file or directory

the file is readable and from 2009 (but in many sub directories , it's an example) :

cd /srcpath

 ll ./subdir1/subdir2/subdir3/subdir4/filename.DAT
 -rw-rw-r--   1 prod       edv           1220 Aug 25  2009 ./subdir1/subdir2/subdir3/subdir4/filename.DAT

"find" command with "-exec ... \;" runs too long:

cd /srcpath && find . -type f -exec cksum {} \;  | sort -k3,3 > /tmp/srcpath

i found i one thread about "find" command with "-exec ...+;"

 'find' with "+" do not spawn an 'cksum' for every file it sees!
 Instead of using the '\;' terminator to '-exec' use '+'.

put "-exec ... \;" doesn't produces errors like "-exec ...+" , it runs too long ....

regards

[support_billa]

cd /srcpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/srcpath

is it a problem that filesystem /srcpath contains 90000 files ?

regards

[Dennis Handly]

>cd /srcpath && find . -type f -exec cksum {} + | sort -k3,3 > /tmp/srcpath

 >produces errors like
>cksum: can't open ./subdir1/subdir2/subdir3/subdir4/filename.DAT: No such file or directory

Are you root?

>the file is readable:
> -rw-rw-r--   1 prod       edv    1220 Aug 25  2009 ./subdir1/subdir2/subdir3/subdir4/filename.DAT

It sure looks like it is readable.  How long is that path?

>"find" command with "-exec ... \;" runs too long:

Why would you use \; when we told you to use "+"?

>put "-exec ... \;" doesn't produces errors like "-exec ...+" , it runs too long

What are you saying?  That cksum is broken and can't handle 1000s of files?

If that's the case, you can use xargs to limit the number of files.

>is it a problem that filesystem /srcpath contains 90000 files?

In one directory, or the whole filesystem?  For the latter, other than taking a long time, no.

[support_billa]

here my answers , ">>" i marked as your questions :

>cksum: can't open ./subdir1/subdir2/subdir3/subdir4/filename.DAT: No such file or directory

>>Are you root?

yes, copy process is only started with "root"
 
>the file is readable:
> -rw-rw-r--   1 prod       edv    1220 Aug 25  2009 ./subdir1/subdir2/subdir3/subdir4/filename.DAT

>>It sure looks like it is readable.  How long is that path?

here 2 examples (real path and real filename), which is report as "No such file or directory"

ll ./LN/spo-zlneu/prt/lsnr/000506620110091_520163_lang.lst
-rw-rw-r--   1 prod       edv          36359 Aug  4 11:32 ./LN/spo-zlneu/prt/lsnr/000506620110091_520163_lang.lst

ll ./LN/spo-zlneu/prt/fax/TAS118883.DAT
-rw-rw-r--   1 prod       edv          1220 Aug 25  2009  ./LN/spo-zlneu/prt/fax/TAS118883.DAT

 
>"find" command with "-exec ... \;" runs too long:

>>Why would you use \; when we told you to use "+"?

it was a test , sorry.

>put ?"-exec ... \;"? doesn't produces errors like "-exec ...+" ??, it runs too long

>> What are you saying?  That cksum is broken and can't handle 1000s of files?
>> If that's the case, you can use xargs to limit the number of files.

it is my opinion, why it works without problems with directories which contains 10000 files ...

>is it a problem that filesystem /srcpath contains 90000 files?

>> In one directory, or the whole filesystem?  For the latter, other than taking a long time, no.

the whole filesystem contains 39 directories , the deepest directory hierarchy are 6 subdirectories

regards

[Dennis Handly]

>here 2 examples (real path and real filename), which is report as "No such file or directory"

>ll ./LN/spo-zlneu/prt/lsnr/000506620110091_520163_lang.lst
-rw-rw-r--   1 prod       edv         36359 Aug  4 11:32 ./LN/spo-zlneu/prt/lsnr/000506620110091_520163_lang.lst
-rw-rw-r--   1 prod       edv           1220 Aug 25 2009 ./LN/spo-zlneu/prt/fax/TAS118883.DAT

I see nothing special/unusual.

>why it works without problems with directories which contains 10000 files ...

You may have to use tusc to trace the bad system call.

>the whole filesystem contains 39 directories , the deepest directory hierarchy are 6 subdirectories

About how many files in each directory?

[support_billa]

>> About how many files in each directory?

in the attachment is a detail report

>> You may have to use tusc to trace the bad system call.

i will do this as soon as possible

regards,

[James R. Ferguson]

Hi (again):

None of the directories you have shown look particularly large.  One thought that occurs to me is to ask if files can be removed frequently.  That is, the directory's by nature are volatile.

If between the time 'find()' finds a file and the time that *name* is passed to 'cksum()' if the file disappears, an error would be produced.  This may be what is happening.

Regards!

...JRF...

[support_billa]

hello,

why i am a litte bit busy .... a new issue occurs :

i have copied and will copy a lot of filesystems with "dd"

a few new questions :

- 1) how can i detect the actual highest possible filesystem layout version on a server (may HPUX 11.31) ?

- 2) how can i detect the filesystem layout version of filesystem ( i have 2 options )

- 3) how should i do the filesystem layout version upgrade ?

- 4) how i change to other filesystem options like "largefiles" ?

@ 1) how can i detect the actual highest possible filesystem layout version on a server (may HPUX 11.31) ?

 swlist ?

@ 2) how can i detect the filesystem layout version of filesystem ( i have 2 options )

option 1 with mounted filesystem : /sbin/vxupgrade
option 2 with lvol of filesystem : /usr/sbin/fstyp -v 

   /sbin/vxupgrade local_fs | \
   awk  '{ stat_pos_version=match($0,"[0-9] layout")
          len=2
          version=substr($0,stat_pos_version,len)
          printf "%s\n",version
         }' | read fstyp_version

    /usr/sbin/fstyp -v local_dev_file | \
    awk  '/version:/    {
        stat_pos=match($0,":")+1
        geslen=length($0)
        len=geslen-stat_pos
        printf "%s", substr($0,stat_pos+1,len) }' | read f_version
   

@  3) how should i do the filesystem layout version upgrade ?

with a loop like ?

 local_fs="/fs"
 local_fs_act_version=4
 local_fs_required_version=7

 typeset -i local_fs_change_version=0

 # Fuer Upgrade auf naechste Filesystem Version
 local_fs_change_version=$(( ${local_fs_act_version} + 1 ))

 while [ ${local_fs_change_version} -le ${local_fs_required_version} ]
 do
   if /sbin/vxupgrade -n ${local_fs_change_version} ${local_fs}  > /dev/null 2>&1
   then
     local_fs_change_version=$(( ${local_fs_change_version} + 1 ))
   else
     echo "/sbin/vxupgrade -n ${local_fs_change_version} ${local_fs} not OK !"
     local_fs_change_version=$(( ${local_fs_required_version} + 1 ))
   fi
 done

@ 4) how i change to other filesystem options like "largefiles" ?

 /usr/sbin/fsadm -F vxfs -o largefiles <filesystem> 

how can i find out , if it is a large filesystem ?

i use following command and parse for "f_flag", if it is greater than 0 it is a large filesystem ?

/usr/sbin/fstyp -v local_dev_file

regards

[James R. Ferguson]

---

@support_billa wrote:

a few new questions :

- 1) how can i detect the actual highest possible filesystem layout version on a server (may HPUX 11.31) ?

- 2) how can i detect the filesystem layout version of filesystem ( i have 2 options )

- 3) how should i do the filesystem layout version upgrade ?

- 4) how i change to other filesystem options like "largefiles" ?

---

Hi (again):

#1 : Use 'swlist' to ascertain the version of JFS you have installed:

# swlist | grep -i jfs

# 2/3 The JFS layout version can be obtained, on a filesystem basis thusly:

# fstyp -v /dev/vgNN/lvolX | grep -i version

 # 4 The state of 'largefiles' can be seen (easily) with:

# fsadm -v /mountpoint

To change from a 'nolargefiles' to a 'largefiles' filesystem, simply use 'fsadm':

# fsadm -F vxfs -o largefiles <device>

Regards!

...JRF...

[Dennis Handly]

>LN/spo-zlneu/prt/lsnr | 52097

This is the largest.  That would slow down commands like ls.

[support_billa]

# swlist | grep -i jfs
   Use 'swlist' to ascertain the version of JFS you have installed:

i tested it for

- HP-UX 11.31

swlist |grep -i jfs
B3929FB        B.05.00.02   OnlineJFS for Veritas File System 5.0 Bundle

- HP-UX 11.23
swlist |grep -i fs
Base-VXFS    4.1.002           VERITAS File System Bundle 4.1 for HP-UX
EnableVxFS   B.11.23.07     File-System library, commands enhancements for VxFS4.1 and 5.0

and how can it detect the highest possible filesystem layout version with "swlist" ?

another input is to parse "man vxupgrade"  like ?

man - vxupgrade |grep -i version

regards

[Dennis Handly]

>how can it detect the highest possible filesystem layout version with "swlist"?

I don't think you can directly do this with swlist.

[James R. Ferguson]

Hi (again):

---

@support_billa wrote:

and how can it detect the highest possible filesystem layout version with "swlist" ?

---

I should have been clearer when I pointed you to 'swlist'.  Once you know the VxFS/JFS product (e,g, 4.1 or 5.0) you need to consult the release notes to ascertain the possible layout versions.  Look at the Veritas FileSystem Release notes.   You should find, for example, that:

The VxFS 4.1 release supports layout version 6.

The VxFS 5.0 release brings layout version 7 and  does not support creating or mounting file systems using layout versions 1, 2, or 3 disk. layout.

VxFS 5.1 SP1 brings layout version 8.

Regards!

...JRF...