- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- copy a filesystem best solution (local host or re...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-16-2011 01:27 PM
05-16-2011 01:27 PM
Re: copy a filesystem best solution (local host or remote host)
But, there is one more to check if you are using autofs and that would be /etc/auto_mounter.
You also have one more problem: A problem that is becomming more common in today's world of 100 to 1 servers per admin - Sometimes the suddirectory / mount point is missing - Which would be a reason for failing to mount - You see most often on a failover node in a VCS or MCSG cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-16-2011 10:58 PM
05-16-2011 10:58 PM
Re: copy a filesystem best solution (local host or remote host)
i did filesystem tests with following commands:
CP...... command "cp"
CPIO.... command "cpio"
DD...... command "dd" (RAW-Device)
PAX..... command "pax"
TAR..... command "tar"
VXDUMP.. command "vxdump" (two options)
in the attachment is a report of my test.
the winner is command "dd".
Fs-Test ... Small : 87489 Small Files
Large : 1 Big File about 10.61 GB
in the table below a overview of times of the commands
Fs-Test | CP | CPIO | DD | PAX | TAR | VXDUMP 1| VXDUMP 2
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Small | 1:30.7 | 1:54.0 | 0:25.5 | 1:54.5 | 3:56.8 | 2:42.7 | 4:49.6 |
Large | 1:53.0 | N.P. | 1:52.5 | 6:31.8 | N.P. | 4:20.6 | 5:25.1 |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-16-2011 11:48 PM
05-16-2011 11:48 PM
Re: copy a filesystem best solution (local host or remote host)
here is script with all the input's of here ( i hope i have done it well) for example copy a filesystem with command "cpio"
fs_cp_cpio_local_host.sh
option : -v
-n
-f .... directory is a filesystem
./fs_cp_cpio_local_host.sh -v /old_fs -n /new_fs -f
Report :
Logfile: /tmp/fs_cp_cpio_local_host.sh.20110517_092423.log
FS OLD: /old_fs
drwxr-xr-x 4 root root 8192 May 16 12:38 /old_fs
FS NEW: /new_fs
drwxr-xr-x 4 root root 8192 May 16 12:38 /new_fs
Statistics
Filesystem | Count. Files. | Count. Directories | du FS
-----------------------------------------------------------------------------------
FS OLD
/old_fs | 26 | 87489 | 2704528
FS NEW
/new_fs | 26 | 87489 | 2704528
when you want to debug:
DEBUG=ON ./fs_cp_cpio_local_host.sh ....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 06:17 AM
05-17-2011 06:17 AM
Re: copy a filesystem best solution (local host or remote host)
When you migrate from one file system to another you want to verify everything got to the destination. So you do a df -i and check if they match.
Also, cpio is limited to files smaller than 2GB. That's why its obsolete.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 06:22 AM
05-17-2011 06:22 AM
Re: copy a filesystem best solution (local host or remote host)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 07:09 AM
05-17-2011 07:09 AM
Re: copy a filesystem best solution (local host or remote host)
thx, i added to my checks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 07:16 AM
05-17-2011 07:16 AM
Re: copy a filesystem best solution (local host or remote host)
are those commands ok for remote copy ?
cd /old_fs
local :
pax -r -w -pe . /new_fs
remote : ?
pax -w . | ssh server " ( cd /new_fs ; pax -r -pe )"
local :
vxdump 0f - /old_fs | (cd /new_fs ; vxrestore xf -)
remote : ?
vxdump 0f - /old_fs | ssh server "(cd /new_fs ; vxrestore xf -) "
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 07:30 AM
05-17-2011 07:30 AM
Re: copy a filesystem best solution (local host or remote host)
vxvm with diskgroup split and join, thin provisioning luns and smartmove functionality, should beat the dd time easily.
and cost shouldnt be an issue as the tests were done on a xp10k. ;)
Greetz,
Chris
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2011 11:27 AM
05-17-2011 11:27 AM
Re: copy a filesystem best solution (local host or remote host)
-or-
cp -p -r /source /dest
-then-
df -i /source /dest
The df -k command is just like bdf - you get a report in bytes utilized.
When you write a script counting files and sub directories you are in essence counting inodes. So use the df -i command which does the counting for you and verify success when the /source inode total = the /dest inode total.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-18-2011 12:51 AM
05-18-2011 12:51 AM
Re: copy a filesystem best solution (local host or remote host)
first command doesn't work,
second comand point out errors, but works ....
cd /old_fs
/usr/sbin/vxdump -0f - /old_fs | ssh host '( cd /new_fs ; /usr/sbin/vxrestore -xf - )'
vxfs vxdump: Date of this level 0 dump: Wed May 18 10:42:49 2011
vxfs vxdump: Date of last level 0 dump: the epoch
Dumping /dev/vgfs/rold_fs to stdout
vxfs vxdump: mapping (Pass I) [regular files]
UX:vxfs vxrestore: ERROR: V-3-20068: cannot open /dev/tty: No such device or address
vxfs vxdump: mapping (Pass II) [directories]
vxfs vxdump: estimated 1948264 blocks (951.30MB) on 6.33 tape(s).
cd /old_fs
/usr/sbin/vxdump -0 -f - -s 1000000 -b 16 /old_fs | ssh host '(cd /new_fs; /usr/sbin/vxrestore -rf -) '
UX:vxfs vxrestore: ERROR: V-3-20068: cannot open /dev/tty: No such device or address
vxfs vxdump: Date of this level 0 dump: Wed May 18 10:45:29 2011
vxfs vxdump: Date of last level 0 dump: the epoch
Dumping o stdout
vxfs vxdump: mapping (Pass I) [regular files]
vxfs vxdump: mapping (Pass II) [directories]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-18-2011 07:33 AM
05-18-2011 07:33 AM
Re: copy a filesystem best solution (local host or remote host)
@ OK - The count of files and subdirs looks OK - Still, just do a df -i.
1) i got an input : is md5sum a way to check two filesystems ?
i tested it and it seems ok:
/usr/bin/md5sum /old_fs
b0292b7e9f9439d25be786f99b22587d /old_fs
/usr/bin/md5sum /io_dst_test
b0292b7e9f9439d25be786f99b22587d /new_fs
when i made a change, the fingerprint changes !
2) when i want to log for example "cpio" output in another logfile in a shell script, this is the only way ?
exec 3>> /tmp/logfile
find . ! -path ./lost+found -depth -print | cpio -padlmuv 1>&3 2>&3 >&3
regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-18-2011 05:00 PM
05-18-2011 05:00 PM
Re: copy a filesystem best solution (local host or remote host)
> i tested it and it seems ok:
> /usr/bin/md5sum /old_fs
b0292b7e9f9439d25be786f99b22587d /old_fs
> /usr/bin/md5sum /io_dst_test
b0292b7e9f9439d25be786f99b22587d /new_fs
> when i made a change, the fingerprint changes !
What you are doing is creating a unique number (a checksum) from the information in the directory. The directory has a list of pointers to inodes and a set of names. Add, delete or change a file in the old_fs directory and the checksum will change. But add, delete or change a file in a subdirectory (/old_fs/testing direcory) and the top level directory (/old_fs) is the same.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2011 03:20 AM
05-19-2011 03:20 AM
Re: copy a filesystem best solution (local host or remote host)
There seems to be a gap in you knowledge about inodes; there are always 4 - 0, 1, 2,and a pointer to the disk address. 0, 1 and 2 are also known as STDIN, STDOUT and STDERR.
When you do a :
exec 3 > /tmp/file
You are over riding the OS and manually assigning STDOUT. When you want to do this I'm sure its fine but also unnecessary and confusing to the next person who comes along and maintains your work after you leave.
The standard in the industry is STDOUT = 1. For example, I'm sure you've seen this:
cat file > logfile 2>%1
What does this mean?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2011 10:30 AM
05-19-2011 10:30 AM
Re: copy a filesystem best solution (local host or remote host)
Did you mean 2>&1 ?
bv
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2011 04:56 PM
05-19-2011 04:56 PM
Re: copy a filesystem best solution (local host or remote host)
Yes, thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2011 11:01 PM
05-19-2011 11:01 PM
Re: copy a filesystem best solution (local host or remote host)
@cat file > logfile 2>&1
STDOUT is directed to "logfile" and STDERR is directed to STDOUT, in this case "logfile" !
@sure its fine but also unnecessary and @confusing to the next person who comes along @and maintains your work after you leave.
what i want is :
- a shell script copy files from fs A to fs B.
- a shell script do all a lot of tasks, we discussed it above, this tasks i want to log
- in the shell script is one task : the "copy" command, which produces thousands of lines and this task i want to log in a external logfile
- after the "copy" command , i have to do the finishing tasks, we discussed it also above
is this idea confusing ?
this shell script will start not only the shell script programmer, also other persons.
regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2011 04:49 AM
05-20-2011 04:49 AM
Re: copy a filesystem best solution (local host or remote host)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2011 04:50 AM
05-20-2011 04:50 AM
Re: copy a filesystem best solution (local host or remote host)
The simple
> file 2>&1
to make sure that err output is included in the file
and
cmd1 2>&1 | cmd2
to make sure that err output is included in the input to cmd2,
are used a lot.
Nothing wrong with using more than the usual 0,1,2 file descriptors and it can be useful.
However, it is an advanced concept, and, for the casual peruser, it can take a while to figure out what's going on.
OTOH, well documenting the script at the beginning takes care of that issue.
In a fairly complex script, you might have several different log files to which output will be sent at various times by various commands.
You could accomodate that in at least two ways
1) set up multiple FDs
exec 3>/tmp/lg1
exec 4>/tmp/lg2
exec 5>/tmp/lg2e
and some commands might use use
acmd >&3 2>&1
while others might use
bcmd >&4 2>&5
This method has the quirk (or advantage) of automatically appending output since the FDs are opened thoughout the script's process (unless closed, of course).
You can see this with this scriptlet:
## (
>/tmp/l1
exec 3>/tmp/l1
echo gkgkgkg >&3 # to FD 3 with no >>
cat /tmp/l1
echo --
echo hhhhhkg >&3 # to FD 3 with no >>
cat /tmp/l1
echo --
ll /tmp/l2 /tmp/xxl1 >&3 2>&1
cat /tmp/l1
)
2) but, in most cases, you could also just set up some variables
LOG1=/tmp/lg1
LOG2=/tmp/lg2
LOG2e=/tmp/lg2e
and manually append them:
> $LOG1 ; > $LOG2 ; > $LOG2e # init them
acmd >> $LOG1 2>&1
bcmd >> $LOG2 2>>$LOG2e
With "disadvantage" of requiring '>>',
but with the advantage of being more readable and maybe self-documenting.
Another scriptlet:
## (
LOG1=/tmp/lg1
LOG2=/tmp/lg2
LOG2e=/tmp/lg2e
> $LOG1 ; > $LOG2 ; > $LOG2e # init them
ll /tmp/l2 /tmp/xxl1 >> $LOG1 2>&1
vgdisplay >> $LOG2 2>>$LOG2e
echo ------------LOG1--------------------------
cat $LOG1
echo ------------LOG2--------------------------
tail -5 $LOG2
echo ------------LOG2e-------------------------
cat $LOG2e
echo ===========
echo ===========
ll /tmp/l3 /tmp/aal1 >> $LOG1 2>&1
ll /tmp/l3 /tmp/aal1 >> $LOG2 2>>$LOG2e
echo ------------LOG1--------------------------
cat $LOG1
echo ------------LOG2--------------------------
tail -5 $LOG2
echo ------------LOG2e-------------------------
cat $LOG2e
)
I like that you used it -- shows that you are unafraid.
And, it's opened up another concept for discussion!
bv
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2011 06:04 AM
05-20-2011 06:04 AM
Re: copy a filesystem best solution (local host or remote host)
> are unafraid.
Some people who learned to write computer
programs other than shell scripts may already
be familiar with file-I/O concepts like
"open". C open() and fopen(), or Fortran
OPEN(), for example.
> However, it is an advanced concept, [...]
Not really. It's just funny-looking in a
shell script, because the word "open" does
not appear anywhere, and because many things
can be done using the usual three pre-opened
"files" (stdin, stdout, stderr).
Once, upon a time long ago, I was writing a
shell script, and knew that I wanted to open
a file, but I spent a lot of time trying to
find out how to do it. The "man sh"
explanation of "exec" didn't reveal very
much, as I recall. On the bright side, it
seems to be true that the worse the ordeal to
find an answer, the harder it is to forget
it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2011 06:24 AM
05-20-2011 06:24 AM
Re: copy a filesystem best solution (local host or remote host)
Good points
> Not really.
Well, I meant that it is advanced within the context of shell scripting.
I would bet that most Unix admins are not familiar with it, but I could be wrong.
>it seems to be true that the worse the ordeal to find an answer, the harder it is to forget it.
That's why I love troubleshooting !
bv
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2011 05:16 AM
08-06-2011 05:16 AM
Re: copy a filesystem best solution (local host or remote host)
hello,
after copied a lot of of filesystems , i have to extend my copy script ...
maybe some users have solutions for following situations :
- i "umount" filesystem(example umount /fs )
info's: users can log into the server ....
a user log in and change to directory /fs
after copy (with command "dd" ) , i want to "mount" filesystem , i doesn't work , because directory is busy
can i protect a directory "read only " , like a "read only" filesystem ? maybe i save permissions , and the i change permission to "000" after "umount" and then i change permissons before i "mount" ?
or must i search with example "fuser" , if directory before "mounting" is busy ?
- where can i find the "exit codes" for "umount" and "mount" ? like exit 1 is "directory busy" ...
- i created a thread nfs mount in service guard, because , when i want to "umount" and "fuser" doesn't show a process, you can not "umount" , when it is a "nfs share" ...
- also i created a thread "check with a script open file(s) of a filesystem" because it isn't easy to work with "fuser" (find a process and handle it in a shell script )
why i ask here so many questions : because i have to copy in little "service window" many filesystem(s) in a service guard environment and my script should near perfect ... last friday i problems like above ... and the my work took longer then i expect ...
a little background : i have to copy about 1000 filesystems (because storage migration to a new box and a new lvm version is also installed with copy process )
regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2011 05:19 AM
08-06-2011 05:19 AM
Re: copy a filesystem best solution (local host or remote host)
additional information: after "itrc" merge i use now user "support_billa" (old_user : Billa-User)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2011 05:47 AM - edited 08-29-2011 01:29 AM
08-06-2011 05:47 AM - edited 08-29-2011 01:29 AM
Re: copy a filesystem best solution (local host or remote host)
>after "itrc" merge I use now user "support_billa" (old_user : Billa-User)
You could ask the moderators to change your screen name, rather than have two different profiles.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2011 05:58 AM - edited 08-29-2011 01:29 AM
08-06-2011 05:58 AM - edited 08-29-2011 01:29 AM
Re: copy a filesystem best solution (local host or remote host)
>can I protect a directory "read only ", like a "read only" filesystem? Maybe I save permissions and the I change permission to "000" after "umount" and then I change permissions before I "mount"?
Yes, you could remove the permissions and then put them back.
>or must I search with example "fuser", if directory before "mounting" is busy?
That maybe be easier, nothing to remember to put back.
>- where can I find the "exit codes" for "umount" and "mount" ? Like exit 1 is "directory busy"
The man page maybe all you get. (And it doesn't mention any exit status. )-:
Or grep the stderr output.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2011 06:06 AM
08-06-2011 06:06 AM
Re: copy a filesystem best solution (local host or remote host)
thx,
>The man page maybe all you get. (And it doesn't mention any exit status. )-:
> Or grep the stderr output.
i tested it, every error is "exit status" 1 ....
better to grep stderr ... but , when the output maybe will change ....