Operating System - HP-UX
1748089 Members
5046 Online
108758 Solutions
New Discussion юеВ

Re: Lost all datafiles in Database Server!

 
Tung Dang
Advisor

Re: Lost all datafiles in Database Server!

Dear Mr.Verschuren,
Thanks for your help.
The attachment is the syslog.conf and dmesg.log. Pls check it.

Thanks and Best Regards,
Tung
Tung Dang
Advisor

Re: Lost all datafiles in Database Server!

Dear Sirs,
This is the syslog.conf.
check the attachment pls.

Tung
F Verschuren
Esteemed Contributor

Re: Lost all datafiles in Database Server!

it looks like a lost of luns, please post a :
ioscan -fnC disk
ioscan -fnC fc
and vgdisplay -v

to check all the disks that are configured are still availeble.
Yogeeraj_1
Honored Contributor

Re: Lost all datafiles in Database Server!

hi Tung,

based on the messages in your alert.log, it seems that the /CCBS2DB/ mount point is missing.

where is this mounted?

do a cat /etc/fstab to get more information.


revert
kind regards
yogeeraj
No person was ever honoured for what he received. Honour has been the reward for what he gave (clavin coolidge)
Tung Dang
Advisor

Re: Lost all datafiles in Database Server!

Dear Sirs,
Thank you for your help.
We are currently using SAN environment with EVA8000, and configuring these volumn group in cluster. The briefly configuration as following:
/dev/vd09/lvol1 /CCBS2DB/backup
/dev/vd01/lvol1 /CCBS2DB/oradata1
/dev/vd01/lvol2 /CCBS2DB/oraindex1
/dev/vd01/lvol3 /CCBS2DB/system
/dev/vd03/lvol1 /CCBS2DB/archive
At the time the problem happened, I used bdf I saw all of these mountpoint still remain, but the "used" of /CCBS2DB/backup, /CCBS2DB/oraindex1, /CCBS2DB/system and /CCBS2DB/archive were 0%; and the "used" of /CCBS2DB/oradata1 was seemly normal. It means that all the directories were still remain, just lost the files and directories inside it.
Now I recovered the database from tape already, and the system run normally. So I think that ioscan and vgdisplay this time do not help us to find out the cause of the problem. If you still want to look it, let me know.
I want to know the root cause of the problem in order to prevent it happen again.
Thank you very much.
Tung
Eric Antunes
Honored Contributor

Re: Lost all datafiles in Database Server!

Hi,

Just one though about discovering what happened:

Login as root and do some "Esc+K" to check the root shell history.

Best Regards,

Eric Antunes

Each and every day is a good day to learn.
Anshumali
Esteemed Contributor

Re: Lost all datafiles in Database Server!


I doubt it will work. I hope root command's history isn't logged by default.

Dreams are not which you see while sleeping, Dreams are which doesnt allow you to sleep while you are chasing for them!!
Eric Antunes
Honored Contributor

Re: Lost all datafiles in Database Server!

If you can't "Esc+k", you can always do as root:

vi $HOME/.sh_history


vi
Each and every day is a good day to learn.
Tung Dang
Advisor

Re: Lost all datafiles in Database Server!

Dear Eric,
I checked .sh_history already, and there was no any "rm" or "lvreduce" command related with this problem.

Rgds,
Tung
Kapil Jha
Honored Contributor

Re: Lost all datafiles in Database Server!

Dear Tung,
sorry for late reply....
There is nothing in alert files they direct up to some unix error.


Error: ORA 1171
Text: datafile is going offline due to error advancing checkpoint
-------------------------------------------------------------------------------
Cause: The checkpoint in the file header could not be advanced.
See accompanying messages for the reason.
The datafile will be taken offline the same as for a write error of a
data block.
Action: See accompanying messages for details.
Restore access to the file, do media recovery, and bring it back
online.


ORA-01115: IO error reading block from file string (block # string)
Cause: Device on which the file resides is probably offline
Action: Restore access to the device



in syslog file some FTP activity was there at the time of crash
Dec 5 15:13:24 COVIS2 ftpd[27915]: USER oss
Dec 5 15:13:24 COVIS2 ftpd[27915]: PASS password
Dec 5 15:13:23 COVIS2 ftpd[29657]: PWD
Dec 5 15:13:24 COVIS2 above message repeats 29 times
Dec 5 15:13:24 COVIS2 ftpd[27915]: FTP LOGIN FROM 10.1.103.31 [10.1.103.31], oss
Dec 5 15:13:24 COVIS2 ftpd[27915]: TYPE ASCII
Dec 5 15:13:24 COVIS2 ftpd[27915]: CWD /ccbs2/stats_raw/
Dec 5 15:13:24 COVIS2 ftpd[27915]: PORT
Dec 5 15:13:24 COVIS2 ftpd[27915]: NLST
Dec 5 15:13:24 COVIS2 ftpd[27915]: QUIT
Dec 5 15:13:24 COVIS2 ftpd[27915]: FTP session closed
Dec 5 15:13:24 COVIS2 ftpd[29657]: PWD
Dec 5 15:13:54 COVIS2 ftpd[28809]: cannot stat private access file /etc/ftpd/ftpgroups: No such file or directory



From oracle error I suppose there was some hardware error but it is strange that only files were deleted not directories.
I would suggest to open a case with HP that would be good because its not a small issue.

BR,
Kapil
I am in this small bowl, I wane see the real world......