Simpler Navigation for Servers and Operating Systems - Please Update Your Bookmarks
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
If you have bookmarked forums or discussion boards in Servers and Operating Systems, we suggest you check and update them as needed.
Operating System - Tru64 Unix
cancel
Showing results for 
Search instead for 
Did you mean: 

Subject: Error loading domain in advfs, inconsistency detected.

jingwp
Occasional Advisor

Subject: Error loading domain in advfs, inconsistency detected.

Yesterday morning, I got the error of my DS25 Server(uname -a, 'OSF1 ds25 v5.1 2650 alpha' displayed):

ADVFS: Domain data_domain not activated - inconsistency detected.

1,I used showfdmn data_domain and the results show as follows:

showfdmn data_domain

#ADVFS : Domain user1 not activated - inconsistency detected
showfdmn: unable to get info for domain 'data_domain'
showfdmn: error = E_DOMAIN_NOT_ACTIVATED (-1067)


2, verify -f data_domain

The message is as follow:
ADVFS : Domain data_domain are not activated-inconsistency detected.
verify: vfast status not available on domain data_domain, E_DOMAIN_NOT_ACTIVATED (-1067)

3,advscan -a, display:
scanning devices: /dev/rdisk/dsk0
/dev/rdisk/dsk2 /dev/rdisk/dsk3 /dev/rdisk/dsk4

Can't open device /dev/rdisk/dsk4a

root_domain : /dev/rdisk/dsk0a (briefly)
home_domain : /dev/rdisk/dsk0f (briefly)
usr_domain : /dev/rdisk/dsk0g (briefly)
gri_domain : /dev/rdisk/dsk0h (briefly)
data_domain : /dev/rdisk/dsk2c (briefly)
data1_domain : /dev/rdisk/dsk3c (briefly)

4,fixfdmn data_domain, message is :

fixfdmn: Cannot allocate 1431125760 bytes.
Check for other processes (besides
fixfdmn) which are using large
amounts of memory, and kill them
if possible or increase swap space.

If neither of those options are available, you may need to increase max_per_proc_data_size, vm_maxvas and max_per_proc_address_space.

Do you want fixfdmn to try this allocation again(y/n/?

5, fixfdmn data1_domain, also display above message. data1_domain is fine. and this means fixfdmn does not work properly.

I am good at SUN Solaris unix, do not know well about DEC Unix.

Thank you for all your help.

I am waiting for replying.
16 REPLIES
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Did you see
Can't open device /dev/rdisk/dsk4a
That can be hardware problem with disk.
Please post output of:
# ls -lR /etc/fdmns
# disklabel -r dsk4
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Thank you for your prompt replying.

1, "Can't open device /dev/rdisk/dsk4a" displayed as command 'advscan -a' runned.

2, # ls -lR /etc/fdmns
total 80
-r-------- 1 root system 0 Jan 1 15:53 .advfslock_data1_domain
-r-------- 1 root system 0 Jan 26 2006 .advfslock_data_domain
-r-------- 1 root system 0 Jan 25 2006 .advfslock_fdmns
-r-------- 1 root system 0 Feb 5 2006 .advfslock_gri_domain
-r-------- 1 root system 0 Feb 5 2006 .advfslock_home_domain
-r-------- 1 root system 0 Jul 1 2006 .advfslock_new_gri_domain
-r-------- 1 root system 0 Jul 1 2006 .advfslock_new_home_domain
-r-------- 1 root system 0 Jul 1 2006 .advfslock_new_root_domain
-r-------- 1 root system 0 Jul 1 2006 .advfslock_new_usr_domain
-r-------- 1 root system 0 Jan 25 2006 .advfslock_root_domain
-r-------- 1 root system 0 Jan 25 2006 .advfslock_usr_domain
drwxr-xr-x 2 root system 8192 Jan 1 15:53 data1_domain
drwxr-xr-x 2 root system 8192 Jun 26 2006 data_domain
drwxr-xr-x 2 root system 8192 Feb 5 2006 gri_domain
drwxr-xr-x 2 root system 8192 Feb 5 2006 home_domain
drwxr-xr-x 2 root system 8192 Jul 1 2006 new_gri_domain
drwxr-xr-x 2 root system 8192 Jul 1 2006 new_home_domain
drwxr-xr-x 2 root system 8192 Jul 1 2006 new_root_domain
drwxr-xr-x 2 root system 8192 Jul 1 2006 new_usr_domain
drwxr-xr-x 2 root system 8192 Jan 25 2006 root_domain
drwxr-xr-x 2 root system 8192 Jan 25 2006 usr_domain

/etc/fdmns/data1_domain:
total 0
lrwxr-xr-x 1 root system 15 Jan 1 15:53 dsk3c -> /dev/disk/dsk3c

/etc/fdmns/data_domain:
total 0
lrwxr-xr-x 1 root system 15 Jan 26 2006 dsk2c -> /dev/disk/dsk2c

/etc/fdmns/gri_domain:
total 0
lrwxr-xr-x 1 root system 15 Feb 5 2006 dsk0h -> /dev/disk/dsk0h

/etc/fdmns/home_domain:
total 0
lrwxr-xr-x 1 root system 15 Feb 5 2006 dsk0f -> /dev/disk/dsk0f

/etc/fdmns/new_gri_domain:
total 0
lrwxr-xr-x 1 root system 15 Jul 1 2006 dsk4h -> /dev/disk/dsk4h

/etc/fdmns/new_home_domain:
total 0
lrwxr-xr-x 1 root system 15 Jul 1 2006 dsk4f -> /dev/disk/dsk4f

/etc/fdmns/new_root_domain:
total 0
lrwxr-xr-x 1 root system 15 Jul 1 2006 dsk4a -> /dev/disk/dsk4a

/etc/fdmns/new_usr_domain:
total 0
lrwxr-xr-x 1 root system 15 Jul 1 2006 dsk4g -> /dev/disk/dsk4g

/etc/fdmns/root_domain:
total 0
lrwxr-xr-x 1 root system 15 Jan 25 2006 dsk0a -> /dev/disk/dsk0a

/etc/fdmns/usr_domain:
total 0
lrwxr-xr-x 1 root system 15 Jan 25 2006 dsk0g -> /dev/disk/dsk0g

3, "disklabel -r dsk4" displaying:
DSK4:NO SUCH DEVICE OR ADDRESS

Thank you. There are a lot of files and data in that disk. I hope the disk is ok.
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Today is Sunday. Hope you have a nice weekend.

I am good at SUN Solaris. If inconsistence occurs, problem is main because of superblock of disk (file system), that means the superblock is damaged, disk can be checked with backup superblock, then every thing will be ok.

Is it same with Tru64 Unix (DEC workstation)? why fixfdmn does not work in my system? Thank you. I hope there are messages from you next week.

Hein van den Heuvel
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

1) Tru64 AdVFS does not use superblocks (UFS does).

2) fixdmn can not fix a bad disk, only inconsistent data on a good disk.

3) 'salvage' might recover more/better.

4) The naming suggest that dsk4a just holds a clone of a root partition, not likely to hold critical, unrecoverable data.

5) The big question is really whether dsk4, all partitions, is useable?
I suggest you use very basic testing for that simply using dd to the RAW device:

dd bs=8192 count=1000 of=/dev/nul if=/dev/rdisk/dsk4a
repeat for 4b, 4f, 4g and 4h

6) check all error logs (/var/adm/messages, uerf,...)

Hope this helps some,
Hein van den Heuvel



Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Looks like you have two problems.
First, you have faulty disk (dsk4).
Second, looks like you have problem with data_domain which is not on dsk4, but is on dsk2.
First try to do:
# fixfdmn -n data_domain
to see what is the problem. Option -n just check, but does nothing.
You also have "salvage" command to try to get data from that domain.
Please also post output of:
# hwmgr -show scsi
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Dear Hein van den Heuvel and Vladimir Fabecic,thank you very much for your kindly help.

Fixfdmn does not work in my system (on this data_domain or other working disk, same error message) , I do not know why.

1, # ./fixfdmn -n data_domain
fixfdmn: Can't allocate 1431125760 bytes.
Check for other processes (besides fixfdmn) which are
using large amounts of memory, and kill them if
possible, or increase swap space.
If neither of those options are available, you may need
to increase max_per_proc_data_size, vm_maxvas,
and max_per_proc_address_space.

Do you want fixfdmn to try this allocation again? ([y]/n)
fixfdmn: Do you want to abort? (y/n) n

2,# hwmgr -show scsi

SCSI DEVICE DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME TYPE SUBTYPE OWNER PATH FILE VALID PATH
-------------------------------------------------------------------------
56: 0 ds25 cdrom none 0 1 (null)
57: 1 ds25 disk none 2 1 dsk0 [2/0/0]
58: 2 ds25 disk none 0 1 (null)
65: 3 ds25 cdrom none 0 1 cdrom1 [0/0/0]
85: 4 ds25 disk none 0 3 dsk2 [4/3/0]
86: 5 ds25 disk none 2 1 dsk3 [3/3/0]
92: 6 ds25 disk none 0 1 (null)

Please find above message of adviced command and give me more ideas. Thank you.
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

OK, to see swap, post output of:
# swapon -s
# grep swapdevice /etc/sysconfigtab
and to see proc settings, post output of:
# sysconfig -q proc
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Thank you. I will follow you. Please find following message.

# swapon -s
Swap partition /dev/disk/dsk0b (default swap):
Allocated space: 262144 pages (2.00GB)
In-use space: 13 pages ( 0%)
Free space: 262131 pages ( 99%)


Total swap allocation:
Allocated space: 262144 pages (2.00GB)
Reserved space: 83434 pages ( 31%)
In-use space: 13 pages ( 0%)
Available space: 178710 pages ( 68%)

# grep swapdevice /etc/sysconfigtab
swapdevice = /dev/disk/dsk0b

# sysconfig -q proc
proc:
max_proc_per_user = 256
max_threads_per_user = 1024
per_proc_stack_size = 8388608
max_per_proc_stack_size = 33554432
per_proc_data_size = 134217728
max_per_proc_data_size = 1073741824
max_per_proc_address_space = 4294967296
per_proc_address_space = 4294967296
executable_stack = 0
autonice = 0
autonice_time = 600
autonice_penalty = 4
open_max_soft = 4096
open_max_hard = 4096
ncallout_alloc_size = 8192
round_robin_switch_rate = 0
sched_min_idle = 0
give_boost = 1
maxusers = 2048
num_wait_queues = 2048
num_timeout_hash_queues = 2048
enhanced_core_name = 0
enhanced_core_max_versions = 16
exec_disable_arg_limit = 0
dump_cores = 1
dump_setugid_cores = 0
executable_data = 0
#
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

fixfdmn: Cannot allocate 1431125760 bytes.
This can really be max_per_proc_data_size kernel parameter.
Try to set:
per_proc_data_size = 4294967296
max_per_proc_data_size = 4294967296
in /etc/sysconfigtab and reboot.
You can also use /usr/bin/X11/dxkerneltuner (if installed) to change this.
After that try fixfdmn again.
Do you know how to change this kernel parameters?
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Sorry, I do not know how to change this kernel parameters.

Can you give me command in detail please? Thank you.
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Do "vi /etc/sysconfigtab"
find section "proc:"
add or change lines to be:
per_proc_data_size = 4294967296
max_per_proc_data_size = 4294967296
I can give you example of one of my sysconfigtabs:


proc:
max_proc_per_user=4096
max_threads_per_user=16348
per_proc_stack_size=33554432
per_proc_data_size=4294967296
max_per_proc_data_size=4294967296

ipc:
shm_max=4278190080
shm_mni=256
shm_seg=128

You should have proc settings like above.
You can change to original value after fixing problem.
In vino veritas, in VMS cluster
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Sorry, I pasted more than needed. You should change just proc parameters, not ipc.
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

I find no section 'proc' , no following parameter inside file '/etc/sysconfigtab'.

May I add the following to this file (/etc/sysconfigtab)?

proc:
max_proc_per_user=4096
max_threads_per_user=16348
per_proc_stack_size=33554432
per_proc_data_size=4294967296
max_per_proc_data_size=4294967296
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

Yes, do it on the end of /etc/sysconfigtab.
In attached file there is an example of how it should look like.
First copy original file i.e.
# cp /etc/sysconfigtab /etc/sysconfigtab.orig
then edit sysconfigtab,
and then reboot.
In vino veritas, in VMS cluster
jingwp
Occasional Advisor

Re: Subject: Error loading domain in advfs, inconsistency detected.

I will add and reboot DS25.

According to your experience, what is wrong with my DS25 system, is it possible to fix data_domain?

There are a lot of files on data_domain.
Vladimir Fabecic
Honored Contributor

Re: Subject: Error loading domain in advfs, inconsistency detected.

It is hard to say what are the chances.
First we will see output of "# fixfdmn -n data_domain".
There is also chance to get data using "salvage". Hope you have enough disk space somewhere for "salvage". See "man salvage" for more info.
I have returned data many times from situation like yours, but also did restore backup few times.
Hope you also have backup.
In vino veritas, in VMS cluster