Oracle 11gR1 RAC database creation(DBCA) on SG 11.18 HP-UX11.31 IA cluster hangs @ 52%.

[raja_stm]

Hi All,

My Oracle 11gR1 RAC database creation(DBCA)on SG 11.18 HP-UX11.31IA Cluster hangs @52%.Please find alert log file below. Please note that i didnt face any issues with Clusteraware and RAC binaries installion and it all went fine.Only Problem that am facing is while creating RAC database DBCA hangs @ 52% and there is no error message comes out.
I have googled enough and i didnt get any solution.Please help me in solving this issue.

Configuration:-
=> T1905CA->A.11.18.00->Serviceguard
=> T1907BA->A.11.18.00->Serviceguard Extension for RAC
=> 2 Node SG 11.18 HP-UX11.31 IA Cluster.
=> Clusterware files(OCR and VOTE disk) are placed in shared raw LVM logical volumes.(/dev/vg_rac/rocr_400m & /dev/vg_rac/rvote_400m)
=> Clusterware and RAC binaries are installed on local file system.(/u01/app/11.1.0/crs & /u01/app/oracle)
=> Trying to create a database on Shared raw LVM logical volumes.(Eg., rorcl_sysaux_raw_1500m).

alert_orcl1.log:-

##############################################
Tue Dec 02 17:20:28 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan1 10.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 lan0 15.154.52.0 configured from OCR for use as a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.6.0.
Using parameter settings in client-side pfile /u01/app/oracle/admin/orcl/pfile/init.ora on machine skyline
System parameters with non-default values:
processes = 150
memory_target = 816M
control_files = "/dev/vg_rac/rorcl_control1_raw_120m"
control_files = "/dev/vg_rac/rorcl_control2_raw_120m"
db_block_size = 8192
compatible = "11.1.0.0.0"
cluster_database_instances= 1
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=orclXDB)"
remote_listener = "LISTENERS_ORCL"
audit_file_dest = "/u01/app/oracle/admin/orcl/adump"
audit_trail = "DB"
db_name = "orcl"
open_cursors = 300
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
10.0.0.51
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Dec 02 17:20:39 2008
PMON started with pid=2, OS id=19633
Tue Dec 02 17:20:39 2008
VKTM started with pid=3, OS id=19635 at elevated priority
VKTM running at (20)ms precision
Tue Dec 02 17:20:39 2008
DIAG started with pid=4, OS id=19637
Tue Dec 02 17:20:42 2008
DBRM started with pid=5, OS id=19639
Tue Dec 02 17:20:43 2008
PING started with pid=6, OS id=19646
Tue Dec 02 17:20:44 2008
PSP0 started with pid=7, OS id=19653
Tue Dec 02 17:20:44 2008
ACMS started with pid=8, OS id=19655
Tue Dec 02 17:20:44 2008
DSKM started with pid=9, OS id=19657
Tue Dec 02 17:20:44 2008
DIA0 started with pid=10, OS id=19659
Tue Dec 02 17:20:45 2008
LMON started with pid=11, OS id=19663
Tue Dec 02 17:20:46 2008
LMD0 started with pid=12, OS id=19669
Tue Dec 02 17:20:47 2008
RMS0 started with pid=13, OS id=19676
Tue Dec 02 17:20:47 2008
MMAN started with pid=14, OS id=19683
Tue Dec 02 17:20:49 2008
DBW0 started with pid=9, OS id=19685
Tue Dec 02 17:20:49 2008
LGWR started with pid=15, OS id=19693
Tue Dec 02 17:20:49 2008
CKPT started with pid=16, OS id=19699
Tue Dec 02 17:20:50 2008
SMON started with pid=17, OS id=19708
Tue Dec 02 17:20:50 2008
RECO started with pid=18, OS id=19714
Tue Dec 02 17:20:51 2008
MMON started with pid=19, OS id=19719
Tue Dec 02 17:20:52 2008
MMNL started with pid=20, OS id=19726
Tue Dec 02 17:20:52 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
Tue Dec 02 17:21:00 2008
lmon registered with NM - instance id 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Reconfiguration complete
Tue Dec 02 17:21:05 2008
ORACLE_BASE from environment = /u01/app/oracle
Tue Dec 02 17:21:14 2008
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ping_19646.trc:
Tue Dec 02 17:21:23 2008
Full restore complete of datafile 4 to datafile copy /dev/vg_rac/rorcl_users_raw_120m. Elapsed time: 0:00:00
checkpoint is 892091
Full restore complete of datafile 3 to datafile copy /dev/vg_rac/rorcl_undotbs1_raw_750m. Elapsed time: 0:00:01
checkpoint is 892091
Tue Dec 02 17:22:26 2008
Full restore complete of datafile 2 to datafile copy /dev/vg_rac/rorcl_sysaux_raw_1500m. Elapsed time: 0:01:04
checkpoint is 892091
last deallocation scn is 891749
Full restore complete of datafile 1 to datafile copy /dev/vg_rac/rorcl_system_raw_900m. Elapsed time: 0:01:14
checkpoint is 892091
last deallocation scn is 887151
Undo Optimization current scn is 824432
Tue Dec 02 17:22:36 2008
Create controlfile reuse set database "orcl"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
Datafile
'/dev/vg_rac/rorcl_system_raw_900m',
'/dev/vg_rac/rorcl_sysaux_raw_1500m',
'/dev/vg_rac/rorcl_undotbs1_raw_750m',
'/dev/vg_rac/rorcl_users_raw_120m'
LOGFILE GROUP 1 ('/dev/vg_rac/rorcl_redo1_1_raw_120m') SIZE 51200K,
GROUP 2 ('/dev/vg_rac/rorcl_redo1_2_raw_120m') SIZE 51200K RESETLOGS
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
Setting recovery target incarnation to 1
Tue Dec 02 17:22:38 2008
Successful mount of redo thread 1, with mount id 1200234717
Completed: Create controlfile reuse set database "orcl"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
Datafile
'/dev/vg_rac/rorcl_system_raw_900m',
'/dev/vg_rac/rorcl_sysaux_raw_1500m',
'/dev/vg_rac/rorcl_undotbs1_raw_750m',
'/dev/vg_rac/rorcl_users_raw_120m'
LOGFILE GROUP 1 ('/dev/vg_rac/rorcl_redo1_1_raw_120m') SIZE 51200K,
GROUP 2 ('/dev/vg_rac/rorcl_redo1_2_raw_120m') SIZE 51200K RESETLOGS
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
Shutting down instance (immediate)
License high water mark = 1
Waiting for dispatcher 'D000' to shutdown
All dispatchers and shared servers shutdown
ALTER DATABASE CLOSE NORMAL
ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Tue Dec 02 17:22:43 2008
Stopping background process VKTM:
Tue Dec 02 17:22:47 2008
freeing rdom 0
Tue Dec 02 17:22:49 2008
Instance shutdown complete
Tue Dec 02 17:22:50 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan1 10.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 lan0 15.154.52.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.6.0.
Using parameter settings in client-side pfile /u01/app/oracle/cfgtoollogs/dbca/orcl/initorclTemp.ora on machine skyline
System parameters with non-default values:
processes = 150
memory_target = 816M
control_files = "/dev/vg_rac/rorcl_control1_raw_120m"
control_files = "/dev/vg_rac/rorcl_control2_raw_120m"
db_block_size = 8192
compatible = "11.1.0.0.0"
cluster_database_instances= 1
_no_recovery_through_resetlogs= TRUE
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=orclXDB)"
remote_listener = "LISTENERS_ORCL"
audit_file_dest = "/u01/app/oracle/admin/orcl/adump"
audit_trail = "DB"
db_name = "orcl"
open_cursors = 300
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
10.0.0.51
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Dec 02 17:22:59 2008
PMON started with pid=2, OS id=20657
Tue Dec 02 17:22:59 2008
VKTM started with pid=3, OS id=20659 at elevated priority
VKTM running at (20)ms precision
Tue Dec 02 17:22:59 2008
DIAG started with pid=4, OS id=20661
Tue Dec 02 17:22:59 2008
DBRM started with pid=5, OS id=20668
Tue Dec 02 17:22:59 2008
PING started with pid=6, OS id=20670
Tue Dec 02 17:22:59 2008
PSP0 started with pid=7, OS id=20672
Tue Dec 02 17:22:59 2008
ACMS started with pid=8, OS id=20674
Tue Dec 02 17:23:00 2008
DSKM started with pid=9, OS id=20676
Tue Dec 02 17:23:00 2008
DIA0 started with pid=10, OS id=20678
Tue Dec 02 17:23:00 2008
LMON started with pid=9, OS id=20680
Tue Dec 02 17:23:00 2008
LMD0 started with pid=11, OS id=20683
Tue Dec 02 17:23:00 2008
RMS0 started with pid=12, OS id=20691
Tue Dec 02 17:23:00 2008
MMAN started with pid=13, OS id=20693
Tue Dec 02 17:23:01 2008
DBW0 started with pid=14, OS id=20695
Tue Dec 02 17:23:01 2008
LGWR started with pid=15, OS id=20697
Tue Dec 02 17:23:01 2008
CKPT started with pid=16, OS id=20699
Tue Dec 02 17:23:01 2008
SMON started with pid=17, OS id=20701
Tue Dec 02 17:23:01 2008
RECO started with pid=18, OS id=20703
Tue Dec 02 17:23:01 2008
MMON started with pid=19, OS id=20706
Tue Dec 02 17:23:01 2008
MMNL started with pid=20, OS id=20708
Tue Dec 02 17:23:01 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
lmon registered with NM - instance id 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Reconfiguration complete
ORACLE_BASE from environment = /u01/app/oracle
Tue Dec 02 17:23:15 2008
Create controlfile reuse set database "orcl"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
Datafile
'/dev/vg_rac/rorcl_system_raw_900m',
'/dev/vg_rac/rorcl_sysaux_raw_1500m',
'/dev/vg_rac/rorcl_undotbs1_raw_750m',
'/dev/vg_rac/rorcl_users_raw_120m'
LOGFILE GROUP 1 ('/dev/vg_rac/rorcl_redo1_1_raw_120m') SIZE 51200K,
GROUP 2 ('/dev/vg_rac/rorcl_redo1_2_raw_120m') SIZE 51200K RESETLOGS
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
Setting recovery target incarnation to 1
Tue Dec 02 17:23:15 2008
Successful mount of redo thread 1, with mount id 1200244739
Completed: Create controlfile reuse set database "orcl"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
Datafile
'/dev/vg_rac/rorcl_system_raw_900m',
'/dev/vg_rac/rorcl_sysaux_raw_1500m',
'/dev/vg_rac/rorcl_undotbs1_raw_750m',
'/dev/vg_rac/rorcl_users_raw_120m'
LOGFILE GROUP 1 ('/dev/vg_rac/rorcl_redo1_1_raw_120m') SIZE 51200K,
GROUP 2 ('/dev/vg_rac/rorcl_redo1_2_raw_120m') SIZE 51200K RESETLOGS
Stopping background process MMNL
Stopping background process MMON
Starting background process MMON
Tue Dec 02 17:23:18 2008
MMON started with pid=19, OS id=20850
Starting background process MMNL
Tue Dec 02 17:23:18 2008
MMNL started with pid=20, OS id=20852
ALTER SYSTEM enable restricted session;
alter database "orcl" open resetlogs
RESETLOGS after incomplete recovery UNTIL CHANGE 892091
Tue Dec 02 17:23:20 2008
Setting recovery target incarnation to 2
Assigning activation ID 1200244739 (0x478a4803)
Tue Dec 02 17:23:20 2008
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ping_20670.trc:
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: /dev/vg_rac/rorcl_redo1_1_raw_120m
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Dec 02 17:23:20 2008
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
This condition can occur when a backup controlfile has
been restored. It may be necessary to add files to these
tablespaces. That can be done using the SQL statement:

ALTER TABLESPACE ADD TEMPFILE

Alternatively, if these temporary tablespaces are no longer
needed, then they can be dropped.
Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is US7ASCII
Opening with internal Resource Manager plan
Tue Dec 02 17:23:25 2008
Starting background process FBDA
Tue Dec 02 17:23:26 2008
FBDA started with pid=24, OS id=20891
Starting background process SMCO
Tue Dec 02 17:23:29 2008
SMCO started with pid=25, OS id=20898
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Tue Dec 02 17:23:31 2008
QMNC started with pid=26, OS id=20915
Tue Dec 02 17:23:36 2008
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Global Name changed to ORCL
Tue Dec 02 17:24:05 2008
Completed: alter database "orcl" open resetlogs
alter database rename global_name to "orcl"
Completed: alter database rename global_name to "orcl"
ALTER TABLESPACE TEMP ADD TEMPFILE '/dev/vg_rac/rorcl_temp_raw_400m' SIZE 20480K REUSE AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Completed: ALTER TABLESPACE TEMP ADD TEMPFILE '/dev/vg_rac/rorcl_temp_raw_400m' SIZE 20480K REUSE AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
ALTER DATABASE DEFAULT TABLESPACE "USERS"
Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
alter database character set INTERNAL_CONVERT WE8MSWIN1252
Tue Dec 02 17:24:47 2008
Thread 1 advanced to log sequence 2
Current log# 2 seq# 2 mem# 0: /dev/vg_rac/rorcl_redo1_2_raw_120m
Tue Dec 02 17:24:55 2008
Updating character set in controlfile to WE8MSWIN1252
Synchronizing connection with database character set information
Refreshing type attributes with new character set information
Tue Dec 02 17:25:47 2008
Completed: alter database character set INTERNAL_CONVERT WE8MSWIN1252
Tue Dec 02 17:27:41 2008
Starting background process CJQ0
Tue Dec 02 17:27:45 2008
CJQ0 started with pid=31, OS id=22420
Tue Dec 02 17:28:23 2008
CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE '/dev/vg_rac/rorcl_undotbs2_raw_750m' SIZE 25600K REUSE AUTOEXTEND ON MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL
Tue Dec 02 17:28:47 2008
Completed: CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE '/dev/vg_rac/rorcl_undotbs2_raw_750m' SIZE 25600K REUSE AUTOEXTEND ON MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL
Tue Dec 02 17:28:54 2008
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_m000_22638.trc:
ORA-04035: unable to allocate 4096 bytes of shared memory in shared object cache "JOXSHM" of size "52428800"
Tue Dec 02 17:52:46 2008
Shutting down instance (abort)
License high water mark = 8
USER (ospid: 3358): terminating the instance
Instance terminated by USER, pid = 3358
Tue Dec 02 17:52:47 2008
Instance shutdown complete
##############################################

[Duncan Edmonstone]

Well if this is related to the ORA-04035 error, see Oracle metalink note 556485.1

HTH

Duncan