Informix ontape problems - "could not fork server connection"

[Robin King_1]

I'm trying to do an informix restore to a new server, or existing databases.
When I issue the "ontape -r" command, I'm getting the error "could not fork server connection" and then it stops.

Is this a system resource issue or something else?

Thanks

[Bill Hassell]

Look at syslog (/var/adm/syslog/syslog.log) at the end to see if there are kernel messages with the same timestamp. Also look at the console (not a telnet login, the 'real' console). Some possibilities:

1. Maximum number of processes for a single user ID has been exceeded (hint: "ps -f -u informix | wc -l" or whatever user ontape is running under). Kernel parameter maxuproc limits this value to prevent runaways on test servers, bump up if needed)

2. Maximum memory needed by a process. Kernel parameter maxdsiz is usually way too low (about 64megs) and some processes may need hundreds of megs.

3. syslog and console error messages: "proc: table is full" means that kernel parameter nproc needs to be doubled in size.

4. syslog and console error messages: "file: table is full" means that kernel parameter nfile needs to be doubled in size.

[Robin King_1]

Thanks Bill

There's nothing on the console or in the syslog that would suggest any problems.
There only seems to be a max of 12 Informix processes running, and maxuprc is at 75.
maxdsiz is at 268MB.

[Steve Lewis]

You need to tune your kernel on the server that you are restoring onto, to be the same as the one which the backup was taken.

As Bill said, this is caused either by running out of NPROC, MAXUPRC, MAXFILES or NFILE on the system.
sar -v 1 1 will give you a quick indication on files in the system.

Informix requires higher than default values for shared memory, semaphore, processes and file parameters in the kernel, as is documented in the $INFORMIXDIR/release/en_us/0333/ ...machine_notes....txt file

[JJ_4]

If you could provide the following it would help :

Informix Version
HP-UX Version
Details from the file mentioned by MSGPATH in the $ONCONFIG
Does the restore start at all (i.e. produce a list of dbspaces, chunks and backup information)
Where was the backup from (i.e. the machine you are restoring to or another)?
Is the Informix version the same for the backup as for the restore?
Can you run oncheck -pr before the restore?

[Robin King_1]

I've tried amending the kernel parameters to match the original system, but it doesn't seem to have made much difference.

Informix Version = Informix Dynamic Server Version 7.31.FD7
HP-UX Version = B.11.11
MSGPATH = "
Tue Sep 14 11:07:34 2004

11:07:34 Event alarms enabled. ALARMPROG = '/apps/informix/dba/bin/alarmprog'
11:07:40 DR: DRAUTO is 0 (Off)
11:07:40 Informix Dynamic Server Version 7.31.FD7 Software Serial Number AAD#J342257
11:07:40 WARNING: Next backup of DBspace rootdbs must be level-0 backup.
11:07:40 WARNING: Next backup of DBspace mvp_dbspace must be level-0 backup.
11:07:40 Informix Dynamic Server Initialized -- Shared Memory Initialized.
11:07:40 Dataskip is now OFF for all dbspaces
11:07:40 Restartable Restore has been ENABLED
11:07:40 Recovery Mode"

The Restore does start, and I get the list of dbspaces and chunks, I'm asked if I want to continue restore? and then if I want to backup the logs?
The restore then bails.

Interestingly, if I then try top restart the restore straight away, it bails out straight away. I've just done an ipcs -m, and then an ipcrm -m on all informix instances, and the restore appears to be doing something now. I'll have to wait and see how it goes.

The backup was taken from another server than is running a different version of HP-UX and Informix.

I'' run the oncheck -pr, if this bails again.

[Robin King_1]

It failed again, but got further. This is what I did (sorry for the length):

# ipcs -m
IPC status from /dev/kmem as of Tue Sep 14 11:54:10 2004
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c0a4f --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412034ee --rw-rw-rw- root root
m 3075 0x0c6629c9 --rw-r----- root root
m 4 0x06347849 --rw-rw-rw- root root
m 1541 0xffffffff --rw-r--rw- root root
m 8710 0x5e10000d --rw------- root root
m 13831 0x52574801 --rw-rw---- root informix
m 8 0x52574802 --rw-rw---- root informix
m 9 0x52574803 --rw-rw-rw- root informix
# ipcrm -m 13831
# ipcrm -m 8
# ipcrm -m 9
# ipcs -m
IPC status from /dev/kmem as of Tue Sep 14 11:54:32 2004
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c0a4f --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412034ee --rw-rw-rw- root root
m 3075 0x0c6629c9 --rw-r----- root root
m 4 0x06347849 --rw-rw-rw- root root
m 1541 0xffffffff --rw-r--rw- root root
m 8710 0x5e10000d --rw------- root root
m 13831 0x00000000 D-rw-rw---- root informix
m 8 0x00000000 D-rw-rw---- root informix
m 9 0x00000000 D-rw-rw-rw- root informix

# su - informix
For copyright information, view /etc/copyright.hp
INFerbhat3:/apps/informix:: ontape -r

Please mount tape 1 on /dev/rmt/0m and press Return to continue ...

Archive Tape Information

Tape type: Archive Backup Tape
Online version: Informix Dynamic Server Version 7.31.UD4
Archive date: Fri Sep 10 07:14:42 2004
User id: informix
Terminal id: /dev/console
Archive level: 0
Tape device: /dev/rmt/1m
Tape blocksize (in k): 64
Tape size (in k): 23000000
Tape number in series: 1

Spaces to restore:1 [rootdbs ]
2 [mvp_dbspace ]

Archive Information

Informix Dynamic Server Copyright(C) 1986-1998 Informix Software, Inc.
Initialization Time 06/15/2002 09:30:08
System Page Size 2048
Version 6
Archive CheckPoint Time 09/10/2004 07:14:43

Dbspaces
number flags fchunk nchunks flags owner name
1 1 1 1 N rmix dbs info
2 2001 2 1 N T rmix dbs info
3 1 3 14 N rmix dbspace info


Chunks
chk/dbs offset size free bpages flags pathname
1 1 0 614400 147389 PO- /dev/rootdbs
2 2 0 393216 393163 PO- /dev/tempdbs
3 3 0 1048575 3 PO- /dev/mvpdbs01
4 3 0 1048575 2 PO- /dev/mvpdbs02
5 3 0 1048575 0 PO- /dev/mvpdbs03
6 3 0 1048575 5549 PO- /dev/mvpdbs04
7 3 0 1048575 101870 PO- /dev/mvpdbs05
8 3 0 1048575 1858 PO- /dev/mvpdbs06
9 3 0 1048575 2 PO- /dev/mvpdbs07
10 3 0 1048575 26786 PO- /dev/mvpdbs08
11 3 0 1048575 15497 PO- /dev/mvpdbs09
12 3 0 1048575 28947 PO- /dev/mvpdbs10
13 3 0 1048575 175794 PO- /dev/mvpdbs11
14 3 0 1048575 166865 PO- /dev/mvpdbs12
15 3 0 1048575 373542 PO- /dev/mvpdbs13
16 3 0 1048575 870402 PO- /dev/mvpdbs14

Continue restore? (y/n)y
Do you want to back up the logs? (y/n)n

****(10 Minutes elapsed)****

Physical restore failed - could not fork server connection

Program over.
INFerbhat3:/apps/informix::

The online.log shows this:

Tue Sep 14 11:55:34 2004

11:55:34 Event alarms enabled. ALARMPROG = '/apps/informix/dba/bin/alarmprog'
11:55:39 DR: DRAUTO is 0 (Off)
11:55:40 Informix Dynamic Server Version 7.31.FD7 Software Serial Number AA
D#J342257
11:55:40 WARNING: Next backup of DBspace rootdbs must be level-0 backup.
11:55:40 WARNING: Next backup of DBspace mvp_dbspace must be level-0 backup.
11:55:40 Informix Dynamic Server Initialized -- Shared Memory Initialized.
11:55:40 Dataskip is now OFF for all dbspaces
11:55:40 Restartable Restore has been ENABLED
11:55:40 Recovery Mode
11:55:40 listener-thread: err = -25572: oserr = 226: errstr = : Network driver
cannot bind a name to the port.
System error = 226.
11:55:40 Attempting to bring listener thread down.

11:55:40 Server stopped.

11:55:40 Informix Dynamic Server Stopped.
11:55:40 mt_shm_remove: WARNING: may not have removed all/correct segments

[Tom Danzig]

Just a shot in the dark ...

Check available swap with swapinfo -ta and check if there is enough. I have seen instances where there was insufficient swap available to fork a new process.

Your message was "could not fork server connection" so I'm not sure if this is applicable.

[daniel m_1 · Accepted solution]

What version of Informix was the backup taken on?
You can use different O/S versions, but Informix support has always told me the Informix version has to be the EXACT version for the restore as the backup.

[Brian Butscher]

Robin,

First thing is to check the online.log. This should give you more information pertaining to the the error. If you find an error in the online.log then use 'finderr xxxx' to get more information from Informix.

Also check the following:
1) 'sqlhosts' file in the etc directory to make sure you have the proper host name, the 'DBSERVERNAME and DBSERVERALIASES should match what you have in the 'etc/onconfig.xxx file.
2) Check for any '.str' files in the /INFORMIXTMP directory, remove them.
3)Check $INFORMIXDIR/release file for shared memory requirements.
4) After you attempt to restore and you get an error, check ipcs. If there are any queues, memory, or shared memory owned by informix, clean them up with the ipcrm -q xxxxx, ipcrm -m xxxxx, or ipcrm -s xxxxx. You will not have enough memory to restart the restore unless you do this.
5) Check the permissions and ownership on the files in the /dev/vginformix directory (assuming you use this directory to create the logical volumes for Informix). Permissions need to be 660, ownership informix:informix for all informix logical volume files created. Make sure user 'informix' has access to that directory and files.

Are you using a cooked or raw database?

Let me know if you still have problems.

Regards,
Brian

[Brian Butscher]

Robin,

In looking at your online.log I see the error 226- I did a finderr and got the following results:

finderr 226
-226 Cannot create index for system catalog table-name.

The database server is trying to create one of the tables for the
system catalog, probably as part of a CREATE DATABASE statement. It
created the table, but a problem with the host operating system
prevents it from making an index. Check the accompanying ISAM error
code for more information, and look for operating-system error
messages. Insufficient disk space probably caused this error.

It looks like you don't have all of the chunks defined for your database as indicated by the 'Insufficient disk space probably caused this error.'

Make sure you have all of the following chunks defined:
/dev/rootdbs
/dev/tempdbs
/dev/mvpdbs01
/dev/mvpdbs02
/dev/mvpdbs03
/dev/mvpdbs04
/dev/mvpdbs05
/dev/mvpdbs06
/dev/mvpdbs07
/dev/mvpdbs08
/dev/mvpdbs09
/dev/mvpdbs10
/dev/mvpdbs11
/dev/mvpdbs12
/dev/mvpdbs13
/dev/mvpdbs14

with the correct permissions and ownership.

Can you attach a detailed listing of the /dev directory? I would recommend creating a separate directory for your Informix chunks as they need to have specific permissions and ownership.

Regards,

Brian

[Robin King_1]

Daniel you are correct. We spoke to IBM and they confirmed that the OS version and the DB version have to be exactly the same at source and destination.

We are now looking into exporting the databases, and then importing on the new server.

At least I've learnt a little about Informix in the process. Thanks for your help.

[daniel m_1]

The support line might have steered you wrong a little.

I have done many Informix restores from HP-UX 10.20 to HP-UX 11.11 machines, the thing to remember is that the Informix engine must be the same.
You can still accomplish what you were trying to do - once you get the database on the new machine, you can upgrade to the new Informix version.

[Brian Butscher]

Robin,


The Informix version must be the same as well Informix support pointed out. You might want to remember also that if you are running a 64 bit version of Informix, you must be running the 64-bit OS as well.

We all learn from these experiences, that's what makes this forum so great!

Brian

[JJ_4]

I think that dbexport / dbimport is the way to go, as the backup was done on IDS 7.31.UD4 and you are restoring onto IDS 7.31.FD7!!

The reason that the engine is failing is different though :


11:55:40 listener-thread: err = -25572: oserr = 226: errstr = : Network driver
cannot bind a name to the port.
System error = 226.
11:55:40 Attempting to bring listener thread down.

# define EADDRINUSE 226 /* Address already in use */

So it is an O/S Error.

Check your sqlhosts file for this new instance, and see whether you have a conflict in the service name ...

dbservername onipcshm hostname dummy_servicename
dbserveralias onsoctcp hostname servicename

Service name may be a port number, you just need to check whether that name / port number is already in use.

An easy way to confirm whether this instance is okay, is to do a full initialisation before the restore (which probably won't work anyway because of the different versions).

So, I think you will have to do a full disk initialisation of the 7.31.FD7, add all of your dbspaces / chunks, move your logs out etc. etc. Then do a full dbexport / dbimport.

An alternative could be to install 7.31.UD4 onto your new machine, and do a restore from your old 7.31.UD4, then look at the migration guide, and do an in-place upgrade to 7.31.FD7.

Food for thought hopefully.

[Matt Siconolfi]

Just wondering what the solution to Robin's problem was?

[ca25089]

Hi Matt, it is future Matt.  Your particular issue was related to hung shared memory segments that didn't clean up after a crash due to extent overlapping:

carsddb (aine): ipcs | grep inf
m 393231 0x00000000 D-rw-rw---- root informix
m 360464 0x00000000 D-rw-rw---- root informix

See these guys above?  You won't be able to remove them with ipcrm, you had to get kmeminfo from HP (through our CSS account rep, which was the easiest way).  Once you did that, you were able to find the processes attached to these by running the following:

carsddb (aine): kmeminfo -shmem 393231
tool: kmeminfo 10.37 - libp4 9.633 - libhpux 1.422 - HP CONFIDENTIAL
unix: HP-UX B.11.31 64bit Montvale 9140N on host "carsddb"
core: /dev/kmem live kernel!
link: Sun Jan 08 09:29:33 EST 2012
boot: Thu Feb 14 08:35:55 2013
time: Wed Feb 20 10:34:10 2013
nbpg: 4096 bytes

Shmid 393231:
Descriptor : struct shmid_ds 0xe0000001d0030618
Pseudo vas : vas_t 0xe0000003a893ba00
Pseudo preg: preg_t 0xe000000328bf4180
Shared reg : reg_t 0xe000000396ff1600
Segment range 0x7ff80031.0xc0000001e0000000-0xc000000204e5ffff
Segment allocated out of "Global 64-bit quadrant 6"
Processes using this segment:
proc=0xe004010010473100 (pid 16955 "oninit"): vas=0xe0000003378ba080, shmem preg=0xe000000328bd2480
proc=0xe000000335e3c400 (pid 16946 "oninit"): vas=0xe00000038bc02a00, shmem preg=0xe0000003383e6e00
proc=0xe000000344e0c400 (pid 16941 "oninit"): vas=0xe0000003c252e980, shmem preg=0xe000000347c81a00
proc=0xe004010010479300 (pid 16940 "oninit"): vas=0xe0000003710d0680, shmem preg=0xe0000003ca497c00
proc=0xe00000033bb70000 (pid 16939 "oninit"): vas=0xe0000003710d0980, shmem preg=0xe0000003682d6e00
proc=0xe0000003367cc400 (pid 16938 "oninit"): vas=0xe0000001dfec0980, shmem preg=0xe0000003a4910080
proc=0xe00000035c0eab80 (pid 16937 "oninit"): vas=0xe0000003cc26e380, shmem preg=0xe0000001aa888280
proc=0xe000000352a49300 (pid 16936 "oninit"): vas=0xe000000337874700, shmem preg=0xe00000039deef480
proc=0xe000000362c1ab80 (pid 16935 "oninit"): vas=0xe0000003600b4400, shmem preg=0xe0000003ce865680
proc=0xe0000003d24ec400 (pid 16930 "oninit"): vas=0xe0000003ac9e1400, shmem preg=0xe00000033b727780
proc=0xe000000344e00000 (pid 16929 "oninit"): vas=0xe0000003d1485980, shmem preg=0xe0000003398f5300
proc=0xe000000333140000 (pid 16915 "oninit"): vas=0xe00000033d40da00, shmem preg=0xe00000033768cb80
proc=0xe00000033f79c400 (pid 16914 "oninit"): vas=0xe0000001dfae8380, shmem preg=0xe0000003398ab300
proc=0xe00000033cc06200 (pid 16913 "oninit"): vas=0xe000000336a47100, shmem preg=0xe000000353e0e400
proc=0xe000000337bbdc80 (pid 16912 "oninit"): vas=0xe0000001dfec0080, shmem preg=0xe0000003974c5580
proc=0xe000000333dedc80 (pid 16911 "oninit"): vas=0xe000000352bcfa00, shmem preg=0xe0000003ba283a00
proc=0xe000000335e33100 (pid 16910 "oninit"): vas=0xe0000003311e6a00, shmem preg=0xe00000033eebe600
proc=0xe000000344ea3100 (pid 16909 "oninit"): vas=0xe000000376108a00, shmem preg=0xe0000003e1b69100
proc=0xe000000344fd7a80 (pid 16908 "oninit"): vas=0xe0000001dfddc680, shmem preg=0xe0000003affed180
proc=0xe000000362c11880 (pid 16907 "oninit"): vas=0xe0000001e0212c80, shmem preg=0xe0000003dbfb6100
proc=0xe000000334bc4980 (pid 16906 "oninit"): vas=0xe000000396f54a00, shmem preg=0xe0000003db953600
proc=0xe000000335e36200 (pid 16905 "oninit"): vas=0xe00000036010ed00, shmem preg=0xe0000003a2b20100
proc=0xe0000003367cab80 (pid 16904 "oninit"): vas=0xe0000001e0212380, shmem preg=0xe00000033d9c6900
proc=0xe000000336d10000 (pid 16903 "oninit"): vas=0xe00000033a2c6400, shmem preg=0xe00000036ac86d00

So basically you threw all the pids in a foreach loop, killed them off and then you were able to do your 'ontape -r'