topic Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX? in Operating System - HP-UX

[ftok()-related] Can 'st.st_dev of the same file' be changed between reboots on HP-UX?

Alex_Vinokur — Sun, 08 Jan 2012 05:54:24 GMT

Hi,

st.st_dev of the same file (that wasn't deleted) can be changed between
reboots on Linux.

See http://groups.google.com/group/comp.unix.programmer/browse_frm/thread/018671b4d364409f

As a result, the ftok() function on Linux may generate different keys for
the same file (that wasn't deleted) between reboots.
It seems to be a bug in design of ftok() on Linux.

Does such a problem exist with ftok() on HP-UX?

Alex

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Sun, 08 Jan 2012 20:43:29 GMT

(Please edit your post with the "post Options > Edit Reply" to remove the bogus wlmailhtml: hyperlink that nobody but you? can access. Using the URL in the text does work but we want lazy people to see it too. :-)

>Does such a problem exist with ftok() on HP-UX?

Yes, even worse, it can be changed without reboots.

It currently exists in the linker if you compile with -ipo or +O4.

The problem exists in the automounter. If it unmounts the filesystem, you are hosed and st_dev seems to be incremented. Unmounting was mentioned by Barry in your URL.

One solution is to keep the file open so it can't be unmounted. Or mmap(2) the file.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed between reboots on HP-UX?

Alex_Vinokur — Sun, 08 Jan 2012 05:52:51 GMT

Thanks.

1. My original post was edited (the link was deleted).

2. The following situation:

-------------------------

create file with name "Name1"

// That file will not be deleted

reboot-1

...

key1 = ftok("Name1", 'A');

...

key2 = ftok ("Name1", 'A');

reboot-2

key3 = ftok("Name1", 'A');

-----------------------------

key3 may differ from key1. It is clear.

It turnsout that key2 can differ from key1. It is very upleasant situation.

We can't keep the file "Name1" open and we can't mmap the file.

I didn't understand witt -ipo or +O4.

Thanks

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Sun, 08 Jan 2012 20:43:48 GMT

>1. My original post was edited (the link was deleted).

(You didn't need to delete it, just make it easy to click on. :-)

>It turns out that key2 can differ from key2.

key2 differ from key1?

>We can't keep the file "Name1" open and we can't mmap the file.

My case was st_dev changing without reboots on a NFS filesystem.

>I didn't understand with -ipo or +O4.

These are compile options that cause the link step to take a long time. So long that the automounter unmounts the filesystem.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed between reboots on HP-UX?

Alex_Vinokur — Sat, 07 Jan 2012 18:22:52 GMT

>>1. My original post was edited (the link was deleted).
> (You didn't need to delete it, just make it easy to click on. :-)
I don't know why the link doesn't work. From my site it works

>>It turns out that key2 can differ from key2.
> key2 differ from key1?

Of course. My typo, sorry.
So, can key2 differ from key1?

>>We can't keep the file "Name1" open and we can't mmap the file.
> My case was st_dev changing without reboots on a NFS filesystem.
Yes, changing st_dev caused changing key returned by ftok(). We want to avoid that.

-------------------------------
>>I didn't understand with -ipo or +O4.
> These are compile options that cause the link step to take a long time. So long that the automounter unmounts the filesystem.

What should be compiled? It seems that I don't understand that issue.
------------------------------

Thanks.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Sun, 08 Jan 2012 20:44:09 GMT

>I don't know why the link doesn't work. From my site it works

What was displayed worked, "http:". But if you clicked on the link, "wlmailhtml:" didn't.

>Of course. My typo. So, can key2 differ from key1?

(You can go back and edit it.)

Yes, I explained how. If over NFS and the file gets unmounted, you get a different st_dev.

>changing st_dev caused changing key returned by ftok(). We want to avoid that.

It doesn't look like you can unless you keep it open. It seems they need to beef up the WARNINGS section.

>What should be compiled? It seems that I don't understand that issue.

I was saying there are several applications that also depend on stat(2) and st_dev.

1) Linker to check dependent shlibs, fails if it takes a long time.

2) dld.so to check if already loaded. This works, since files are mmap(2).

3) aCC compiler driver. This works since stdout/stderr is still open. And wouldn't matter anyway. ;-)

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed between reboots on HP-UX?

Alex_Vinokur — Sun, 08 Jan 2012 08:17:00 GMT

Hi,

>>I don't know why the link doesn't work. From my site it works
>What was displayed worked, "http:". But if you clicked on the link, "wlmailhtml:" didn't.

I edited my original message once again.
Added:
See http://groups.google.com/group/comp.unix.programme?r/browse_frm/thread/018671b4d364409f

>>Of course. My typo. So, can key2 differ from key1?
>(You can go back and edit it.)
Thanks. Edited.

>Yes, I explained how. If over NFS and the file gets unmounted, you get a different st_dev.

>>changing st_dev caused changing key returned by ftok(). We want to avoid that.

>It doesn't look like you can unless you keep it open. It seems they need to beef up the WARNINGS section.
They must beef up the WARNINGS section. But actually it doesn't help.

Currently it seems to there be a bug in design of ftok().

key = ftok("Name1", 'A').
File "Name1" is existing an accessible and never deleted.

Of course, an user expects to get the same key in all calls of ftok(), at least, between reboots.

--------------------------------------------

HP-UX hpx418 B.11.23 U ia64 1139467043 unlimited-user license

> man ftok

ftok(3C) ftok(3C)

NAME
ftok() - create interprocess communication identifier

SYNOPSIS
#include <sys/ipc.h>

key_t ftok(const char *path, int id);

DESCRIPTION
      All interprocess communication facilities require the user to supply a
      key to be used by the msgget(), semget(), and shmget() system calls to
      obtain interprocess communication identifiers (see msgget(2),
      semget(2), and shmget(2)).

ftok() returns a key based on path and id that is usable in subsequent
msgget(), semget(), and shmget() system calls.

The parameters for the ftok() function are as follows:

path must be the path name of an existing file
that is accessible to the process.

    id         is a character that uniquely identifies a
          project. This means that only the low order
          8-bits of id are significant. Note that
          ftok() returns the same key for linked files
          when called with the same id and that it
          returns different keys when called with the
          same file name but different ids.

RETURN VALUE
ftok() returns (key_t)-1 if path does not exist or if it is not
accessible to the process.

EXAMPLES
The following call to ftok() returns a key associated with the file
myfile and id A:

key_t mykey;
mykey = ftok ("myfile", 'A');

WARNINGS
      If the file whose path is passed to ftok() is removed when keys still
      refer to the file, future calls to ftok() with the same path and id
      will return an error. If the same file is recreated, ftok() is likely
      to return a different key than it did the original time it was called.

SEE ALSO
intro(2), msgget(2), semget(2), shmget(2), thread_safety(5).

Hewlett-Packard Company - 1 - HP-UX 11i Version 2: August 2003

--------------------------------------------

That man is very "smart":
1. No info about (the same or not the same) key returned by ftok() with file that wasn't deleted.
2. No info about reboots
3. No info about mounting/unmounting.

>>What should be compiled? It seems that I don't understand that issue.

>I was saying there are several applications that also depend on stat(2) and st_dev.

>1) Linker to check dependent shlibs, fails if it takes a long time.

>2) dld.so to check if already loaded. This works, since files are mmap(2).

>3) aCC compiler driver. This works since stdout/stderr is still open. And wouldn't matter anyway. ;-)
....

I didn't understand how we can use it.

We have situation like this.

int main()
{
key = ftok("Name1", 'A');

   id = shmget (key, ..., ...);
   // If Shm segment with key is not existing we create, it.
   // Otherwise we must work with existing Shm segment
   // So, if while run-1 we get key1, while run-2 we get key2
   //      and key2 != key1
   //      then

// 1) data from Shm segment key1 will be lost.

// 2) we never can get key1 via ftok()

   shmat(...);

   ...

shmdt();

...

}

Let myexe is executable of that main()

Now, we work as it follows:

====================================
create file "Name1"

...

reboot-1

myexe

...

myexe

...

myexe

...

reboot-2

myexe

...

myexe

...

myexe

...

reboot-3

====================================

OK, keys in intervals (reboot-1, reboot-2) and (reboot-2, reboot-3) may be different. No problem.

How can we use linker options and aCC compiler driver to guarantee the same key within each interval in our situation?
We have myexe + libraries already created.

Dennis, thank you very much.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Sun, 08 Jan 2012 20:43:00 GMT

>I edited my original message once again.

Yes, that works fine.

>They must beef up the WARNINGS section. But actually it doesn't help.

>Currently it seems to there be a bug in design of ftok().

There is nothing they can do about it unless key_t is larger. Or an index in some ftok(3) table.

>2. No info about reboots

A computer is not suppose to go down and that isn't defined by any standard. And if does go down, you will have to get your shared memory again, anyway.

>3. No info about mounting/unmounting.

That's something that should be defined, since mount/umount are defined in the standard.

>I didn't understand how we can use it.

Make sure that the file is on a filesystem that isn't or can't be unmounted.

>How can we use linker and compiler driver to guarantee the same key within each interval in our situation?

I was pointing out other applications that may have the same problem you do. And how to possibly workaround the problem. The linker is a real world case that I've ran into many times before and finally we got a RCA.

>keys in intervals (reboot-1, reboot-2) and (reboot-2, reboot-3) may be different.

(Better to not even mention reboots.)

>We have myexe + libraries already created.

That may be your solution. Pick myexe as the name of your file. That can't be unmounted.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Mon, 09 Jan 2012 05:09:23 GMT

>> They must beef up the WARNINGS section. But actually it doesn't help.
>> Currently it seems to there be a bug in design of ftok().
> There is nothing they can do about it unless key_t is larger. Or an index in some ftok(3) table.
Yes.

>> 2. No info about reboots
> A computer is not suppose to go down and that isn't defined by any standard. And if does go down, you will have to get your shared memory again, anyway.
Of course. Different keys after different reboots is not a problem because shared memory is volatile memory.
About "that isn't defined by any standard". However, people are working in real circumstances (with down), and it would be useful to note this.

>> 3. No info about mounting/unmounting.
> That's something that should be defined, since mount/umount are defined in the standard.
It is very important. Thanks.

>> I didn't understand how we can use it.
> Make sure that the file is on a filesystem that isn't or can't be unmounted.
That the program works with many of our customers. We have no control over this.

>That may be your solution. Pick myexe as the name of your file. That can't be unmounted.

Where to pick? Via any program that "works" with myexe .

Dennis, thank you.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Mon, 09 Jan 2012 20:58:00 GMT

>Where to pick? Via any program that "works" with myexe.

Use the myexe executable itself.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Tue, 10 Jan 2012 08:13:22 GMT

>> Where to pick? Via any program that "works" with myexe.

> Use the myexe executable itself.

Our situation is more complicated. We need several ftok-files, their names depend on some input parameters from database.

OK, we will think of that.

Dennis, thank you.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Tue, 10 Jan 2012 08:45:00 GMT

>We need several ftok-files

You can use the same file and up to 256 values of "id".

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Tue, 10 Jan 2012 10:19:22 GMT

>> We need several ftok-files

> You can use the same file and up to 256 values of "id".

Our "id's" are actually some strings (that are selected from database).

But there is another problem: file "myexe" may be updated (i.e. re-created) from time to time.

We will investigate another solution (possible using also existing ftok()-function).

Thanks.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Tue, 10 Jan 2012 17:35:56 GMT

>another problem: file "myexe" may be updated (i.e. re-created) from time to time.

Are there multiple executables that access the shared memory segment? Or you want it to stick around after you update myexe?

I assume you realize that if you file isn't on NFS, you don't have to worry?

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Wed, 11 Jan 2012 06:11:32 GMT

>>another problem: file "myexe" may be updated (i.e. re-created) from time to time.

>Are there multiple executables that access the shared memory segment? Or you want it to stick around after you update myexe?

No, we don't have multiple executables that access the shared memory segment. The only executable creates (or attaches) different shared memory segments, Names of ftok-files are built on basis some info from data base.

>I assume you realize that if you file isn't on NFS, you don't have to worry?
....

1. It seems that I don't understand this issue. Where may file be: on NFS, something else?

2. But we actually came across this problem: ftok() returned other key for the same non-recreated file between reboots.

Thanks

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Wed, 11 Jan 2012 07:33:42 GMT

>1. Where may file be: on NFS, something else?

Well more precisely, somewhere where it can be unmounted and mounted. With the automounter, this is transparent.

>2. But we actually came across this problem: ftok() returned other key for the same non-recreated file between reboots.

On HP-UX and not NFS or umount/mount?

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Wed, 11 Jan 2012 08:08:14 GMT

>>1. Where may file be: on NFS, something else?

>Well more precisely, somewhere where it can be unmounted and mounted. With the automounter, this is >transparent.

I am not familiar with that issue.

>>2. But we actually came across this problem: ftok() returned other key for the same non-recreated file between reboots.

>On HP-UX and not NFS or umount/mount?

Dennis, sorry. My mistake.

We came across that on Linux. After that I started to check the ftok()-problem on other platforms.

Thanks

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Wed, 11 Jan 2012 19:40:29 GMT

>We came across that on Linux.

As far as I know this only happens transparently with NFS and automounter. Or some sysadmin manually unmounting and mounting filesystems.

Were there any other cases for Linux?

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Alex_Vinokur — Thu, 12 Jan 2012 06:15:59 GMT

@Dennis Handly wrote:
>We came across that on Linux.

As far as I know this only happens transparently with NFS and automounter. Or some sysadmin manually unmounting and mounting filesystems.

Were there any other cases for Linux?

About other cases for Linux - I don't know.

I came accross that accidently, because it happened on Linux-machine that I have nothing to do with it.

How was the problkem detected?

While each reading the ftok file I write to that file some relevant info, including generated Key (see attachment TC_SHMTOK1_avinokur_TTT2_TTT1.docx).

In addition, on request we can get brief info about ftok-file (see attachment FtokFilesInfo.docx), that includes all generated Keys for ftok-files.

I was asked about something for that machine and accidently detected that some row in FtokFilesInfo.docx contained 3 keys (not the only key, as it was expected).

It seems, that two keys were from the same reboot interval (so, here we have problematic situation).

Third key was from other reboot interval (so, it may be OK).

I think, we should check generated keys within the same reboot interval (not only write to ftok-file).

Thanks.

Re: [ftok()-related] Can 'st.st_dev of the same file' be changed without reboots on HP-UX?

Dennis Handly — Thu, 12 Jan 2012 06:43:28 GMT

>How was the problem detected?

>While each reading the ftok file I write to that file

> /devjuser1/jp/ccjp/avinokur/tmp/shm_keys_holder

Did someone remove the file since it was in "tmp"?