createing files with the same permissions from two different machines

[Charles Holland]

OK Guru's here goes. Three systems all running 11.11 all patched to current. System "C" has a directory exported to "A" and "B".

When files are written from "A" to "C" (via purchased program) the permissions of the directories that are created are 775 and the files under that are created with 666. - GREAT! -

When files are written from "B" to "C" (via purchased program) the permissions of the directories that are created are 755 and the files under those are created with 644 - NOT GOOD -.

The user and group are created correctly from both machines.

Users on "A" and "B" are identical... same home directory, profiles....

Directory structure (for purchased program) has been copied from "B" to "A" in an effort to cause "A" to then fail also.

No luck it created files just the same as always.

[A. Clay Stephenson]

This is almost certainly a difference in the umasks.

[Charles Holland]

If you log in to "A" and "B", type umask you get "00" on both systems.

[A. Clay Stephenson]

Well in that case, the difference has to be the creation mode of the two programs. When a file is created, one of the arguments to the open() or creat() system call is the mode; umask is then NAND'ed with this mode to produce the actual mode of the file. Are the two programs identical? Have you cksum'ed them to be sure? Is there a configuration file for your program? You could run the code under tusc and look for the arguuments to open and creat and there has to be a difference.

[Patrick Wallek]

I have to agree with Clay 100%. This is a classic umask issue. It appears to be set to 002 on one machine and 022 on another.

Perhaps the purchase program is doing something different on one machine than the other?

Processes / programs can modify their umask so it may not be the same as when you check it from the shell prompt.

[A. Clay Stephenson]

By the way, umask is often set in .profiles so they problem could be tied to a small group of users. You need to be able to produce different file modes with the same login and then test that user's umask.

[Charles Holland]

A.Clay and Patrick... Are the programs identical? Yes, they are identical. We created a tar on system "B" (which wasn't creating the directory/file permissions we wanted) and untared it on "A" fully expecting it to create bad directories/files there as well.

BTW this directory structure also contains the home directory for the use. Again, if I copy the directory, I also copy the home directory and therefore the profiles as well.

In the home directory, if "grep -i mask .*" the only response you get is from the .sh_history file.

[Biswajit Tripathy]

I'm assuming that directory in 'C' is NFS mounted on
machine 'A' and 'B'. Another potential source of
problem could be the NFS mount permissions on
machines 'A' and 'B'.

- Biswajit

[Charles Holland]

Biswajit,
The directory on "C" is exported the same to both "A" and "B".

The entries in the /etc/fstab on "A" and "B" are identical. I believe this answers your question.

[Darrel Louis]

Charles,

Can you please check if umask is set in /etc/profile; /etc/default/security;

Clay is right, the only other thing I can think of is that the user on server B is using a different shell to create the file on server B.

If you create a file on server B in /tmp, what is the permission for this file.

Darrel

[A. Clay Stephenson]

At this poiint it's time to isolate the problem. I assume that the same user on both boxes has the the same UID and GID. As user "tom" on both boxes, explicitly do a umask 000 and then echo "test" > newfile. Is newfile created with identical mode from both nodes? If so, the difference is in the application.

[Biswajit Tripathy]

Charles,

>Biswajit,
> The directory on "C" is exported the same to both
> "A" and "B".

Okay, but that's only first part of it. Are they also
mounted with same permission on both A and B ?
Make sure /etc/fstab mount permission is also exactly
same on A and B.

- Biswajit

[Fred G. Claypool, Jr.]

Hello everyone,

I'm a colleague of Chuck's. I've got some follow-up information for you all to consider.

The user on servers A & B (the user under whose account the "purchased program" runs) is UID 111 with GID 5. A. Clay's umask 000 test on both machines yields the same results: 666 permissions. Therefore, logic suggests that the problem does exist in the application.

However, as Chuck mentioned before, the application (contained within a single filesystem along with that user's home directory) was copied from host B to A. Therefore, the application is identical between both servers too.

There is no umask setting within either /etc/profile files, and none of the systems contain /etc/default/security.

Concerning Biswajit's idea about /etc/fstab, both /etc/fstab's contain identical lines:

:/ext_dir /ext_dir nfs rw,suid,soft,bg 0 0

Both users on A & B use the very same shell, too: /usr/bin/sh

So there you have it. We have the very same application software, very same users, very same environment produce different results.

We feel drawn to the concept that something outside of this application -- something on the HP-UX level for example -- is causing these different results.

[Fred G. Claypool, Jr.]

Something else to consider...

We also tried to use tusc to trace what was going on with this problem, but I'm afraid we're not familiar enough with that tool in order to understand what we see. To compound the tracing difficulty, the "purchased program" runs two daemons even while it isn't necessarily doing anything. The second PID of these two daemons forks three other programs. It is our belief that one of these three other programs is the actual executable that produces the files with undesired permissions.

In our brief experimentation with tusc yesterday, we never saw any mention of the created files, but that may have been because we only started tusc with the PIDs of those two initial daemons. (The PID of the program that we feel actually writes the files is unknown, and it only "stays around" for a matter of moments.)

[A. Clay Stephenson]

Now that the symptom has been isolated to the application (at least to my satisfaction), it's now time to figure out why. I mentioned earlier that the difference could be in the existence of a "configuration file" (e.g. an .ini file) that might be anywhere on the box. On one box the file is found and on the other it is not so default behavior is asserted. A very powerful technique (and I mentioned this earlier as well) is to employ tusc to examine the system calls and their arguments invoked by this application. There has to be a difference in the open(), creat(), and/or umask() system calls on these two platforms.
It would also be helpful to do an ldd on your application and compare via cksum each shared library used on both platforms. At some point, this application zigs on one platform and zags on the other.

[Fred G. Claypool, Jr.]

A. Clay,

After experimenting with tusc for a while, we've finally managed to capture the creation of the files in question. However, if we're interpreting tusc correctly, the files do not end up with permissions as tusc says they should.

Here's what the end result looks like. (Before running the test, /ext_dir was completely empty.)

legends: /ext_dir
# ls -alR
total 20
drwxrwxrwx 3 jetform ixossys 2048 Feb 24 08:22 .
drwxr-xr-x 23 root root 8192 Feb 7 15:49 ..
drwxr-xr-x 3 jetform daemon 96 Feb 24 08:22 ext536

./ext536:
total 6
drwxr-xr-x 3 jetform daemon 96 Feb 24 08:22 .
drwxrwxrwx 3 jetform ixossys 2048 Feb 24 08:22 ..
drwxr-xr-x 2 jetform daemon 1024 Feb 24 08:22 24022005044

./ext536/24022005044:
total 30
drwxr-xr-x 2 jetform daemon 1024 Feb 24 08:22 .
drwxr-xr-x 3 jetform daemon 96 Feb 24 08:22 ..
-rw-r--r-- 1 jetform daemon 111 Feb 24 08:22 COMMANDS
-rw-r--r-- 1 jetform daemon 11649 Feb 24 08:22 DATA.PDF
-rw-r--r-- 1 jetform daemon 293 Feb 24 08:22 IXATTR
-rw-r--r-- 1 jetform daemon 0 Feb 24 08:22 LOG

And here's every line from our tusc trace that has the phrase "/ext_dir" in it or around it:

stat(0x40006860, 0x7f7f1248) .............. [entry]
stat("/ext_dir", 0x7f7f1248) .............. = 0

stat(0x7f7f0fb0, 0x7f7f1248) .............. [entry]
stat("/ext_dir/ext536", 0x7f7f1248) ....... ERR#2 ENOENT

mkdir(0x7f7f0fb0, 0775) ................... [entry]
mkdir("/ext_dir/ext536", 0775) ............ = 0

stat(0x7f7f0fb0, 0x7f7f1248) .............. [entry]
stat("/ext_dir/ext536", 0x7f7f1248) ....... = 0

stat(0x7f7f10b8, 0x7f7f1248) .............. [entry]
stat("/ext_dir/ext536/24022005044", 0x7f7f1248) ERR#2 ENOENT

stat(0x7f7f10b8, 0x7f7f1248) .............. [entry]
stat("/ext_dir/ext536/24022005044", 0x7f7f1248) ERR#2 ENOENT

mkdir(0x7f7f10b8, 0775) ................... [entry]
mkdir("/ext_dir/ext536/24022005044", 0775) = 0

open(0x7f7f0cf8, O_WRONLY|O_CREAT|O_TRUNC, 0666) [entry]
open("/ext_dir/ext536/24022005044/ar3DA3421DD50A31a.dat", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4

open(0x80000001001b81d0, O_RDONLY, 0666) .. [entry]
open("/ext_dir/ext536/24022005044/ar3DA3421DD50A31a.dat", O_RDONLY|O_LARGEFILE, 0666) = 4

open(0x80000001001b6290, O_RDONLY, 0666) .. [entry]
open("/ext_dir/ext536/24022005044/ar3DA3421DD50A31a.dat", O_RDONLY|O_LARGEFILE, 0666) = 4

open(0x80000001001b8250, O_WRONLY|O_CREAT|O_TRUNC, 0666) [entry]
open("/ext_dir/ext536/24022005044/jfixosa0.pdf", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 5

open(0x40005bd0, O_RDONLY, 0666) .......... [entry]
open("/ext_dir/ext536/24022005044/jfixosa0.pdf", O_RDONLY, 0666) = 4

open(0x40005bd0, O_RDONLY, 0666) .......... [entry]
open("/ext_dir/ext536/24022005044/jfixosa0.pdf", O_RDONLY, 0666) = 4

open(0x40005d78, O_RDONLY, 0666) .......... [entry]
open("/ext_dir/ext536/24022005044/DATA.PDF", O_RDONLY, 0666) ERR#2 ENOENT

lstat64(0x40005d78, 0x7f7f19f0) ........... [entry]
lstat64("/ext_dir/ext536/24022005044/DATA.PDF", 0x7f7f19f0) ERR#2 ENOENT

rename(0x40005bd0, 0x40005d78) ............ [entry]
rename("/ext_dir/ext536/24022005044/jfixosa0.pdf", "/ext_dir/ext536/24022005044/DATA.PDF") = 0

open(0x7f7f0f90, O_WRONLY|O_APPEND|O_CREAT, 0666) [entry]
open("/ext_dir/ext536/24022005044/IXATTR", O_WRONLY|O_APPEND|O_CREAT, 0666) = 4

open(0x7f7f1118, O_WRONLY|O_CREAT|O_TRUNC, 0666) [entry]
open("/ext_dir/ext536/24022005044/COMMANDS", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4

open(0x7f7f0f90, O_WRONLY|O_CREAT|O_TRUNC, 0666) [entry]
open("/ext_dir/ext536/24022005044/LOG", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4

unlink(0x7f7f0cf8) ........................ [entry]
unlink("/ext_dir/ext536/24022005044/ar3DA3421DD50A31a.dat") = 0


I assume that those "0775" and "0666" references are the directory/file permissions. But if so, how can the end result look different from the trace?

Thanks.

[Robert-Jan Goossens_1]

Fred + Charles,

Check this doc.

Europe
http://www4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062936612

US
http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062936612

Regards,
Robert-Jan

[Fred G. Claypool, Jr.]

Robert-Jan,

Interesting idea... thanks for sharing it, but it doesn't look like it's applicable here.

For one, the file referenced in the link (/etc/netlinkrc) doesn't even exist on the NFS server ("legends").

However, to remove any possibility of doubt, I unmounted /ext_dir from both servers that host the application. That left me with a simple local directory, /ext_dir.

I then reran my test. One host yields the 666 permissions and the other host yields 644. In other words, the exact same results are generated when an NFS mountpoint is involved as well as when no NFS mountpoint is involved.

This is quite a puzzler, eh? :(

Thanks.

[Robert-Jan Goossens_1]

Could you check the umask variable in the inetd script on the servers

# grep umask /sbin/init.d/inetd

Regards,
Robert-Jan

[Charles Holland]

Robert-Jan
The output of your request, on all 3 servers in question is the same......

# grep mask /sbin/init.d/inetd
mask=`umask`
umask 000
umask $mask

[A. Clay Stephenson]

Okay, tusc indicates that the open()'s are done with the expected mode arguments asserted (0775 and 0666). So how can the results be different? Easy. Even with 0666 for the mode, umask still comes into play and it is NAND'ed with the mode in the open call to yield the actual mode. Another possibility is the the files are chmod()'ed after the fact but that is unlikely. Now that you are getting good with tusc, look for ulimit(), chmod(), and fchmod() system calls as well. Ulimit()'s can be done in the application code as well as the shell and the last one wins because a given ulimit applies until changed by a subsequent one or until the process terminates.

[Fred G. Claypool, Jr.]

A. Clay,

We looked for ulimit(), chmod(), and fchmod() calls in the trace file, but found none.

However, we did find four umask() calls, and sure enough, on one host, these umask calls were set to 022 and on the other host, they were set to 0.

This definitely points to a difference in system configuration, but we can't imagine anything that we've done on the OS level over the past year or so that would account for anything like this. (We keep a change log, but we can't imagine anything done that would cause this.)

Is there any way to determine exactly why these umask() calls are different?

Thanks.

[A. Clay Stephenson]

Note that the errant ulimit call is in the application. This means that somehow the application is being instructed to set this value. It's almost certainly not a difference in the OS configuration although there is a slight chance that somehow a difference in a shared library could cause this. However, the most likely explanation is the detection of some sort of configuration file on one host but not on the other or a possible environment variable defined one one host and not the other. You could look for all the open() system calls and then look at all the pathnames on the two hosts. You could also run the strings command on your application and maybe an environment variable name (or a pathname) will pop out at you. I know you have copied the directory tree from one box to the other but this imaginary configuration file could be anywhere and a popular (though bad) location for them is /.


Of course, now that you have got this puppy analyzed to this level, by far, the most expeditious approach would be to contact the developer. I suppose that approach is too radical.

[Fred G. Claypool, Jr.]

Ok, everyone, we finally have it figured out! :-)

First, let me say that Chuck & I really appreciate the help we received from everyone -- A. Clay, Patrick, Biswajit, Darrel, and Robert-Jan. I know Chuck reads the HP Forums almost daily, and we both get tremendous worth from everyone's postings.

Learning about the tusc utility was a great experience. I really think that's going to come in handy in the future.

I think the explanation of our problem might prove useful to others, simply because it doesn't necessarily seem limited to our application.

From the very beginning of this journey, we were focused in on the differences between the two servers which hosted our application. But the thing that ultimately explained the differences in output turned out to be the method by which the application was started.

Our application has its start script in /sbin/init.d, like so many other applications. During system startup, /sbin/rc encountered its start link (from /sbin/rc3.d/S950...) and properly followed the application's instructions. Careful examination of /sbin/rc will show a "umask 022" on line 15.

During our testing, however, when we would try to "change one thing" / "test one thing", we would routinely execute the start script in /sbin/init.d by hand (to first stop it, then start it). Our root accounts apparently have a default umask of 00, so this umask setting was in effect during our restarts, rather than /sbin/rc's umask setting in effect during system startup.

I just included an explicit "umask 00" within the application's /sbin/init.d startup script, and we're going to test it during a full reboot in a few minutes.

Again... thanks everyone.

[Charles Holland]

Resolved.