topic POSIX file range locking problem in Operating System - OpenVMS

POSIX file range locking problem

zinal — Wed, 31 Oct 2012 20:27:59 GMT

Hello and have a good time of day!

I'm trying to build current SQLite (3.7.14.1) under OpenVMS V8.4 using HP C X7.3-289.

I've managed to create a small patch, which allows SQLite to work under VMS:

http://sqlite.1065341.n5.nabble.com/Building-SQLite-3-7-14-1-for-OpenVMS-td65277.html

The problem is that POSIX file range locking used in SQLite refuses to work. The very first call to fcntl(... F_SETFL ...) which tries to set a shared single-byte lock returns (-1) and errno is set to 75.

strerror(75) returns "no locks available", so I suspect that I have some sort of a system configuration problem.

The most strange thing is that when I wrote a small testcase which opens a file, reads a block from it and then establishes the very same lock - that test just work as expected, no problems at all.

There is a significant difference between my small test and real SQLite code in that SQLite sets numerous defines, including NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE. But IMHO this should not result in any such a strange behaviour.

Any thoughts? Any help will be appreciated.

Best regards,

Maxim Zinal

Re: POSIX file range locking problem

Craig A Berry — Fri, 02 Nov 2012 14:56:58 GMT

Since we can't see your simple example, it's hard to know what you're doing. The SQLite patch appears to have you opening every file like so:

 open(zFile, flags, mode, "shr = del, get, put, upd", "ctx = stm", "ctx = bin");

So you're explicitly setting the sharing characteristics of the entire file. Then you're trying to explicitly set the sharing characteristics of portions of the file using the byte range locking feature of fcntl. I don't know if the two mix. I don't know if they don't, either, but it's something to look into.

Re: POSIX file range locking problem

zinal — Wed, 07 Nov 2012 18:01:31 GMT

Hello and thank you for your answer!

My simple test looks like the following:

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char* argv[]) {
  struct flock fl;
  int status;
  int fd = open("sample.c", 0, 0644,
                "shr = del, get, put, upd", "ctx = stm", "ctx = bin");
  if ( fd==-1 ) {
    perror("Cannot open file");
    return 0;
  }
  fl.l_type = F_RDLCK;
  fl.l_whence = SEEK_SET;
  fl.l_start = 1073741824;
  fl.l_len = 1;
  status = fcntl(fd, F_SETFL, &fl);
  if ( status==0 )
    printf("OK [1]!\n");
  else
    perror("Cannot lock file");
  close(fd);
  return ERROR_SUCCESS;
}

It works as expected: advisory lock is taken successfully (no error), and a newly created file cannot be deleted, because it is still open.

A modified SQLite file open and locking code looks nearly identical, but the very first call to fcntl(F_SETFL) returns -1 with errno set to 75. This is an example debugging trace from SQLite:

OPENX   3   /datum$dkb0/users/zinal/sqlite/test.db 01002
OPEN    3   /datum$dkb0/users/zinal/sqlite/test.db
READ    3     100       0 0
LOCK    3 SHARED was NONE(NONE,0) pid=874008 (unix)
fcntl T=0 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=-1073724143 S=-1 ERR=75
fcntl-failure-reason: UNLCK LS=1073741824 LL=1 PID=-1073724143
LOCK    3 SHARED failed (unix)
UNLOCK  3 0 was 0(0,0) pid=874008 (unix)

Please note the "fcntl-failure-reason" line which contains "UNLCK" - no lock prevents new lock to be taken.

I also tried to remove file-sharing flags from the open() call in sqlite3.c, but got the same error condition and exactly the same trace. The very reason for setting file-sharing flags when opening the database file is the need to enable multiple processes open the same file concurrently.

So I still don't know the reason for non-working POSIX locks.

The only thing I can try is to play with compilation options (_LARGEFILE, etc) to reproduce the same error in a small testcase.

Best regards,

Maxim Zinal

Re: POSIX file range locking problem

zinal — Wed, 07 Nov 2012 18:11:40 GMT

One more thing - as a quick test I appended the contents of my test main() function to sqlite3.c, compiled it with all the required compilation options (NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE,_POSIX_EXIT=1), and linked just sqlite3. The test worked - no errors again.

So I'm completely stuck, compilation flags are irrelevant here.

Re: POSIX file range locking problem

H.Becker — Thu, 08 Nov 2012 11:55:06 GMT

On V8.3/Alpha (I don't have any PATCH/ECO info from this node), it (sqlite3 with your patch - which has a few annoying wraps and an unexpected flag 'b') seems to work (with a hacked getrusage(), /noopt - otherwise with SQLITE_DEBUG the compiler becomes too hyungry for memory - and a static link):

$ mc []shell x.db

SQLite version 3.7.14.1 2012-10-04 19:37:12

Enter ".help" for instructions

Enter SQL statements terminated with a ";"

sqlite> .database

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=2061392752 S=0 ERR=2

seq name file

--- --------------- ----------------------------------------------------------

0 main /usr_ods5/becker_h/sqlite-amalgamation-3071401/x.db

sqlite> .tables

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=1850896 S=0 ERR=2

phonebook

sqlite> select * from phonebook;

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

Joe|498912345678

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=2079260464 S=0 ERR=2

sqlite>

Re: POSIX file range locking problem

zinal — Thu, 08 Nov 2012 17:50:36 GMT

Could you please post your compilation and linking options? And the required hack for getrusage()?

Re: POSIX file range locking problem

H.Becker — Thu, 08 Nov 2012 19:47:54 GMT

$ mc GNV$GNU:[bin]diff -u shell.c~ shell.c

--- shell.c~ Thu Nov 8 05:00:36 2012

+++ shell.c Thu Nov 8 04:40:32 2012

@@ -97,6 +97,10 @@

/* Saved resource information for the beginning of an operation */

static struct rusage sBegin;

+#define RUSAGE_SELF 0

+void getrusage(int who, struct rusage *usage) {

+ memset (usage, 0, sizeof *usage);

** Begin timing an operation

$ cc/opt/define=(NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE,_POSIX_EXIT=1,SQLITE_DEBUG,SQLITE_LOCK_TRACE) -

/noopt -

/float=ieee/ieee=denorm -

sqlite3.c

$ ! two informationals, one warning

$ cc shell

$ link shell,sqlite3

$ ! linker warning because of the compiler warning

Re: POSIX file range locking problem

GuentherF — Sat, 08 Dec 2012 02:46:55 GMT

Shouldn't the 2nd parameter for fcntl() to do locking be F_SETLK rather than F_SETFL?

/Guenther

Re: POSIX file range locking problem

Ph Vouters — Tue, 18 Dec 2012 10:05:00 GMT

Provided you had read my previous post about SQLite on OpenVMS, you would have read the document I posted the URL to. Provided you would have had carefully read this URL, you would have had downloaded SQLite from my Web site and you would not have got any compile-tlme warnings. Provided you would have read this URL, you have had understood you can use SQLite in a single VMS process but NOT in two concurrent VMS processes accessing the same SQLite database. Provided you would have had a VMS support contrat with HP, you would have logged a call at HP to get the described problem fixed without the need to ask people in this forum. Provided you had read the URL I posted, you would NOt have had reinvented the wheel.

Philippe

Re: POSIX file range locking problem

zinal — Tue, 18 Dec 2012 10:42:29 GMT

Thank you for your reply. It's always nice to know that someone tries to solve the very same problem as you.

Provided that I wrote my original post at November, 01, 2012, your "previous post about SQLite on OpenVMS" did not exist at that time, nor did your URL-linked web document. So your words about "reinventing the wheel" seem pretty strange to me, really.

Provided that you are sure you've found a bug in one of the most well-tested and widely-used VMS components, you seem to ignore the signs that there is really some sort of a VMS-specific resource allocation problem.

Provided...

Re: POSIX file range locking problem

Ph Vouters — Tue, 18 Dec 2012 11:48:22 GMT

So would it happen someone answered your post, making your post suddenly appears first ? As far as it looks, Guenther resurrected your post.

If you or any reader interested in running SQLite on OpenVMS and owning a support contract with HP, he has to open a call at HP if he wants to get this described problem cured. Although I have contacts within HP, I have no HP contract. HP won't move unless a duly logged call. And this is not sure, even with a support contract, that HP OpenVMS engineering will ever fix the problem.

My guess is that HP will move because its PostGreSQL VMS port might depend upon the resolution of the described problem.

Philippe