POSIX file range locking problem

[zinal]

Hello and have a good time of day!

I'm trying to build current SQLite (3.7.14.1) under OpenVMS V8.4 using HP C X7.3-289.

I've managed to create a small patch, which  allows SQLite to work under VMS:

   http://sqlite.1065341.n5.nabble.com/Building-SQLite-3-7-14-1-for-OpenVMS-td65277.html

The problem is that POSIX file range locking used in SQLite refuses to work. The very first call to fcntl(... F_SETFL ...) which tries to set a shared single-byte lock returns (-1) and errno is set to 75.

strerror(75) returns "no locks available", so I suspect that I have some sort of a system configuration problem.

The most strange thing is that when I wrote a small testcase which opens a file, reads a block from it and then establishes the very same lock - that test just work as expected, no problems at all.

There is a significant difference between my small test and real SQLite code in that SQLite sets numerous defines, including NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE. But IMHO this should not result in any such a strange behaviour.

Any thoughts? Any help will be appreciated.

Best regards,

   Maxim Zinal

[Craig A Berry]

Since we can't see your simple example, it's hard to know what you're doing.  The SQLite patch appears to have you opening every file like so:

 open(zFile, flags, mode, "shr = del, get, put, upd", "ctx = stm", "ctx = bin");

So you're explicitly setting the sharing characteristics of the entire file.  Then you're trying to explicitly set the sharing characteristics of portions of the file using the byte range locking feature of fcntl.  I don't know if the two mix.  I don't know if they don't, either, but it's something to look into.

[zinal]

Hello and thank you for your answer!

My simple test looks like the following:

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char* argv[]) {
  struct flock fl;
  int status;
  int fd = open("sample.c", 0, 0644,
                "shr = del, get, put, upd", "ctx = stm", "ctx = bin");
  if ( fd==-1 ) {
    perror("Cannot open file");
    return 0;
  }
  fl.l_type = F_RDLCK;
  fl.l_whence = SEEK_SET;
  fl.l_start = 1073741824;
  fl.l_len = 1;
  status = fcntl(fd, F_SETFL, &fl);
  if ( status==0 )
    printf("OK [1]!\n");
  else
    perror("Cannot lock file");
  close(fd);
  return ERROR_SUCCESS;
}

 It works as expected: advisory lock is taken successfully (no error), and a newly created file cannot be deleted, because it is still open.

A modified SQLite file open and locking code looks nearly identical, but the very first call to fcntl(F_SETFL) returns -1 with errno set to 75. This is an example debugging trace from SQLite:

OPENX   3   /datum$dkb0/users/zinal/sqlite/test.db 01002
OPEN    3   /datum$dkb0/users/zinal/sqlite/test.db
READ    3     100       0 0
LOCK    3 SHARED was NONE(NONE,0) pid=874008 (unix)
fcntl T=0 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=-1073724143 S=-1 ERR=75
fcntl-failure-reason: UNLCK LS=1073741824 LL=1 PID=-1073724143
LOCK    3 SHARED failed (unix)
UNLOCK  3 0 was 0(0,0) pid=874008 (unix)

 Please note the "fcntl-failure-reason" line which contains "UNLCK" - no lock prevents new lock to be taken.

I also tried to remove file-sharing flags from the open() call in sqlite3.c, but got the same error condition and exactly the same trace. The very reason for setting file-sharing flags when opening the database file is the need to enable multiple processes open the same file concurrently.

So I still don't know the reason for non-working POSIX locks.

The only thing I can try is to play with compilation options (_LARGEFILE, etc) to reproduce the same error in a small testcase.

Best regards,

  Maxim Zinal

[zinal]

One more thing - as a quick test I appended the contents of my test main() function to sqlite3.c, compiled it with all the required compilation options (NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE,_POSIX_EXIT=1), and linked just sqlite3. The test worked - no errors again.

So I'm completely stuck, compilation flags are irrelevant here.

[H.Becker]

On V8.3/Alpha (I don't have any PATCH/ECO info from this node), it (sqlite3 with your patch - which has a few annoying wraps and an unexpected flag 'b') seems to work (with a hacked getrusage(), /noopt - otherwise with SQLITE_DEBUG the compiler becomes too hyungry for memory - and a static link):

$ mc []shell x.db

SQLite version 3.7.14.1 2012-10-04 19:37:12

Enter ".help" for instructions

Enter SQL statements terminated with a ";"

sqlite> .database 

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=2061392752 S=0 ERR=2

seq  name             file                                                      

---  ---------------  ----------------------------------------------------------

0    main             /usr_ods5/becker_h/sqlite-amalgamation-3071401/x.db       

sqlite> .tables

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=1850896 S=0 ERR=2

phonebook

sqlite> select * from phonebook;

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK RDLCK LS=1073741826 LL=510 PID=134096 S=0 ERR=2

fcntl T=2075954496 FD=3 SETLK UNLCK LS=1073741824 LL=1 PID=134096 S=0 ERR=2

Joe|498912345678

fcntl T=2075954496 FD=3 SETLK UNLCK LS=0 LL=0 PID=2079260464 S=0 ERR=2

sqlite>  

[zinal]

Could you please post your compilation and linking options? And the required hack for getrusage()?

[H.Becker]

$ mc GNV$GNU:[bin]diff -u shell.c~ shell.c

--- shell.c~    Thu Nov  8 05:00:36 2012

+++ shell.c     Thu Nov  8 04:40:32 2012

@@ -97,6 +97,10 @@

 /* Saved resource information for the beginning of an operation */

 static struct rusage sBegin;

+#define RUSAGE_SELF 0

+void getrusage(int who, struct rusage *usage) {

+       memset (usage, 0, sizeof *usage);

+}

 /*

 ** Begin timing an operation

 */

$

$ cc/opt/define=(NO_GETTOD,RTLD_GLOBAL=0,_USE_STD_STAT=1,_LARGEFILE,_POSIX_EXIT=1,SQLITE_DEBUG,SQLITE_LOCK_TRACE) -

  /noopt -

  /float=ieee/ieee=denorm -

  sqlite3.c

$ ! two informationals, one warning

$

$ cc shell

$

$ link shell,sqlite3

$ ! linker warning because of the compiler warning

[GuentherF]

Shouldn't the 2nd parameter for fcntl() to do locking be F_SETLK rather than F_SETFL?

/Guenther

[Ph Vouters]

Provided you had read my previous post about SQLite on OpenVMS, you would have read the document I posted the URL to. Provided you would have had carefully read this URL, you would have had downloaded SQLite from my Web site and you would not have got any compile-tlme warnings. Provided you would have read this URL, you have had understood you can use SQLite in a single VMS process but NOT in two concurrent VMS processes accessing the same SQLite database. Provided you would have had a VMS support contrat with HP, you would have logged a call at HP to get the described problem fixed without the need to ask people in this forum. Provided you had read the URL I posted, you would NOt have had reinvented the wheel.

Philippe

[zinal]

Thank you for your reply. It's always nice to know that someone tries to solve the very same problem as you.

Provided that I wrote my original post at November, 01, 2012, your "previous post about SQLite on OpenVMS" did not exist at that time, nor did your URL-linked web document. So your words about "reinventing the wheel" seem pretty strange to me, really.

Provided that you are sure you've found a bug in one of the most well-tested and widely-used VMS components, you seem to ignore the signs that there is really some sort of a VMS-specific resource allocation problem.

Provided...