File got corrupted. Zeros are removed !

[donatella sabellico]

Hi,I am sending this like a "message in the bottle"..
I would like to have some hints on this problem.I had a binary file in which all the zeros (0) are missing(!).We never saw a corruption like that.
Comparing the Good file againt the corrupted one we discovered that the two files are equivalent except for the fact that within the Bad file
all the zeros (0) are missing.
Our correct file had record has a fixed size of 512 bytes. Since the
record space is not completely used, there is some field that is set to
0 (zero).
The corruption of this file is evident but we cannot figure out who or what on HP11i can cause it (I dont' consider the fact that intentionally caused it) We simulate this opening the binary file with the vi and then save it with wq!. On this machine there is Openview and also Data protector HP. 11.11i.
The corruption is evident also because the orginal (good file) was of like 2mb.
When the corruption occurred it was 500k.
The zeros are not more there.
We had different records of 512 bytes each and part of this 512 were written with significant data.
Our Application write/read on this file each time 512 bytes (not less!) but our file always grows (in this case it decrease of size) because we wrote on them;
Some other part of 512 are not used and they are filled with zeros.
In the whole file, in each 512 records, the zeros are not more there!
The file was significant smaller that the original one and the zeros are not more there. We also verify this situation using HexComp a plug-in of Beyondcompare.

We don't do any action that can cause that the file will be more smaller of the original one.
What I saw using HexCoomp
is that on each of 512 records of this file the zeros are not more there.
We don't do that from code point of view.

Thx.

[James R. Ferguson]

Hi Donatella:

This is your third post of the same problem. You had considerable, very good feedback, in my opinion in at least one other thread.

It is to everyone's benefit to continue the dialog in one thread rather than post and repost.

PLEASE SEE:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=954374

and:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=954511

Regards!

...JRF...

[Rodney Hills]

Don't no if this is relevant, but as I understand unix writes, if you start with an empty file, write to the 1st sector, then write to the 99th sector, unix will only allocate disk space for the 2 sectors. The other sectors will no be their and will appear as nothing if you try to read.

HTH

-- Rod Hills

[donatella sabellico]

Hi, thanks for your attention to monitor the answers. Anyway I am trying to avoid a big customer escalation (they did already opened a ticket against HP) and what I am asking are *hints* that can cause this problem instead of point against my application that was deeply debug and we can prove that we are not the cause of the corruption.I was only searching an OS expert.
Thanks anyway.

[Patrick Wallek]

The way I see it, there are 2 ways this could have happened:

1) An application error, which you don't believe.

2) Human error. Someone did some command against that file and messed it up. There are a number of commands that could cause corruption of a binary file. If you have shell history enabled for users, go through them and see if the filename of your data file shows up.

[A. Clay Stephenson]

GOOD IDEA. Blame the OS BUT if I were your customer my very first question would be "How does it do it to just your file?"
Maybe the OS knows your pathname. Why not rename the file? Somehow, I'll bet that mean OS still decides to trash just your file. I am sometimes amazed at how my own perfect, error-free code manages to have a bug every now and then.



Is the data that used to be zero's garbage now? If so that might be unitialized data.

It would really help if you you at least explain if you are using low-level io (read,write,lseek) or hi-level i/o (fread,fwrite,fseek) or even text-based i/o (fprintf).

Does you application code indicate any sort of error when this happens?

Is your code multi-threaded? If so, are you using thread-safe mechanism's? Those are the kinds of issues that can prove very difficult to ytrackdown and even seem to be OS related but are more accurately the misuse of functions.

[Rory R Hammond]

Hi I created a binary file with "zeros" by:

rm xxx
for arg in 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
do
echo "\0" |tr -d"\n" >> xxx
done

The above script created a file with 17 binary "zeros" otherwise known as nulls (od xxx or any hex viewer will ussually show them a zeros)

I then used vi to open the file.
Sure enough a wq! emptied the file.

You are either checking with or using a text oriented application to view or modified a binary file. Text applictions written in c uses arrays to minuplate strings. The simple answer is that a "null" within this type of applications is a end of string, thus truncating all of your zeros or "nulls". You have to make sure that you use applications that are binary aware when manipulating your particular data.

Your applications have to be aware of string lengths, Apps that process binary files need to read and write and update data using fixed blocks, or at least well defined block length based on files size.