Sample code of writing DATA to LTO tape

[SimonLTO]

 I am looking for sample c++ source code of writing binary data to LTO tape(4,5,6).. Is there anything out there? I have looked genrally on the web and nothing. Itr seems like fwrite should work but not when I try it.

[Dennis Handly]

If you aren't writing backup software, you shouldn't be writing directly to a tape.  Write to a file and use backup software to write to the tape.

>It seems like fwrite should work

Are you on Linux/Unix?  If so, you should be using write(2) and not libc routines.  And write huge blocks and make sure you have lots of memory so can keep the tape streaming.

[SimonLTO]

I'm on windows but also have tried linux.

I tried write and didn't get much further, here is what I tried:

#include "stdafx.h"
#include <stdio.h>
#include <stdlib.h>
#include <ctime>
#include <iostream>
#include <sstream>
#include <cstring>
#include <fstream>
using namespace std;
char sourceName[]="YCTIWY_7A_DN."; //file without count
char fileNameWithCount[100];
char sourcePath[]="V:\\Simon_Projects4\\LTO_test\\"; //sourcePath
char lto[]="tape0" ; //pathe to LTO
int startCount=0; //firstFrameToTAR
int stopCount=4; //last frame to TAR
 long lSize;//file size
char leadingzeros[6];
char shortpathe[50]="C:\\Users\\Cineric\\Documents\\";
char shortpathetemp[50];
void makeFrameNumber(){
 strcpy(fileNameWithCount,sourcePath);
 strcpy(shortpathetemp,shortpathe);
 if (startCount<100000) strcpy(leadingzeros,"0");
 if (startCount<10000) strcpy(leadingzeros,"00");
 if (startCount<1000) strcpy(leadingzeros,"000");
 if (startCount<100) strcpy(leadingzeros,"0000");
 if (startCount<10) strcpy(leadingzeros,"00000");
 
 strcat(fileNameWithCount,sourceName);
 strcat(shortpathetemp,sourceName);
 strcat(fileNameWithCount,leadingzeros);
 strcat(shortpathetemp,leadingzeros);
 sprintf(leadingzeros, "%d", startCount );
 strcat(fileNameWithCount,leadingzeros);
 strcat(shortpathetemp,leadingzeros);
 strcat(fileNameWithCount,".tar");
 strcat(shortpathetemp,".tar");

 
 startCount++;

}

int main()
{
     std::ofstream outfile (lto,std::ofstream::binary);

 while (startCount<=stopCount){
 
 makeFrameNumber();// create souce frame number
 
 std::cout<<shortpathetemp<<"\n";
 std::ifstream infile (fileNameWithCount,std::ifstream::binary);
 

  // get size of file
  infile.seekg (0,infile.end);
  long size = infile.tellg();
  infile.seekg (0);

  // allocate memory for file content
  char* buffer = new char[size];

  // read content of infile
 infile.read (buffer,size);
 if ( (infile.rdstate() & std::ifstream::failbit)  != 0 )
 {std::cerr << "Error opening \n";}
 
  // write to outfile
  outfile.write (buffer,size);
  if ( (outfile.fail() )  != 0 )
 {std::cerr << "Error writing \n";}
  delete[] buffer;

 
  infile.close();
 
 }

 outfile.close();
 string empname="";
cout << "Hit enter to exit:" <<endl;
getline(cin, empname);

 return 0;
}

[SimonLTO]

Just to add, the reason I'm trying to do this is our clients requires that each indiviual file (49mb each) is it's own tar file on the tape. If you try to do this with a script from linux, it takes 6 seconds a file for each tar operation, no matter the file size and what should take hours ends up taking days. I'm hoping to add the tar headers and file markers and stream the data to tape to avoid this slowdown.

[Dennis Handly]

>I tried write and didn't get much further, here is what I tried:

You should NOT be using C++ I/O.  You need to use the lowest level, for Linux write(2).

How are you setting up block size?  How to your write an EOF?

>If you try to do this with a script from linux,

What is your example script?

[SimonLTO]

Here is where I am at the moment, if I try to load the file into a single buffer of 49mb and then write the whole buffer to tape, I get nothing. If I write in blocks of 1024, It seems to work but goes slowly, 4 seconds for each 49mb file. I can extract the first file but after that, TAR sees the data on the tape as invalid wich I assume there is a problem with the file markers. I assumed that TAR wites the same file markers at the end of the file wether writing to disk or tape but maybe I'm wrong but looking at the file with a binary viewer, it seems like there is zero padding at the end. Why so slow? What am I doing wrong with the file markers?

Thanks,

Simon

#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <sstream>
//#include <io.h>
#include  <fcntl.h>


using namespace std;
char sourceName[]="testFile."; //file without count
char fileNameWithCount[100];
char sourcePath[]="/mnt/windows/4096x3112/"; //sourcePath
char lto[]="/dev/nst0" ; //pathe to LTO
int startCount=15108; //firstFrameToTAR
int stopCount=15150; //last frame to TAR
char leadingzeros[6];
char *zero[1024];
void makeFrameNumber(){
 strcpy(fileNameWithCount,"");
 // strcpy(filenamewithcountProxy,pathOut);
 // strcat(filenamewithcountProxy,"proxy\\");
 if (startCount<100000) strcpy(leadingzeros,"0");
 if (startCount<10000) strcpy(leadingzeros,"00");
 if (startCount<1000) strcpy(leadingzeros,"000");
 if (startCount<100) strcpy(leadingzeros,"0000");
 if (startCount<10) strcpy(leadingzeros,"00000");
 
 strcat(fileNameWithCount,sourceName);
 //strcat(filenamewithcountProxy,fileNameOut);
 strcat(fileNameWithCount,leadingzeros);
 //strcat(filenamewithcountProxy,leadingzeros);
 //strcat(fileNameWithCount,(*str)startcount);
 sprintf(leadingzeros, "%d", startCount );
 strcat(fileNameWithCount,leadingzeros);
 //strcat(filenamewithcountProxy,leadingzeros);
 // strcat(fileNameWithCount,".tif");
 strcat(fileNameWithCount,".dpx");
 //strcat(filenamewithcountProxy,".dpx");
 
 startCount++;

}

int main()
{
 

system("mt -f /dev/nst0 setblk 0");
system("mt -f /dev/nst0 rewind");

 
int oFile;
oFile=open(lto,O_WRONLY);
char changeDir[100]="cd ";
strcat(changeDir,sourcePath);
 system(changeDir);

cout<<changeDir<<"\n";
system("pwd");
FILE * pFile;
long int blocks;
 long int lSize;
 long int lSizeCounter=0;
 char * buffer;
 size_t result;
 while (startCount<=stopCount){
 char systemOut[100]= "tar -cv -b 1024 -f /mnt/ramdisk/tarfile.tar ";
 makeFrameNumber();// create souce frame number
    strcat(systemOut,fileNameWithCount);

 cout<<systemOut<<"\n";
 system(systemOut); //tar to ram drive
 
 pFile=fopen("/mnt/ramdisk/tarfile.tar","rb");// load file from ram drive
 fseek (pFile , 0 , SEEK_END);
 lSize = ftell (pFile);
cout<<lSize<<"\n";
 rewind (pFile);
 //buffer = (char*) malloc (sizeof(char)*lSize);
buffer = (char*) malloc (sizeof(char)*1024);
 if (buffer == NULL) {fputs ("Memory error",stderr); exit (2);}

 while( blocks<(lSize)){

result = fread (buffer,1,1024,pFile);
if (result != 1024) {fputs ("Reading error",stderr); exit (3);}
 write (  oFile, buffer, 1024 );

blocks+=1024;
}
blocks=0;
write(oFile, zero,1024);

cout<<"writing zero"<<"\n";
fclose (pFile);
 system("rm -r /mnt/ramdisk/tarfile.tar");
 cout<<"writing "<<fileNameWithCount<<" bytes="<<lSize<<"\n";
 

 
 
free (buffer);
 }


 
 
 close (oFile);
 
 return 0;
}

[Dennis Handly]

>single buffer of 49mb and then write the whole buffer to tape, I get nothing.

Tape blocks can't be that big.  You need to figure out what is the max size your OS and the tape can handle.

I've heard 64 KB or 128 KB.

>If I write in blocks of 1024

That is WAY too small!

My tar says the default blocking factor is (-b) 20 with a logical recordsize of 512 bytes.  So you at least need to write that much.  If you write larger, you would need to use -b N to specify the multiplier.

>TAR sees the data on the tape as invalid which I assume there is a problem with the file markers.

Or blocking factor.

>I assumed that TAR writes the same file markers at the end of the file whether writing to disk or tape

No intermediate EOF markers on disk.  A tape can have multiple EOF markers.

>it seems like there is zero padding at the end.

At least to fill up to 512 and possibly to 20 * 512.

>if (startCount<100000) strcpy(leadingzeros,"0");

There is no need for this logic.  You can use this format for zero fill: %.7d

[SimonLTO]

All very helpful, thanks! 

So in the original TAR archive, the block size is 1024 (tar -cv -b 1024 -f) so if I'm writing larger blocks to tape, I have to take this into account when I write the last block from the file and write a smaller block, no?

I was trying to write the file marker by this: write(oFile, zero,1024); based on this "Physically, an archive consists of a series of file entries terminated by an end-of-archive entry, which consists of two 512 blocks of zero bytes." This is not working,  what don't I understand about how the marker works? With or without this, I can read the first TAR written sucessfully but the second it sees as invalid.

"There is no need for this logic.  You can use this format for zero fill: %.7d", yes, much simpler...

All the best,

Simon

[Dennis Handly]

>the block size is 1024 (tar -cv -b 1024 -f)

If the blocksize is 1024 bytes then it's -b 2 (blocking factor).

>I have to take this into account when I write the last block from the file and write a smaller block?

Either that or make sure the rest of the large block is padded with zeroes.

>I was trying to write the file marker by this: write(oFile, zero,1024)

I seriously doubt this will work.  This writes data to the tape and a file marker is something special.

But you need to look at your OS documentation.

>what don't I understand about how the marker works?

You shouldn't need to have this with a tar file.  I assume a file marker is for quick access to a multiple file tape.

My only experience with round tapes was decades ago.  And more recently I used mt(1) to fiddle with EOF marks.