- Community Home
- >
- Storage
- >
- Legacy
- >
- Automated Backup
- >
- Sample code of writing DATA to LTO tape
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2014 01:23 PM
03-12-2014 01:23 PM
Sample code of writing DATA to LTO tape
I am looking for sample c++ source code of writing binary data to LTO tape(4,5,6).. Is there anything out there? I have looked genrally on the web and nothing. Itr seems like fwrite should work but not when I try it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2014 11:36 PM
03-12-2014 11:36 PM
Re: Sample code of writing DATA to LTO tape
If you aren't writing backup software, you shouldn't be writing directly to a tape. Write to a file and use backup software to write to the tape.
>It seems like fwrite should work
Are you on Linux/Unix? If so, you should be using write(2) and not libc routines. And write huge blocks and make sure you have lots of memory so can keep the tape streaming.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2014 09:11 AM - edited 03-25-2014 02:53 PM
03-25-2014 09:11 AM - edited 03-25-2014 02:53 PM
Re: Sample code of writing DATA to LTO tape
I'm on windows but also have tried linux.
I tried write and didn't get much further, here is what I tried:
#include "stdafx.h"
#include <stdio.h>
#include <stdlib.h>
#include <ctime>
#include <iostream>
#include <sstream>
#include <cstring>
#include <fstream>
using namespace std;
char sourceName[]="YCTIWY_7A_DN."; //file without count
char fileNameWithCount[100];
char sourcePath[]="V:\\Simon_Projects4\\LTO_test\\"; //sourcePath
char lto[]="tape0" ; //pathe to LTO
int startCount=0; //firstFrameToTAR
int stopCount=4; //last frame to TAR
long lSize;//file size
char leadingzeros[6];
char shortpathe[50]="C:\\Users\\Cineric\\Documents\\";
char shortpathetemp[50];
void makeFrameNumber(){
strcpy(fileNameWithCount,sourcePath);
strcpy(shortpathetemp,shortpathe);
if (startCount<100000) strcpy(leadingzeros,"0");
if (startCount<10000) strcpy(leadingzeros,"00");
if (startCount<1000) strcpy(leadingzeros,"000");
if (startCount<100) strcpy(leadingzeros,"0000");
if (startCount<10) strcpy(leadingzeros,"00000");
strcat(fileNameWithCount,sourceName);
strcat(shortpathetemp,sourceName);
strcat(fileNameWithCount,leadingzeros);
strcat(shortpathetemp,leadingzeros);
sprintf(leadingzeros, "%d", startCount );
strcat(fileNameWithCount,leadingzeros);
strcat(shortpathetemp,leadingzeros);
strcat(fileNameWithCount,".tar");
strcat(shortpathetemp,".tar");
startCount++;
}
int main()
{
std::ofstream outfile (lto,std::ofstream::binary);
while (startCount<=stopCount){
makeFrameNumber();// create souce frame number
std::cout<<shortpathetemp<<"\n";
std::ifstream infile (fileNameWithCount,std::ifstream::binary);
// get size of file
infile.seekg (0,infile.end);
long size = infile.tellg();
infile.seekg (0);
// allocate memory for file content
char* buffer = new char[size];
// read content of infile
infile.read (buffer,size);
if ( (infile.rdstate() & std::ifstream::failbit) != 0 )
{std::cerr << "Error opening \n";}
// write to outfile
outfile.write (buffer,size);
if ( (outfile.fail() ) != 0 )
{std::cerr << "Error writing \n";}
delete[] buffer;
infile.close();
}
outfile.close();
string empname="";
cout << "Hit enter to exit:" <<endl;
getline(cin, empname);
return 0;
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2014 09:16 AM
03-25-2014 09:16 AM
Re: Sample code of writing DATA to LTO tape
Just to add, the reason I'm trying to do this is our clients requires that each indiviual file (49mb each) is it's own tar file on the tape. If you try to do this with a script from linux, it takes 6 seconds a file for each tar operation, no matter the file size and what should take hours ends up taking days. I'm hoping to add the tar headers and file markers and stream the data to tape to avoid this slowdown.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2014 09:57 PM
03-25-2014 09:57 PM
Re: Sample code of writing DATA to LTO tape
>I tried write and didn't get much further, here is what I tried:
You should NOT be using C++ I/O. You need to use the lowest level, for Linux write(2).
How are you setting up block size? How to your write an EOF?
>If you try to do this with a script from linux,
What is your example script?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-02-2014 08:52 AM
04-02-2014 08:52 AM
Re: Sample code of writing DATA to LTO tape
Here is where I am at the moment, if I try to load the file into a single buffer of 49mb and then write the whole buffer to tape, I get nothing. If I write in blocks of 1024, It seems to work but goes slowly, 4 seconds for each 49mb file. I can extract the first file but after that, TAR sees the data on the tape as invalid wich I assume there is a problem with the file markers. I assumed that TAR wites the same file markers at the end of the file wether writing to disk or tape but maybe I'm wrong but looking at the file with a binary viewer, it seems like there is zero padding at the end. Why so slow? What am I doing wrong with the file markers?
Thanks,
Simon
#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <sstream>
//#include <io.h>
#include <fcntl.h>
using namespace std;
char sourceName[]="testFile."; //file without count
char fileNameWithCount[100];
char sourcePath[]="/mnt/windows/4096x3112/"; //sourcePath
char lto[]="/dev/nst0" ; //pathe to LTO
int startCount=15108; //firstFrameToTAR
int stopCount=15150; //last frame to TAR
char leadingzeros[6];
char *zero[1024];
void makeFrameNumber(){
strcpy(fileNameWithCount,"");
// strcpy(filenamewithcountProxy,pathOut);
// strcat(filenamewithcountProxy,"proxy\\");
if (startCount<100000) strcpy(leadingzeros,"0");
if (startCount<10000) strcpy(leadingzeros,"00");
if (startCount<1000) strcpy(leadingzeros,"000");
if (startCount<100) strcpy(leadingzeros,"0000");
if (startCount<10) strcpy(leadingzeros,"00000");
strcat(fileNameWithCount,sourceName);
//strcat(filenamewithcountProxy,fileNameOut);
strcat(fileNameWithCount,leadingzeros);
//strcat(filenamewithcountProxy,leadingzeros);
//strcat(fileNameWithCount,(*str)startcount);
sprintf(leadingzeros, "%d", startCount );
strcat(fileNameWithCount,leadingzeros);
//strcat(filenamewithcountProxy,leadingzeros);
// strcat(fileNameWithCount,".tif");
strcat(fileNameWithCount,".dpx");
//strcat(filenamewithcountProxy,".dpx");
startCount++;
}
int main()
{
system("mt -f /dev/nst0 setblk 0");
system("mt -f /dev/nst0 rewind");
int oFile;
oFile=open(lto,O_WRONLY);
char changeDir[100]="cd ";
strcat(changeDir,sourcePath);
system(changeDir);
cout<<changeDir<<"\n";
system("pwd");
FILE * pFile;
long int blocks;
long int lSize;
long int lSizeCounter=0;
char * buffer;
size_t result;
while (startCount<=stopCount){
char systemOut[100]= "tar -cv -b 1024 -f /mnt/ramdisk/tarfile.tar ";
makeFrameNumber();// create souce frame number
strcat(systemOut,fileNameWithCount);
cout<<systemOut<<"\n";
system(systemOut); //tar to ram drive
pFile=fopen("/mnt/ramdisk/tarfile.tar","rb");// load file from ram drive
fseek (pFile , 0 , SEEK_END);
lSize = ftell (pFile);
cout<<lSize<<"\n";
rewind (pFile);
//buffer = (char*) malloc (sizeof(char)*lSize);
buffer = (char*) malloc (sizeof(char)*1024);
if (buffer == NULL) {fputs ("Memory error",stderr); exit (2);}
while( blocks<(lSize)){
result = fread (buffer,1,1024,pFile);
if (result != 1024) {fputs ("Reading error",stderr); exit (3);}
write ( oFile, buffer, 1024 );
blocks+=1024;
}
blocks=0;
write(oFile, zero,1024);
cout<<"writing zero"<<"\n";
fclose (pFile);
system("rm -r /mnt/ramdisk/tarfile.tar");
cout<<"writing "<<fileNameWithCount<<" bytes="<<lSize<<"\n";
free (buffer);
}
close (oFile);
return 0;
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2014 11:26 PM - edited 04-03-2014 11:30 PM
04-03-2014 11:26 PM - edited 04-03-2014 11:30 PM
Re: Sample code of writing DATA to LTO tape
>single buffer of 49mb and then write the whole buffer to tape, I get nothing.
Tape blocks can't be that big. You need to figure out what is the max size your OS and the tape can handle.
I've heard 64 KB or 128 KB.
>If I write in blocks of 1024
That is WAY too small!
My tar says the default blocking factor is (-b) 20 with a logical recordsize of 512 bytes. So you at least need to write that much. If you write larger, you would need to use -b N to specify the multiplier.
>TAR sees the data on the tape as invalid which I assume there is a problem with the file markers.
Or blocking factor.
>I assumed that TAR writes the same file markers at the end of the file whether writing to disk or tape
No intermediate EOF markers on disk. A tape can have multiple EOF markers.
>it seems like there is zero padding at the end.
At least to fill up to 512 and possibly to 20 * 512.
>if (startCount<100000) strcpy(leadingzeros,"0");
There is no need for this logic. You can use this format for zero fill: %.7d
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2014 08:30 AM
04-10-2014 08:30 AM
Re: Sample code of writing DATA to LTO tape
All very helpful, thanks!
So in the original TAR archive, the block size is 1024 (tar -cv -b 1024 -f) so if I'm writing larger blocks to tape, I have to take this into account when I write the last block from the file and write a smaller block, no?
I was trying to write the file marker by this: write(oFile, zero,1024); based on this "Physically, an archive consists of a series of file entries terminated by an end-of-archive entry, which consists of two 512 blocks of zero bytes." This is not working, what don't I understand about how the marker works? With or without this, I can read the first TAR written sucessfully but the second it sees as invalid.
"There is no need for this logic. You can use this format for zero fill: %.7d", yes, much simpler...
All the best,
Simon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-11-2014 01:09 PM - edited 04-11-2014 01:09 PM
04-11-2014 01:09 PM - edited 04-11-2014 01:09 PM
Re: Sample code of writing DATA to LTO tape
>the block size is 1024 (tar -cv -b 1024 -f)
If the blocksize is 1024 bytes then it's -b 2 (blocking factor).
>I have to take this into account when I write the last block from the file and write a smaller block?
Either that or make sure the rest of the large block is padded with zeroes.
>I was trying to write the file marker by this: write(oFile, zero,1024)
I seriously doubt this will work. This writes data to the tape and a file marker is something special.
But you need to look at your OS documentation.
>what don't I understand about how the marker works?
You shouldn't need to have this with a tar file. I assume a file marker is for quick access to a multiple file tape.
My only experience with round tapes was decades ago. And more recently I used mt(1) to fiddle with EOF marks.