IMC FTP transfer issues

[rpfister]

Hi All,

We've got an issue with the HP IMC v7.1 Autobackup via FTP. We had this issue before with an older version of the software, an update did not fix it. We are backing up the switch configuration using HP IMC's Auto Backup, which runs on a weekly schedule. Since a while when checking the Auto Backup "log" we see that some devices fail to backup correctly (Due to failed server connection or FTP issues according to the log).

I did some research and investigated the issue and found out that the configuration acuatlly get's backed up using Filezilla on the HP IMC Server and get's stored in location 1 on this server (correctly), but HP IMC fails to copy these configuration files to location 2, which is in the HP IMC folders on this server. This is the reason why we see the error in the HP IMC Auto Backup Log. These configuration files work fine, so there's no issue with that. It just seems that HP IMC fails to copy some config files(not from a specific switch model or anything) from location 1 to location 2, for most it works fine.

Has anyone an idea how to fix this?

Thank you, remop102

[LindsayHill]

Are you able to post a sanitised version of the imccfgbakdm log file?

Also, are these devices using SNMP read/write? 

If devices support SNMP read/write (and IMC has an SNMP device adapter for that device), then it uses an snmpset to get the device to upload its config via FTP to the IMC server.

If it doesn't support SNMP (either no r/w, or no SNMP adapter), then it will connect via Telnet/SSH, and enter the required CLI commands to tell the device to back up via FTP.

IMC will then watch the output of that session, and when it thinks the backup has finished, it will exit. IMC then copies the backup file from the tmp directory (aka the ftp root). The key bit is "when it thinks the backup has finished" - it guesses this by watching the session output, and looks for something like "File transfer successful."

The problem is that the message they look for doesn't always match what your FTP server sends. So sometimes the backup file has been FTP'd, but IMC thinks it never completed properly. Since IMC thinks it didn't finish, it just leaves the backup file in the tmp directory.

I'm not sure if that is exactly what's going on, but that's what I'd be starting with. The imcfgbakdm log file should be able to tell you more.

[rpfister]

Hi Lindsay,

Thank you for your reply.

I can see that it works for most devices.

2015-05-03 20:00:30.224 [INFO (0)] [THREAD(4800)] [CFileTransferTask::transferFileEx] Do file transfer, device ip: XXX.XXX.XXX.XXX, transfer protocol: TRANSFER_PROTOCOL_CONFIG_MIB_XXX, result code:  0
2015-05-03 20:00:31.224 [INFO (0)] [THREAD(4884)] Process backup transfer task completed, result:

.....

But others receive this message:

2015-05-03 20:00:31.739 [INFO (0)] [THREAD(4828)] [CFileTransferTask::transferFileEx] Do file transfer, device ip: XXX.XXX.XXX.XXX, transfer protocol: TRANSFER_PROTOCOL_CONFIG_MIB_XXX, result code:  0
2015-05-03 20:00:31.755 [ERROR (18)] [THREAD(4884)] [CComponentAdapter::isDevSupportServiceAction()] fail to call getDevAdapterName().DevID=XXX,ServiceName = ConfigBackup
2015-05-03 20:00:31.755 [INFO (0)] [THREAD(4884)] [CComponentAdapter::isDevSupportServiceAction] dev_id: XXX, adapter_name: XXX
2015-05-03 20:00:31.802 [WARNING (0)] [THREAD(4884)] [CQvDBReaderADP::~CQvDBReaderADP] Cancel current SQL when data have not be fetched out.

Or:

2015-05-03 20:00:31.833 [INFO (0)] [THREAD(4824)] [CFileTransferTask::transferFileEx] Do file transfer, device ip: XXX.XXX.XXX.XXX, transfer protocol: TRANSFER_PROTOCOL_CONFIG_MIB_XXX, result code:  0
2015-05-03 20:00:31.833 [WARNING (0)] [THREAD(4884)] [CQvDBReaderADP::~CQvDBReaderADP] Cancel current SQL when data have not be fetched out.
2015-05-03 20:00:32.020 [INFO (0)] [THREAD(4800)] [C3ComSysManTableImp::isXrnDevice()] is statck dev,fabric type = 1.

Thank you very much,

rpfister

[LindsayHill]

Is that all it has in those logs? I would have expected to see more info in there about what's failing.

[rpfister]

Hi Lindsay,

Sorry I somehow did not see this the first time I had a look through the file.

This looks like the issue:

2015-05-03 20:00:39.583 [ERROR (4)] [THREAD(4892)] [imcdm::generateFileKey()] Can't open input file: %install%/server/tmp\running_1421858352_1.cfg
2015-05-03 20:00:39.583 [ERROR (4)] [THREAD(4892)] [CCfgFileRestoreTask::processXrnDeviceBackupCompleted()] Failed to caculate file digest: running_1421858352_1.cfg
2015-05-03 20:00:39.583 [ERROR (-1)] [THREAD(4892)] [CCfgFileDataMgr::copyFile] Open source file ( call ACE::fopen() ) failed in 'CCfgFileDataMgr::copyFile'
2015-05-03 20:00:39.583 [INFO (6006)] [THREAD(4892)] [CCfgFileRestoreTask::processDeviceBackupCompleted()] Copy file: running_1421858352_1.cfg from tftp dir to config backup dir failed.
2015-05-03 20:00:39.614 [WARNING (0)] [THREAD(4892)] [CQvDBReaderADP::~CQvDBReaderADP] Cancel current SQL when data have not be fetched out.
2015-05-03 20:00:39.645 [ERROR (4)] [THREAD(4892)] [imcdm::generateFileKey()] Can't open input file: %install%/server/tmp\startup_1421858433_1.cfg
2015-05-03 20:00:39.645 [ERROR (4)] [THREAD(4892)] [CCfgFileRestoreTask::processXrnDeviceBackupCompleted()] Failed to caculate file digest: startup_1421858433_1.cfg
2015-05-03 20:00:39.645 [ERROR (-1)] [THREAD(4892)] [CCfgFileDataMgr::copyFile] Open source file ( call ACE::fopen() ) failed in 'CCfgFileDataMgr::copyFile'
2015-05-03 20:00:39.645 [INFO (6006)] [THREAD(4892)] [CCfgFileRestoreTask::processDeviceBackupCompleted()] Copy file: startup_1421858433_1.cfg from tftp dir to config backup dir failed.
2015-05-03 20:00:39.692 [WARNING (0)] [THREAD(4892)] [CQvDBReaderADP::~CQvDBReaderADP] Cancel current SQL when data have not be fetched out.

Thank you very much,

rpfister

[LindsayHill]

Sometimes you'll see logs a bit out of order - some stuff gets written to other temp logs, then later inserted in this log file. Can make a it a little confusing following it all. 

That's still not really telling us what's going on though. Do those files actually exist in the tmp directory? If they do, something odd is going on. If they don't exist at all, then the upload failed for some reason. 

As an aside, I find SCP/SFTP more reliable (plus more secure). If your devices support it, it might be easier to switch to that, rather than fixing FTP.

[rpfister]

Hi Lindsay,

Oh, right. I did not know that.

The strange thing is that the files actually are in this folder, but somehow cannot be copied over to the imc backup folder.

So the transfer from the device to the ftp is not an issue it's the moving/copying over to the other location on the same server. For 90% of the switches it works and for around 10% not. It has nothing to do with the model or firmware version as these are different models/versions.

The newer devices support SCP/SFTP but some older models will have problems with that.

Thank you again for your help.

Best Regards,

rpfister

[LindsayHill]

Very odd.

Is it always the same devices that fail? If it is, it might be worth changing them to SCP/SFTP if possible. At least it narrows down the list to investigate further.

Only other thing I'd be wondering about would be disk space - e.g. if the filesystem is full or nearly full you might get some odd behaviour when files are being moved around.

[rpfister]

Hi Lindsay,

Sorry for letting you wait for such a long time. I was on holiday.

Yes, it's always the same devices. But it gets more with time. And the strange thing is that it did not start for all devices at the same time.

These devices also backed up fine first but somewhen (no specifig time or date) stopped to do so.

It's really odd.

Regarding the disk space.

This could actually be an issue. I'll check that too.

:-)

Thank you very much and have a nice day,

rpfister