topic Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl in StoreOnce Backup Storage

Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offline

TheSeeker — Wed, 15 May 2024 13:41:37 GMT

The backup application shows the following log to this specific catalyst store

Discover Status ERROR = 0xA0009B23 (E_PVL_OST_CONNECT_FAILURE

however, I connected to a diffrent Catalyst store on the HP Storeonce and it connected fine

on Catalyst console the sotre showing fine on the Storeonce console however it says house keeping is pending and the time is stuck aganist this store and not progressing; I will try and remove the housekeeping block window as one of the steps

The other catalyst stores on the same server are showing fine

This seem to have occured after we aged some data

Now we cant send any data to the catalyst store; it is disabled and when we tried enabling it across diffrent servers /rebooted /tried readding to another server it shows Unable to conect to open storge device Ensure the network is properly configured.

Is there a command that I can run to check if the intigity of the catalyst store is intact?

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

yilmazaydin — Wed, 15 May 2024 05:50:24 GMT

Hi @TheSeeker .

What backup software do you use with StoreOnce Catalyst?

How media server connected to the catalyst Store - via CoFC or via IP ?

Regards, YA.

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

TheSeeker — Wed, 15 May 2024 05:53:19 GMT

BE software

connecting it to using netwrok to connect it to HP storeonce stores

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

yilmazaydin — Wed, 15 May 2024 10:16:41 GMT

You can find additional information within log file by path <installationPath>\HPE\StoreOnce\isvsupport\OST\logs on the media server.

What version of BE and OST Plugin used?

What model of the SO Device and version system software running on the SO Device?

Regard, YA.

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

TheSeeker — Thu, 16 May 2024 06:06:16 GMT

@yilmazaydin housekeeping completed however this perticular store we cant connect ; otherstores on the same servers we can connect without any issues; tested this from diffrent test servers ; no alerts on Storeonce webconsole

Is there a way to test this connectivity outside of Backup Exec may be using a powershell console; or if I add the store to a diffrent backup product(Veeam) and some some test data to it. Or can I run some commands via Storonce command console to check if everything is fine and may be trigger a copy by creating another store without having it to rely on BE client

I suspect it is something with the store although the error is misleading on BE; is there a command to check the data intigrity or database intigrity on the dedupe database locally using command cli of HPstoreonce; or some file got deleted that BE metadata file got removed;

we can connect to other catalyst stores on the same server from the BE client; nothing changed except for some data from BE was aged manually

I enabled extended logging on OST config however it does not flag anything on those logs

plugin logs; appears to query the store okay

however backup exec logs/debugging show which match with admm log for be
ERROR = 0xE000C039 (E_PVL_OST_AUTH_CONNECT_FAILURE)

114 PVLSVR 4068 16/05/2024 11:20:47 AM 8384 PvlAlerts::SendAlrt()
AlertID = {F86F172C-D219-4D99-971B-1B1D21D70BD6}
AlertMessage = hp-StoreOnceCatalyst:Server:Server-N21 - Unable to authenticate or connect to the OpenStorage device. Ensure the following:
- The network is properly configured between the device and the Backup Exec server.
- The logon account that is required to access the device has the correct credentials."

OST HPE Plugin Log (replaced servername with the actual name of the server; enabled extended loggin from the OST config file)

4068_8384 : -1 : - : Global : ost::ost::get_server_prop : Ln 1168 : OST ----- Leaving ost::ost::get_server_prop {Server=hp-StoreOnceCatalyst:ServerName Session=beost} return: OK
2024-05-15 23:17:05.852637 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1529 : OST ----- Entering ost::ost::get_lsu_prop_byname {Server=hp-StoreOnceCatalyst:ServerNAme, Lsu=ServerName-N21, Session=beost}
2024-05-15 23:17:05.854632 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1539 : OST ost::ost::get_lsu_prop_byname - LSU: [ServerName-N21] Server: [hp-StoreOnceCatalyst:ServerName]
2024-05-15 23:17:10.288488 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : ost_get_topology : Ln 239 : OST ----- Entering ost_get_topology {Server=hp-StoreOnceCatalyst:ServerName, Lsu=ServerName-N21, Session=beost}
2024-05-15 23:17:10.290487 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v4::TopologyImage>::Exists : Ln 199 : OST ----- Entering TopologyImageBase<struct v4::TopologyImage>::Exists
2024-05-15 23:17:10.293488 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 398 : OST ----- Entering stsp_image_base<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.372409 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 405 : OST V4 encoded image [ost4.configurations] does not exist
2024-05-15 23:17:12.387414 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 398 : OST ----- Leaving stsp_image_base<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.389410 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v4::TopologyImage>::Exists : Ln 199 : OST ----- Leaving TopologyImageBase<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.391411 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v3::TopologyImage>::Exists : Ln 199 : OST ----- Entering TopologyImageBase<struct v3::TopologyImage>::Exists
2024-05-15 23:17:12.393408 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 398 : OST ----- Entering stsp_image_base<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.127599 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 405 : OST V4 encoded image [ost.configurations] does not exist
2024-05-15 23:17:14.146597 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 398 : OST ----- Leaving stsp_image_base<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.149595 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v3::TopologyImage>::Exists : Ln 199 : OST ----- Leaving TopologyImageBase<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.151596 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : ost_get_topology : Ln 253 : OST ----- Leaving ost_get_topology {Server=hp-StoreOnceCatalyst:ServerName, Lsu=ServerName-N21, Session=beost} return: file not found
2024-05-15 23:17:14.154595 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1576 : OST lsu_info->lsu_def.sld_rep_targets=0, lsu_info->lsu_def.sld_rep_sources=0
2024-05-15 23:17:14.157596 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1611 : OST lsu_capacity=1551303012, lsu_capacity_phys=1925603328, lsu_used=2118320128, lsu_used_phys=-1802346852, lsu_images=3375
2024-05-15 23:17:14.161595 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1613 : OST LSU INFO

Calatyst logs

4-05-15 23:20:04.526438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 613 : QueueObjectCopyJob 0 0.000, CancelOriginObjectCopyJob 0 0.000, CancelDestinationObjectCopyJob 0 0.000
2024-05-15 23:20:04.529437 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 615 : ----------------------- ACCUMULATED COMMAND API STATISTICS --------------- --------------------
2024-05-15 23:20:04.531438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : Duration 12 209.118, Open 12 22.231, Close 12 7.157, GetServerProperties 4 9.926, SetServerProperties 0 0.000, SetClientPermissionsChecking 0 0.000
2024-05-15 23:20:04.535438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListClientPermissions 0 0.000, CreateClientPermissions 0 0.000, ModifyClientPermissions 0 0.000, ModifyClientPassword 0 0.000, DeleteClientPermissions 0 0.000
2024-05-15 23:20:04.538438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectStoreClientPermissions 0 0.000, AddObjectStoreClientPermission 0 0.000, RemoveObjectStoreClientPermission 0 0.000
2024-05-15 23:20:04.541438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectStores 9 18.726, CreateObjectStore 0 0.000, ModifyObjectStore 0 0.000, DeleteObjectStore 0 0.000, TeamedExpandObjectStore 0 0.000, TeamedContractObjectStore 0 0.000
2024-05-15 23:20:04.544436 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjects 6 149.939, CreateObject 0 0.000, ModifyObject 0 0.000, DeleteObject 0 0.000, TeamedExpandObject 0 0.000, TeamedContractObject 0 0.000
2024-05-15 23:20:04.546437 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectDataJobs 0 0.000, ListObjectDataJobsCount 0 0.000, ListOriginObjectCopyJobs 0 0.000, ListOriginObjectCopyJobsCount 0 0.000, ListDestinationObjectCopyJobs 0 0.000, ListDestinationObjectCopyJobsCount 0 0.000
2024-05-15 23:20:04.551436 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : QueueObjectCopyJob 0 0.000, CancelOriginObjectCopyJob 0 0.000, CancelDestinationObjectCopyJob 0 0.000
2024-05-15 23:20:04.554437 (local 11:20) : INFO : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 630 : Closed command session (ID : 12) successfully.
2024-05-15 23:20:04.557436 (local 11:20) : INFO : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 632 : --------------- Exiting osCltCmd_CloseCommandSession, Return Status : 0 (OSCLT_ERR_SUCCESS) ---------------
2024-05-15 23:20:04.559437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::Close : Ln 213 : ------------------------------ Leaving oscpp::CommandSession::Close -----------------------------
2024-05-15 23:20:04.562444 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::~CommandSession : Ln 90 : ------------------------------ Entering oscpp::CommandSession::~CommandSession -----------------------------
2024-05-15 23:20:04.566437 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::~CommandSession : Ln 90 : ------------------------------ Leaving oscpp::CommandSession::~CommandSession -----------------------------
2024-05-15 23:20:04.569439 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 78 : ------------------------------ Entering oscpp::CommandSession::CommandSession -----------------------------
2024-05-15 23:20:04.571445 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : [IN] sessionInfo =
2024-05-15 23:20:04.575439 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Server Address = ServerName, Copy Address = , Command Port = 9387, Data Port = 9388
2024-05-15 23:20:04.579435 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Credentials : Vendor = VeritasBackupExec, Identifier = BackupExecOST, Password = (Set)
2024-05-15 23:20:04.582436 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Log Descriptor : ID = OST(MT), Level = 0, File = C:\Program Files\HPE\StoreOnce\isvsupport\OST\logs\catalyst.log, Max Size = 10485760
2024-05-15 23:20:04.585437 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Max Retry Count: Command = 12, Data = 120
2024-05-15 23:20:04.593435 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Session Options: (only non-zero options logged)
2024-05-15 23:20:04.596433 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 78 : ------------------------------ Leaving oscpp::CommandSession::CommandSession -----------------------------
2024-05-15 23:20:04.598433 (local 11:20) : DEBUG : 4068_8384 : 12 : +CO : OST(MT) : oscpp::Object::Refresh : Ln 212 : ------------------------------ Entering oscpp::Object::Refresh -----------------------------
2024-05-15 23:20:04.601437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::GetCachedServerCapabilities : Ln 290 : ------------------------------ Entering oscpp::CommandSession::GetCachedServerCapabilities -----------------------------
2024-05-15 23:20:04.604437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::GetCachedServerCapabilities : Ln 290 : ------------------------------ Leaving oscpp::CommandSession::GetCachedServerCapabilities -----------------------------
2024-05-15 23:20:04.606435 (local 11:20) : INFO : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::Open : Ln 110 : ------------------------------ Entering oscpp::CommandSession::Open -----------------------------
2024-05-15 23:20:04.609450 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : osCltCmd_OpenCommandSessionInternal : Ln 180 : --------------- Entering osCltCmd_OpenCommandSessionInternal[Non-Teamed]. Session ID : -1 ---------------
2024-05-15 23:20:04.612436 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogLogContextType : Ln 177 : rDbgPrv_LogLogContextType invoked from osCltCmd_OpenCommandSessionInternal Ln 212
2024-05-15 23:20:04.615439 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogLogContextType : Ln 203 : Log Session ID : OST(MT), Log File : C:\Program Files\HPE\StoreOnce\isvsupport\OST\logs\catalyst.log, Level : Cmd 0 Data 0 Pln 0 Chk 0 Stat 0 Bid 0 TmC 0 TmD 0 TxC 1 TxD 1, CMapC 0, CoFC 0, WIP 0, SP 0, CBT 0, Max log file size : 10485760.
2024-05-15 23:20:04.618443 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : osCltCmd_OpenCommandSessionInternal : Ln 213 : pSessionOptions :
2024-05-15 23:20:04.620443 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogSessionOptionsType : Ln 266 : rDbgPrv_LogSessionOptionsType invoked from osCltCmd_OpenCommandSessionInternal Ln 214

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

yilmazaydin — Thu, 16 May 2024 15:06:54 GMT

@TheSeeker , you can use StoreOnce Catalyst Copy tool for reading and writing to the Catalyst Store - https://myenterpriselicense.hpe.com/cwp-ui/product-download-info/StoreOnceCATTOOLS/1.2.0/sw_free?&.

If you can open support case - HPE support provide StoreOnceCatalystDiagnostic Tool for basic operations with Catalyst Store.

Don't forget that the Catalyst is controlled by ISV - BE, NBU or Veeam. And all changes made without their participation will lead to data desynchronization.

The device itself has internal data integrity verification mechanisms that cannot be started on the administrator's command line.

Regards, YA.

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

TheSeeker — Thu, 16 May 2024 18:18:58 GMT

thanks for that; do you have a link for this catalyst store copy utility

unfortunetly the storeonce devices are old and out of support otherwise I would have logged a job

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

TheSeeker — Thu, 16 May 2024 18:26:38 GMT

sorry my bad; I will try this cat tool to copy data to another store and try connecing to that store to do an inventory and catalog

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

TheSeeker — Mon, 20 May 2024 08:31:38 GMT

@yilmazaydin I tried it however it appear to be not able to copy the data

I created a new store with the correct permission; updated the config file however on the new store

when I run the command it can identify the source and destination server and the stores okay

however when I look at the hpstoreonce event it says

[ssid1] : Catalyst outbound copy job from item BEOST_00000255 in store cancelled by user

I am using plugn files; it does copy data on the target sotre which shows the file name however the data size is 0 bytes

Re: Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

yilmazaydin — Mon, 20 May 2024 19:58:56 GMT

Hi @TheSeeker .

Unfortunately, the device is not supported and the task is very specific, which takes this question beyond the scope of this forum.

You can write me directly, maybe I can help.

Regards, YA.