Housekeeping Service not progressing : Backup app cant connect to Catalyst Store; offl

[TheSeeker]

The backup application shows the following log to this specific catalyst store

Discover Status ERROR = 0xA0009B23 (E_PVL_OST_CONNECT_FAILURE

however, I connected to a diffrent Catalyst store on the HP Storeonce  and it connected fine 

on Catalyst console the sotre showing fine on the Storeonce console however it says house keeping is pending and the time is stuck aganist this store and not progressing; I will try and remove the housekeeping block window as one of the steps 

The other catalyst stores on the same server are showing fine 

This seem to have occured after we aged some data 

Now we cant send any data to the catalyst store;  it is disabled and when we tried enabling it across diffrent servers /rebooted /tried readding to another server it shows Unable to conect to open storge device Ensure the network is properly configured.

Is there a command that I can run to check if the intigity of the catalyst store is intact? 

[yilmazaydin]

Hi @TheSeeker .

What backup software do you use with StoreOnce Catalyst?

How media server connected to the catalyst Store - via CoFC or via IP ?

Regards, YA.

[TheSeeker]

BE software

connecting it to using netwrok to connect it to HP storeonce stores 

[yilmazaydin]

You can find additional information within  log file by path <installationPath>\HPE\StoreOnce\isvsupport\OST\logs on the media server.

What version of BE and OST Plugin used?

What model of the SO Device and version system software running on the SO Device?

Regard, YA.

[TheSeeker]

@yilmazaydin housekeeping completed however this perticular store we cant connect ; otherstores on the same servers we can connect without any issues; tested this from diffrent test servers ; no alerts on Storeonce webconsole 

Is there a way to test this connectivity outside of Backup Exec may be using a powershell console; or if I add the store to a diffrent backup product(Veeam) and some some test data to it.   Or can I run some commands via Storonce command console to check if everything is fine and may be trigger a copy by creating another store without having it to rely on BE client 

I suspect it is something with the store although the error is misleading on BE; is there a command to check the data intigrity or database intigrity on the dedupe database locally using command cli of HPstoreonce; or some file got deleted that BE metadata file got removed; 

we can connect to other catalyst stores on the same server from the BE client; nothing changed except for some data from BE was aged manually 

I enabled extended logging on OST config however it does not flag anything on those logs

plugin logs; appears to query the store okay 

however backup exec logs/debugging show which match with admm log for be 
ERROR = 0xE000C039 (E_PVL_OST_AUTH_CONNECT_FAILURE)

114 PVLSVR 4068 16/05/2024 11:20:47 AM 8384 PvlAlerts::SendAlrt()
AlertID = {F86F172C-D219-4D99-971B-1B1D21D70BD6}
AlertMessage = hp-StoreOnceCatalyst:Server:Server-N21 - Unable to authenticate or connect to the OpenStorage device. Ensure the following:
- The network is properly configured between the device and the Backup Exec server.
- The logon account that is required to access the device has the correct credentials."

OST HPE Plugin Log (replaced servername with the actual name of the server; enabled extended loggin from the OST config file)

4068_8384 : -1 : - : Global : ost::ost::get_server_prop : Ln 1168 : OST ----- Leaving ost::ost::get_server_prop {Server=hp-StoreOnceCatalyst:ServerName Session=beost} return: OK
2024-05-15 23:17:05.852637 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1529 : OST ----- Entering ost::ost::get_lsu_prop_byname {Server=hp-StoreOnceCatalyst:ServerNAme, Lsu=ServerName-N21, Session=beost}
2024-05-15 23:17:05.854632 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1539 : OST ost::ost::get_lsu_prop_byname - LSU: [ServerName-N21] Server: [hp-StoreOnceCatalyst:ServerName]
2024-05-15 23:17:10.288488 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : ost_get_topology : Ln 239 : OST ----- Entering ost_get_topology {Server=hp-StoreOnceCatalyst:ServerName, Lsu=ServerName-N21, Session=beost}
2024-05-15 23:17:10.290487 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v4::TopologyImage>::Exists : Ln 199 : OST ----- Entering TopologyImageBase<struct v4::TopologyImage>::Exists
2024-05-15 23:17:10.293488 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 398 : OST ----- Entering stsp_image_base<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.372409 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 405 : OST V4 encoded image [ost4.configurations] does not exist
2024-05-15 23:17:12.387414 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v4::TopologyImage>::Exists : Ln 398 : OST ----- Leaving stsp_image_base<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.389410 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v4::TopologyImage>::Exists : Ln 199 : OST ----- Leaving TopologyImageBase<struct v4::TopologyImage>::Exists
2024-05-15 23:17:12.391411 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v3::TopologyImage>::Exists : Ln 199 : OST ----- Entering TopologyImageBase<struct v3::TopologyImage>::Exists
2024-05-15 23:17:12.393408 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 398 : OST ----- Entering stsp_image_base<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.127599 (local 11:17) : INFO : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 405 : OST V4 encoded image [ost.configurations] does not exist
2024-05-15 23:17:14.146597 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : stsp_image_base<struct v3::TopologyImage>::Exists : Ln 398 : OST ----- Leaving stsp_image_base<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.149595 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : TopologyImageBase<struct v3::TopologyImage>::Exists : Ln 199 : OST ----- Leaving TopologyImageBase<struct v3::TopologyImage>::Exists
2024-05-15 23:17:14.151596 (local 11:17) : TRACE : 4068_8384 : -1 : - : Global : ost_get_topology : Ln 253 : OST ----- Leaving ost_get_topology {Server=hp-StoreOnceCatalyst:ServerName, Lsu=ServerName-N21, Session=beost} return: file not found
2024-05-15 23:17:14.154595 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1576 : OST lsu_info->lsu_def.sld_rep_targets=0, lsu_info->lsu_def.sld_rep_sources=0
2024-05-15 23:17:14.157596 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1611 : OST lsu_capacity=1551303012, lsu_capacity_phys=1925603328, lsu_used=2118320128, lsu_used_phys=-1802346852, lsu_images=3375
2024-05-15 23:17:14.161595 (local 11:17) : DEBUG : 4068_8384 : -1 : - : Global : ost::ost::get_lsu_prop_byname : Ln 1613 : OST LSU INFO

Calatyst logs

4-05-15 23:20:04.526438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 613 : QueueObjectCopyJob 0 0.000, CancelOriginObjectCopyJob 0 0.000, CancelDestinationObjectCopyJob 0 0.000
2024-05-15 23:20:04.529437 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 615 : ----------------------- ACCUMULATED COMMAND API STATISTICS --------------- --------------------
2024-05-15 23:20:04.531438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : Duration 12 209.118, Open 12 22.231, Close 12 7.157, GetServerProperties 4 9.926, SetServerProperties 0 0.000, SetClientPermissionsChecking 0 0.000
2024-05-15 23:20:04.535438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListClientPermissions 0 0.000, CreateClientPermissions 0 0.000, ModifyClientPermissions 0 0.000, ModifyClientPassword 0 0.000, DeleteClientPermissions 0 0.000
2024-05-15 23:20:04.538438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectStoreClientPermissions 0 0.000, AddObjectStoreClientPermission 0 0.000, RemoveObjectStoreClientPermission 0 0.000
2024-05-15 23:20:04.541438 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectStores 9 18.726, CreateObjectStore 0 0.000, ModifyObjectStore 0 0.000, DeleteObjectStore 0 0.000, TeamedExpandObjectStore 0 0.000, TeamedContractObjectStore 0 0.000
2024-05-15 23:20:04.544436 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjects 6 149.939, CreateObject 0 0.000, ModifyObject 0 0.000, DeleteObject 0 0.000, TeamedExpandObject 0 0.000, TeamedContractObject 0 0.000
2024-05-15 23:20:04.546437 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : ListObjectDataJobs 0 0.000, ListObjectDataJobsCount 0 0.000, ListOriginObjectCopyJobs 0 0.000, ListOriginObjectCopyJobsCount 0 0.000, ListDestinationObjectCopyJobs 0 0.000, ListDestinationObjectCopyJobsCount 0 0.000
2024-05-15 23:20:04.551436 (local 11:20) : QUIET : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 620 : QueueObjectCopyJob 0 0.000, CancelOriginObjectCopyJob 0 0.000, CancelDestinationObjectCopyJob 0 0.000
2024-05-15 23:20:04.554437 (local 11:20) : INFO : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 630 : Closed command session (ID : 12) successfully.
2024-05-15 23:20:04.557436 (local 11:20) : INFO : 4068_8384 : 12 : C : OST(MT) : osCltCmd_CloseCommandSession : Ln 632 : --------------- Exiting osCltCmd_CloseCommandSession, Return Status : 0 (OSCLT_ERR_SUCCESS) ---------------
2024-05-15 23:20:04.559437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::Close : Ln 213 : ------------------------------ Leaving oscpp::CommandSession::Close -----------------------------
2024-05-15 23:20:04.562444 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::~CommandSession : Ln 90 : ------------------------------ Entering oscpp::CommandSession::~CommandSession -----------------------------
2024-05-15 23:20:04.566437 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::~CommandSession : Ln 90 : ------------------------------ Leaving oscpp::CommandSession::~CommandSession -----------------------------
2024-05-15 23:20:04.569439 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 78 : ------------------------------ Entering oscpp::CommandSession::CommandSession -----------------------------
2024-05-15 23:20:04.571445 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : [IN] sessionInfo =
2024-05-15 23:20:04.575439 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Server Address = ServerName, Copy Address = , Command Port = 9387, Data Port = 9388
2024-05-15 23:20:04.579435 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Credentials : Vendor = VeritasBackupExec, Identifier = BackupExecOST, Password = (Set)
2024-05-15 23:20:04.582436 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Log Descriptor : ID = OST(MT), Level = 0, File = C:\Program Files\HPE\StoreOnce\isvsupport\OST\logs\catalyst.log, Max Size = 10485760
2024-05-15 23:20:04.585437 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Max Retry Count: Command = 12, Data = 120
2024-05-15 23:20:04.593435 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 79 : Session Options: (only non-zero options logged)
2024-05-15 23:20:04.596433 (local 11:20) : DEBUG : 4068_8384 : 0 : +C : OST(MT) : oscpp::CommandSession::CommandSession : Ln 78 : ------------------------------ Leaving oscpp::CommandSession::CommandSession -----------------------------
2024-05-15 23:20:04.598433 (local 11:20) : DEBUG : 4068_8384 : 12 : +CO : OST(MT) : oscpp::Object::Refresh : Ln 212 : ------------------------------ Entering oscpp::Object::Refresh -----------------------------
2024-05-15 23:20:04.601437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::GetCachedServerCapabilities : Ln 290 : ------------------------------ Entering oscpp::CommandSession::GetCachedServerCapabilities -----------------------------
2024-05-15 23:20:04.604437 (local 11:20) : DEBUG : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::GetCachedServerCapabilities : Ln 290 : ------------------------------ Leaving oscpp::CommandSession::GetCachedServerCapabilities -----------------------------
2024-05-15 23:20:04.606435 (local 11:20) : INFO : 4068_8384 : 12 : +C : OST(MT) : oscpp::CommandSession::Open : Ln 110 : ------------------------------ Entering oscpp::CommandSession::Open -----------------------------
2024-05-15 23:20:04.609450 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : osCltCmd_OpenCommandSessionInternal : Ln 180 : --------------- Entering osCltCmd_OpenCommandSessionInternal[Non-Teamed]. Session ID : -1 ---------------
2024-05-15 23:20:04.612436 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogLogContextType : Ln 177 : rDbgPrv_LogLogContextType invoked from osCltCmd_OpenCommandSessionInternal Ln 212
2024-05-15 23:20:04.615439 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogLogContextType : Ln 203 : Log Session ID : OST(MT), Log File : C:\Program Files\HPE\StoreOnce\isvsupport\OST\logs\catalyst.log, Level : Cmd 0 Data 0 Pln 0 Chk 0 Stat 0 Bid 0 TmC 0 TmD 0 TxC 1 TxD 1, CMapC 0, CoFC 0, WIP 0, SP 0, CBT 0, Max log file size : 10485760.
2024-05-15 23:20:04.618443 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : osCltCmd_OpenCommandSessionInternal : Ln 213 : pSessionOptions :
2024-05-15 23:20:04.620443 (local 11:20) : INFO : 4068_8384 : 13 : C : OST(MT) : rDbgPrv_LogSessionOptionsType : Ln 266 : rDbgPrv_LogSessionOptionsType invoked from osCltCmd_OpenCommandSessionInternal Ln 214

[yilmazaydin]

@TheSeeker , you can use StoreOnce Catalyst Copy tool for reading and writing to the Catalyst Store - https://myenterpriselicense.hpe.com/cwp-ui/product-download-info/StoreOnceCATTOOLS/1.2.0/sw_free?&.

If you can open support case - HPE support provide StoreOnceCatalystDiagnostic Tool for basic operations with Catalyst Store. 

Don't forget that the Catalyst is controlled by ISV  - BE, NBU or Veeam. And all changes made without their participation will lead to data desynchronization.

The device itself has internal data integrity verification mechanisms that cannot be started on the administrator's command line.

Regards, YA. 

[TheSeeker]

thanks for that; do you have a link for this catalyst store copy utility 

unfortunetly the storeonce devices are old and out of support otherwise I would have logged a job

[TheSeeker]

sorry my bad; I will try this cat tool to copy data to another store and try connecing to that store to do an inventory and catalog 

[TheSeeker]

@yilmazaydin I tried it however it appear to be not able to copy the data

I created a new store with the correct permission; updated the config file however on the new store

when I run the command it can identify the source and destination server and the stores okay 

however when I look at the hpstoreonce event it says

[ssid1] : Catalyst outbound copy job from item BEOST_00000255 in store cancelled by user

I am using plugn files; it does copy data on the target sotre which shows the file name however the data size is 0 bytes 

[yilmazaydin]

Hi @TheSeeker . 

Unfortunately, the device is not supported and the task is very specific, which takes this question beyond the scope of this forum.

You can write me directly, maybe I can help.

Regards, YA.