Restore speed of HPE StoreOpen_for_windows became 1/10 after applied WindowsUpdate(20

[kawamoto76]

 I am using HPE_StoreOpen_Software(64bit) for windows on windows10Pro(64bit)-22H2_19045_3086(now) on hp z820(16C-32T, mem=128GB) with LTO-6 Ultimum6250(SAS). There, WindowsUpdates have been applied on my z820 system many times scince installed first.

Recently I found restoring(form LTFS tape to HDD ) speed with HPE_StoreOpen_Software became too slow comparing with that of before, it became lower than almost 1/10 of previos value.  

As to about 4GB .iso file, ignorering tape seek time to find the record, restoring was completed within 1 min. on 2023 Jan., as is shown [OK]. 

However, only very low restore speed(below 10MB) could be got for windows10_21H1, for windows10_22H2 and for windows11_22H2 restoring to internal SAS-HDD as is shown [slow]. This is also almost the same as for external USB HDD.

[OK] Normal restore speed as before

[slow]  speed down of restore  after WindowsUpdate

 In my file bank maintenance, backup(from HDD to LTFS tape) operation is done frequently, whereas scarcely I met a necessity for restore operation.  This is common condition about other users, I think. Then I have not been awaring this slow-down change.

Then checked restore speed as to different OS, the result is as follows.

1. HPE_StoreOpen_Software for RHEL on Scientific_Linux_7.9  >>  100-150MB/sexc
2. HPE_StoreOpen_Software for windows on windows8.1Pro  >>  about 100MB/sec
3. HPE_StoreOpen_Software for windows on windows10Pro(22H2)  >>  about  8MB/sec
4. HPE_StoreOpen_Software for windows on windows11Pro(22H2)  >>  about  8MB/sec

My hardware:

Hp z820 workstation + SAS9217-4i4e + Ultimum6250SAS

CPU: E5-2687W v2  x 2 ( 16C 32T tottally)

HDD: ST3600057SS(15rpm,600GB,6Gbps) x4

RAM: 128GB

High tranfer speed(over 100MB/sec) were get for (1)windows8.1 and for (2)windows10_1809 for internal SAS-HDD.

(This is almost the same as for external USB HDD.  )

But for , (3)windows10_21H2, (4)windows10_22H2, (5)windows11_22H2, restore speed( from tape to HDD) is drastically downed.

In order to make clear the problem what happened now, I prepared z420(E5-2667v2 8C-16T, mem=64GB + SAS9217-4i4e ) as test machine.

And I prepared many system-disks, one SAS-data-disk and SATA(USB3.0 Ext.)-data-disk.

Windows 10 release history can be found in

   https://learn.microsoft.com/en-us/windows/release-health/release-information

Windows 11 release history can be found in

  https://learn.microsoft.com/en-us/windows/release-health/windows11-release-information

And install USB install media can be made through "UUP dump"  and "Rufus".

   https://uupdump.net/

 In order to find the turning point from what revision the restoring speed become slow down, I made experiments about each revisions of windows10Pro_22H2. There some revisions were skipped to spend time.

After OS installed, hp drivers, LTO tape driver and HPE_StoreOpen_Software(64bit) for windows are installed for all system disks.

The data_disks were formatted as NTFS, and ACL was set full control for 'Everyone' on the disk, respectively.

Then, restore(from LTFS to data-HDD) speed of 4GB sized file was monitored as to each OSs.

Results of each revision are shown below, where "OK" represents normal restoring speed (above 100MB/sec)  and "slow" represents slow one (below 10MB/sec).

 (1)windows8.1

    REVISION        released_date   test_result

    final                 2023-Jan.          OK

(2)windows10_1809

    REVISION        released_date   test_result

17763.4499         2023-06-13         OK      KB5027222

(3)windows10_21H2

    REVISION        released_date   test_result

19044.3086         2023-06-13        slow   KB5027215

(4)windows10_22H2

22H2_REVISION released_date   test_result   KB

19045.2130         2022-10-18         OK

19045.2546         2023-01-19         OK

19045.2673         2023-02-21         OK

19045.2728         2023-03-14         OK

19045.2788         2023-03-21         slow   KB5023773 ( changed point )

19045.2913         2023-04-25         slow   KB5025221

19045.3086         2023-06-13         slow   KB5027215

(5)windows11_22H2 

22H2_Build       release_date      test_result  KB

22621.521           2022/9/20           OK

22621.819           2022/11/8           OK

22621.963           2022/12/13         OK

22621.1265         2023/2/14           OK

22621.1344         2023/2/28           slow   KB5022913 ( changed point )

22621.1413         2023/3/14           slow   KB5023706

22621.1635         2023/4/25           slow

22621.1848         2023/6/13           slow   KB5027231

22621.1926         2023/6/20           slow

( The date speed down is earlier than that of windows10_22H2. )

I can find, after applying WindowsUpdate 2023-03-21 (KB5023773) updating towords win10_22H2_19045.2788, the restoring speed will become slow down.

Viewing known issues of KB5023773 in "March 21, 2023—KB5023773 (OS Builds 19042.2788, 19044.2788, and 19045.2788) Preview"

https://support.microsoft.com/en-us/topic/march-21-2023-kb5023773-os-builds-19042-2788-19044-2788-and-19045-2788-preview-5850ac11-dd43-4550-89ec-9e63353fef23

and also viewing known issues of KB5023706 "February 28, 2023—KB5022913 (OS Build 22621.1344) Preview"

https://support.microsoft.com/en-us/topic/february-28-2023-kb5022913-os-build-22621-1344-preview-3e38c0d9-924d-4f3f-b0b6-3bd49b2657b9

************************************************************************

You might have intermittent issues saving, copying, or attaching files using 32-bit apps, which are large address aware and using the CopyFile API. Windows devices are more likely to be affected by this issue when using some commercial or enterprise security software that uses extended file attributes. Microsoft Office apps, such as Microsoft Word or Microsoft Excel are only affected when using 32-bit versions and you might receive the error, "Document not saved."

This issue is unlikely to be experienced by consumers using Windows devices in their home or on non-managed commercial devices. Apps are not affected by this issue if they are 64-bit or 32-bit and NOT large address aware.

This issue is resolved in KB5027215   ( for win11 KB5027231     2023/6/13).

*********************************************************************

This seems likely the reason of the slow restoring speed.

Though Microsoft says "This issue is resolved in KB5027215", our restoring speed is still slow even 19045.3086 where applied KB5027215 already.

Apart from HPE StoreOpen, different issue  has been reported for IBM's LTFS software.

"Windows update KB5025221 causing issues writing to tape with IBM Spectrum SDE LTFS software."

https://answers.microsoft.com/en-us/windows/forum/all/windows-update-kb5025221-causing-issues-writing-to/8c73a51c-63b9-45be-b2a3-0a277c5b9ccb

There reported KB5025221 make IBM Spectrum SDE LTFS unavailable.  This is more serious than HPE's issue.

Every month, many users apply only security WindowsUpdate( like KB5025221 3/14) ,  but do not feature WindowsUpdate( like  KB5023773 2/28).

Then I suppose this issue had been happened already when applied KB5023773 2/28.  Serching "KB5025221", or "KB5023706" with "LTFS", you can find some articles about IBM's issue.

  Just return to HPE StoreOpen for windows.

 Even If Microsoft’s KB5023773 is a victim for our slow restoring speed,  Microsoft will not supply new WindowsUpdates to fix StoreOpen  issue.

“HPE_StoreOpen_Software(64bit) for windows” should be changed to be affected from it.

[kawamoto76]

About the same issue, I found another article.

"Slow LTFS Copying from tape in Windows 11"

https://www.reddit.com/r/LTO_LTFS_Tape_Copy/comments/1297uxr/slow_ltfs_copying_from_tape_in_windows_11/

[LTOboy]

The issue is known to HPE.

They are working on a solution.

Does not appear to be any workaround other than going back to an earlier build of Windows.

The read from LTFS volume transfer speed can be even worse than seen above.

[NarayanKK-SO]

@kawamoto76    @LTOboy

The work is still in progress and the fix is expected in future release. There is no ETA available as of now. We just need to wait for the next versions and check the release notes for fixes and enhancements.

You have put a lot of effort in collecting the data regarding the issue. Appreciate all the efforts.

[MichaelMa]

Thank you very much for your efforts.

I have exactly the same problem with an HP LTO5 and an HP LTO6 drive.
I also have an HP Z420 and an LSI SAS controller.

At some point between windows10_21H2 and windows10_22H2, the speed of recovery from LTFS tapes dropped from more than 100MB/s to <8MB/s.

The write speed is still close to the theoretical maximum for both drives.

I have another LTO5 drive running FreeBSD where read and write speeds are >100MB/s. Even if I swap LTFS tapes between the drives, the read speed under Windows 10 is always incredibly slow and never higher than 9MB/s.

[fff8e7]

I am currently experiencing the same issue, recovering backups from an LTFS volume on a Windows operating system is an absolute disaster.

Has there been any progress on the issue so far?

[MichaelMa]

Is there now a time horizon for a release when the problem will be fixed? 6 months have passed.

@NarayanKK-SO 

[LTOboy]

Possible workaround.

Try using the free version of TeraCopy 3.17.

This seems to be immune to the Windows 10/11 bug that drastically slows down LTFS read operations.

Explorer, ltfscopy and robocopy are all affected by this bug, but TeraCopy gets the full performace back, as far as I can tell.

Hope this helps.

[MichaelMa]

Thank you very much for this useful tip. I was already using Teracopy to write the LTO tapes as the software has a logging feature. My version 3.26 (which is older than 3.17 in Teracopy - I think it should be 3.2.6) had the same problem (about 8Mb/s read speed of LTO6). After updating Teracopy to 3.17 I now have ~150MB/s read speed in the first tests. Thanks for the workaround. Michael

[fff8e7]

Try LTFSCopyGUI, it's an open source software designed for LTFS, and its code is hosted on GitHub, and despite the name LTFSCopyGUI, it actually implements a set of LTFS itself, capable of extracting archives from tapes at full speed (160MB/s for LTO-6 device).

https://github.com/zhaoyangwx/LTFSCopyGUI 

[MichaelMa]

Thank you very much - I will give it a try.

I use the goole autotranslate function on your GitHub Page from ZH to EN ... I think I get 80%.

What I don´t understand is the link to "LtfsCommand from inaxeon/ltfscmd"

[fff8e7]

This software is not developed by me, you can follow the instructions in the README.md file to set the language to English.

lang.ini to set language (Currently en for English, zh for Chinese Simplified. zh Default)
if no lang.ini exist, will follow system language setting

LTFS interface in LTFSCopyGUI modified from https://github.com/inaxeon/ltfscmd project