- Community Home
- >
- Software
- >
- HPE Morpheus Software
- >
- HPE Morpheus VM Essentials Software
- >
- Re: tee /sys/class/scsi_host builds up
Categories
Company
Local Language
Forums
Discussions
- Integrity Servers
- Server Clustering
- HPE NonStop Compute
- HPE Apollo Systems
- High Performance Computing
Knowledge Base
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Knowledge Base
Forums
Discussions
- Cloud Mentoring and Education
- Software - General
- HPE OneView
- HPE Ezmeral Software platform
- HPE OpsRamp Software
Knowledge Base
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
3 weeks ago
tee /sys/class/scsi_host builds up
Hello,
We are now running an internal PoC and I've found the nodes to run 'tee' every 2 minutes. They are never finished and piling up, increasing load to 1000s after a few days and blocking reboots since even kill -9 does not kill them. Is it a known problem? I've seen it with 8.1.2 and now with 9.0 as well.
root 223698 223690 0 08:08 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 223699 223698 0 08:08 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 224170 224162 0 08:10 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 224171 224170 0 08:10 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 224792 224784 0 08:12 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 224793 224792 0 08:12 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 225266 225258 0 08:14 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 225267 225266 0 08:14 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 225733 225725 0 08:17 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 225734 225733 0 08:17 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 226209 226201 0 08:19 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 226210 226209 0 08:19 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 226692 226684 0 08:21 ? 00:00:00 sudo tee /sys/class/scsi_host/host1/scan
root 226693 226692 0 08:21 ? 00:00:00 tee /sys/class/scsi_host/host1/scan
root 227279 19366 0 08:22 pts/1 00:00:00 grep --color=auto tee
root@chdc2hvm01:/home/superuser# ps -ef|grep tee|wc -l
805
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago - last edited 3 weeks ago
3 weeks ago - last edited 3 weeks ago
Re: tee /sys/class/scsi_host builds up
I’ve seen something intermittent in the past.
Can you please check the kernel version in the hosts using “uname -a”?
It might also be worth it updating the packages in the host by running:
sudo apt update
sudo apt upgrade
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Re: tee /sys/class/scsi_host builds up
Good morning,
We have the same issue with 2 customers.
Both customers are using the following kernel (all hosts are updated with apt update && apt upgrade):
6.8.0-117-generic
Both are also using the smartpqi driver:
version: 2.1.26-030
description: Driver for Microchip Smart Family Controller version 2.1.26-030
srcversion: 4E3F014CA5DB70DCF642B3B
vermagic: 6.8.0-117-generic SMP preempt mod_unload modversions
1 customer is a GFS2 2-node cluster with a SAS attached MSA2070
1 customer is a single node with local storage.
Another customer that is using the megaraid_sas driver does not have this issue.
I have now created a ticket for both.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Ynt: tee /sys/class/scsi_host builds up
Hello,
The following command is normally used to trigger a SCSI bus rescan and discover new LUNs;
echo "- - -" | sudo tee /sys/class/scsi_host/host1/scan
In this case, the SCSI scan appears to become blocked inside the kernel or storage driver. The tee process is therefore likely entering an uninterruptible sleep state (D state). Processes in this state cannot be terminated with kill -9 until the underlying kernel operation returns.
Since another scan is initiated approximately every two minutes, the blocked processes continue to accumulate. This also explains the extremely high load average, as uninterruptible I/O processes are included in the Linux load calculation.
Similar behavior has been reported on VM Essentials 8.1.x and 9.0, including HPE Support Case 5402620210. Recent reports show a
possible correlation with;
Kernel: 6.8.0-117-generic
Driver: smartpqi 2.1.26-030
Smart Array/local storage and SAS-attached MSA environments
A comparable environment using the megaraid_sas driver reportedly did not experience the issue. However, this correlation has not yet been published as a confirmed root cause by HPE.
Could you please collect the following outputs from an affected host?
uname -a
cat /sys/class/scsi_host/host1/proc_name
readlink -f /sys/class/scsi_host/host1/device/driver
modinfo smartpqi | egrep "filename|version|srcversion|vermagic"
lspci -nnk | grep -A4 -Ei "RAID|SAS|SCSI"
ps -eo pid,ppid,stat,wchan:40,etime,cmd | grep -E "tee|scsi_host"
pstree -aps <TEE_PID>
sudo cat /proc/<TEE_PID>/stack
sudo journalctl -k -b | egrep -i \
"smartpqi|scsi|timeout|reset|abort|blocked|hung|I/O error"
In particular, please check whether the tee processes are in D state and whether their kernel stacks contain scsi_scan, smartpqi, or pqi functions.
As immediate precautions, we recommend;
Do not initiate any additional manual SCSI scans.
Identify the process or HVM Agent task launching the scan by checking its parent process.
Do not stop or disable the HVM Agent without HPE guidance, as this may affect management and HA functions.
If a reboot is required, migrate or safely shut down the workloads and place the host into maintenance mode first.
Open an HPE support case and reference the similar case 5402620210.
Ask HPE whether the VM Essentials v9.0.0_2 hotfix includes a correction for this issue.
Update the kernel, smartpqi driver, or controller firmware only after confirming compatibility with the installed VM Essentials version.
Based on the available evidence, this should be investigated as a possible smartpqi/kernel-level SCSI scan issue. The accumulating tee processes appear to be a symptom rather than the actual cause.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Ynt: tee /sys/class/scsi_host builds up
Good afternoon,
Below is the output of the requested commands. I'm also going to add the mentioned casenumber and potential hotfix to the support case I have running. Some commands I have cropped the output due to the character limit.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Ynt: tee /sys/class/scsi_host builds up
Hello,
Thank you for providing the requested command outputs. The results strongly support the initial assessment that tee itself is not the root cause. The process becomes stuck while the kernel is performing a SCSI/SAS rescan.
The following findings are particularly significant;
Kernel: 6.8.0-117-generic
Driver: smartpqi 2.1.26-030
Controller: HPE Smart Array P408i-a SR Gen10
Controller firmware: 8.00
Multiple tee processes are in uninterruptible D state.
The kernel repeatedly reports blocked tasks in the following call path:
scsi_scan_target
scan_channel_zero
sas_user_scan
This confirms that the processes are blocked inside the SCSI/SAS scanning path. Because the system call does not return from the kernel, kill -9 cannot terminate these processes.
The most likely sequence is that the first SCSI scan becomes stuck inside sas_user_scan(). Subsequent scans are then launched and wait behind the original scan, causing the number of blocked processes and the system load average to increase continuously.
Based on the process timestamps, this host appears to launch a new scan approximately every six minutes and thirty seconds;
01:19:55
01:13:25
01:06:55
01:00:25
We also noticed that the oldest tee process, PID 8834, is in R state rather than D state and has been running for more than one day. This may be the original scan that is consuming CPU or holding the scan operation while the subsequent processes wait.
The kernel log does not currently show an obvious controller reset, controller-offline event, or general disk I/O failure. Therefore, the available evidence points more strongly to a software interaction between the SCSI/SAS rescan mechanism, the smartpqi driver, and the installed kernel rather than an immediate physical storage failure.
Firmware version 8.00 also appears to be the current version for the P408i-a controller, so there is no clear indication that outdated controller firmware is the cause.
If possible, could you also collect the following information?
ps -p 8834 -o pid,ppid,stat,pcpu,psr,etime,wchan:40,cmd
sudo cat /proc/8834/syscall
For one of the current D-state tee processes, please also run:
pstree -aps <D_STATE_TEE_PID>
sudo cat /proc/<D_STATE_TEE_PID>/stack
cat /proc/<D_STATE_TEE_PID>/cgroup
The previously provided pstree output was collected for PID 8834, while the stack output was collected for its sudo parent. Checking a current D-state tee process may help identify the service or HVM Agent component launching the recurring scans.
Adding the findings to the existing HPE support case is the correct next step. We recommend asking HPE to confirm;
Whether this is a known defect involving kernel 6.8.0-117 and smartpqi 2.1.26-030
Whether a supported newer smartpqi driver or kernel patch is available for VM Essentials
Whether VM Essentials v9.0.0_2 contains a relevant hotfix
Whether there is a supported method to temporarily prevent overlapping periodic SCSI scans
Whether the case can be correlated with the similar HPE case 5402620210
We would not recommend manually installing an out-of-tree smartpqi driver or disabling the HVM Agent without HPE guidance, as these actions could affect platform support, storage access, management, or HA functionality.
Until HPE provides a supported fix or workaround, please avoid initiating additional manual SCSI scans. If the host must be restarted, the workloads should first be migrated or safely shut down and the host placed into maintenance mode.
In summary, the new outputs confirm a persistent kernel-level SCSI/SAS scan stall. The accumulating tee processes are a symptom of that stall, with the smartpqi driver and kernel interaction currently being the strongest suspected cause.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Ynt: tee /sys/class/scsi_host builds up
Hello everyone,
this issue is being looked at so we can provide a solution or an acceptable workaround in the near term.
There's a recommendation from the Proliant team stating the following and the HVM team is looking at how to incorporate the recommended driver:
"It is strongly recommended to use controller firmware version 8.00 for SR SAS/SATA controllers and firmware version 03.01.44.040 for SR tri-mode controllers, along with Windows 2022/2025 driver version 1016.30.0.1014, Linux driver version 2.1.38-022, and VMware ESXi driver version 80.4880.0.109/90.4880.0.109, as this combination has been fully qualified." (Firmware Package - HPE Smart Array P408i-p, P408e-p, P408i-a, P408i-c, E208i-p, E208e-p, E208i-c, E208i-a, P204i-c, P416ie-m and P816i-a SR Gen10 and SR308i-o,SR308i-p Gen11 controllers | HPE Support Center)
As a temporal workaround, the support case stated above has provided a workaround for the time being; as soon as engineering confirms it is an acceptable workaround, I'll post it in this thread.
PS. This is what makes this community great! Thanks @CalvinZito for caring for it!
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
a week ago
Ynt: tee /sys/class/scsi_host builds up
I'm bringing some news about this topic.
The problem will be completely solved in our next feature release, an update to the HVM OS will be needed because the fix is part of the kernel that will ship then.
For current systems running kernel 6.8.0-117 that present the issue, a workaround has been provided to block the writes to the SCSI scan file for the smartpqi driver. This is done by using the following commands:
echo 'SUBSYSTEM=="scsi_host", DRIVERS=="smartpqi", ATTR{proc_name}=="smartpqi", RUN+="/bin/chmod 000 /sys/%p/scan"' > /etc/udev/rules.d/99-smartpqi.rules
udevadm control --reload-rules
udevadm trigger --subsystem-match=scsi_host
Note: this won't fix existing stacked processes - only a reboot (or likely a hard reset since they're in uninterruptible sleep) will get rid of the process that have already got stuck, workaround will prevent the new process from getting stuck.
PS. This is the recommendation made in the case @duru19 referenced and our HVM engineering team provided it
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]