You have multiple options, two of which involve using HP Network Optimizer. First, the APIs used by the Lync solution will work for any app. You would need to get the app vendor to write an API similar to the API written by Microsoft. Second, the latest version of Network Optimizer allows for the prioritization of generic services. You would need to be able to identify the app by server IP address, optionally adding port number.
Lastly, you can always write your own app to apply QoS policy. The Network Optimizer app pushes OpenFlow flows to the access layer switches in hybrid mode that set the queue number and rewrite the DSCP value for the packet and forward the traffic normally.
