802.1x authentication - authenticated users not sonsitently showing as online

[NeilR]

I currently run 802.1x wired on all user access ports (HP 2910). I have need to extend the network past

the access switch using a small switch (HP 2915) in some areas of the shop. These ports use the same 802

config, same radius server etc.

As I don't want to leave open ports, I use the supplicant on the 2915 to authenticate against an access

service in imc. The users have another imc access service.

When i check via terminal, the user will show as authenticated on either 2910 or 2915, has been assigned

the correct vlan, and is able to access the network.

The problem is that users do not show up consistently as online in imc. I can toggle a user's interface

enabled/disabled - Some show up right away, others don't.

Sometimes the 2910 is more reliable and the 2915 less so, but not consistently.

Its good that authentication is seemingly taking place reliably but tracking users is an important

management and troubleshooting task.

Perhaps the iMC server performance or db performance is an issue?

Is rapid toggling of user state causing the db to lag?

Are there radius accounting settings that might help? Any other suggestions?

thx!

[Pack3tL0ss · Accepted solution]

Neil,

Accounting is likely the crucial bit here.  UAM will send an access accept, IMC will then wait for an accounting packet for that user, and if it gets the appropriate accounting packet the user will then show online.  By default UAM will wait 5 seconds for the accounting packet after it sends the Access Accept.

I have seen this inconsistent behavior on some clients that take longer, for whatever reason, to send the accounting packet.  Fortunately the "Authentication Lock Time" (Amount of time UAM will wait for the initial accounting packet from an authenticated user) is adjustable.

User-->User Access Policy-->Service Parameters-->System Settings-->System Parameters...

Top Right field should be "Authentication Lock Time(Seconds) which I believe is defaulted to 5, mine is set to 15 as I was seeing the same inconsistency you were.  A packet trace run on the IMC server, filtered on radius, could confirm... The Time between the access accept and the accounting packet... if it's longer than that setting then the user will not show online.

Hope it helps,

PL

[NeilR]

This was the problem or at least one aspect of it. ImC seems to detect user log in and log out consistently.

However, I noticed if I sign in with same username more than once on the same switch I onlly see 1 entry for that user. If I log in more than once across different devices I see the correct number of entries.

I'm hoping this will solve the problem when the reauth period expires and the switch requests reauthentication. So far my users "drop off" imc after some period - I'm not sure since its overnight usually when I'm working on this.

Think that would be related to the same setting or is there something else going on.

thanks!

PS - it appears that users synched from LDAP are showing up. But local users created on imc only show up once as logged in. After that not shown as on line, even though they are shown authenticated on the switch.

[NeilR]

So looks like the change to Authentication Lock Time has reoslved the issue of users showing as auth'd on both primary and subsidairy access switches, both as changes occur and over time.

I'm set at 20 secs. Is there a downside to too long?

The description of this parameter would not have led me to see that as the solution, but makes me wonder about too long a period:

• Authentication Lock Time: A time range from the time the user is authorized to the accounting start time in which the authenticated user cannot be reauthenticated.

Still have issue with non LDAP users not showing up as logged in.

[Pack3tL0ss]

Yeah, it doesn't come right out and say that the setting can impact a users online status.  If a accounting packet comes in after the lock time has expired, that user won't show as online.

I don't see much of a down-side to bumping it up, it only locks that user.  So once user 'bob' has succesfully authenticated, for 20 seconds he is not allowed to re-authenticate, actually it's probably 20 seconds or however long it takes for the initial accounting packet to be sent.  I would imagine the timer goes away once the accounting packet comes in.

The only potential down-side I could think of is if a user authenticating wireless connects to an AP and authenticates and gets online, but roams to a new AP before the initial accounting packet is received by IMC.  I believe when that user roams another authentication request is made.  I think the likelihood is fairly low.  On the HP 830 wireless controller, the accounting packet was coming in after about 9 seconds.

By non-LDAP users, you mean they are configured directly in IMC?  I would run wireshark on the IMC server, filter by radius (and the IP of the access device if you have a lot) and see if you are seeing the Accounting packet come in after the access accept.

[NeilR]

Yes - those users configured directly in imc. In most cases I synched them via LDAP then unbound them because they were using MD5 challenge and for some reason LDAP won't accept an MD5 challenge - even though it can be configured in the services. Setting passwords later.

Have not yet w/s but I see the accouting and authtnication success for these users in the radius track feature under Users/ Access Log

UPDATE: WIreshark confirms that for these users the delay can be anywhere from 10  to 45 secs or so. I have set for a minute now and they are now showing up in the logged in users area.

THe LDAP authetnicated users seem to generate accoutning packets quicker, but still variable - not sure why

thx

[NeilR]

The on-line users are still inconsistent.  After serveral days its only displaying 3 of 8 users previously shown as on-line.

BUt if I log into the access devices I can see all of them as shown authenticated.

And its not by switch - on the same switch I have both visible and invisible users to imc.

Any ideas?

Increase the authentication lock time? its set at 60 right now.

[NeilR]

Increased to lock time to 90 secs. Seems to have helped the mac authenticated printers, phones and non LDAP users stay logged in over night.

However the LDAP users have disappeared. NAS error in the Access Details log:

Access Duration:    23Hr59Min59Sec                Offline Cause:     Nas Error

Which happens to correspond to the Max Session duration. However this doesn't really log people off - just closes out their session information.

Have I overidden the lock out somewhere else?

Value can be from zero(unlimited?) to 315360000 (ten years) - ten years will exceed my time here most likely so that will work....

[Pack3tL0ss]

I'd never ran across that, I tested it in the lab and setting it to 0 is apparently not = unlimited.  Once I killed the connections for all the clients none of them could get back on.  I set it to 315360000 and they immediately started connecting again.

I expect the feature is there as a security mechanism, essentially making clients re-authenticate after 24 hours, but it sounds like the way you are describing it, it's not sending a disconnect to the device, so the client still thinks the session is active and doesn't know to authenticate again.

If I get the chance I'll set it to a lower number and capture packets to see if UAM is sending a radius disconnect when it hits the timer.

[NeilR]

no 0 is not unlimited, unlike some other settings in imc (one of my frustrations - inconsistent implementation). 

I'm re-authing the ports at the switch level every 2 hours if that has any bearing. But I don't recall ever being denied access or cut off.

Since updating the setting to the max value, I now have some user sessions at 26 hours. Sessions under way when the value was changed expired at the 24 hour mark.

[Pack3tL0ss]

I agree, either 0 should in fact be unlimited, or it shouldn't be allowed (I can't imagine a valid use-case for effectively disabling authentication system wide... which can be done in more graceful ways).

I'll submit an enhancement request to see if we can get it changed.  For clarification when the sessions were expiring, it would just remove the session information from UAM, it would not actually send a disconnect?  Were the users still online and just not showing up in IMC as such?

Thanks,

PL

[NeilR]

correct - user sessions gone from imc, but switch shows users logged in. 

Switches are procurve 2910al-24g-4XG (W15.14.0007) & 2915-8G-PoE (A15.14.007)

I did a capture using the kick out function to test - I would think it would behave the same as a time based session expiration. (but not see below)

I also set up a show port-access authe clients on repeat. This also shows the port reset, then user reauth

I can see the accounting send the disconnect request and the switch acknowledge admin reset

Then there is an unreachable icmp from imc,

an accounting request w port disabled from switch,

then a disconnect-nak with error cause; unsupported service

The client then requests authentication

a reject message sent: E63018: The user does not exist or has not subscribed for this service.

then about 9 cycles of request/challenge, then accept

then some accounting

User is back on line on the switch and also shown online in imc.  

UPDATE: session expiration

I set session expiration to 180 secs (lock time is 90)

I disable the interface on the workstation to trigger authenctication

see the request challenge packets

then the accounting packets 

User now shows as online in imc

after session expires:

no disconnect requests from imc - switch just restarts access request/challenge

after acceptance, no accounting from imc, which signals user showing as online

some times an accounting interim update, depends on timing

then session expires and repeats

User never shows as online.

SO:

kick out is different then session expiration

radius continues to respond to switch/client requests to auth client, so user comes back online

Subsequent re-auths get no accounting acknowledge to add user to online

Neither kick or session expire really keeps user from re-authenticating anyway, at least on procurve

User session keeps session time from when they logged in - so I can change it w/o killing online sessions - but the expiration/reauth cycle does not reset the timer to a new value if I changed it in the meantime

[Pack3tL0ss]

Thanks for the effort and testing Neil.

I submitted the feature request to change the behavior of the field so that 0 is unlimited.  I'll have to gather some data against Procurve and other devices to see what it's doing at expiration and see if there is a difference and I'll update the request with the details if there is something that needs to be enhanced on the IMC side.

On the 2920 do you have dyn-authorization (required for Radius CoA) enabled against the radius (UAM) server?

radius-server host <UAM IP> dyn-authorization

It may be requesting a CoA to force the re-auth, if it's not enabled that may explain the "unsupported" response.

Again, enhancement request is in, and I'll give them more detail once I gather the data on the session expiration scenario (against multiple device types).

PL

[NeilR]

Thx for the enhancement request.

Yes dyn-authorization is set along with the key and time-window 0. These statements are added when I use the Deploy AAA configuration in User>User Access Policy>Access Device Management>Access Device section, along with the accounting update interval of 3 mins.

The procurves are shown as fully supported on the access device Details screen for Radius accounting

BTW my users are now peristing on line as I expected. Expiration and disconnect are rarely used, but still should work as designed. Good for now.

[Monty101]

Hi Neil,

 I am facing the exact same problem and have read the entire post. but am uncertain as to what you finaly put in place to resolve the issue

Authentication Lock Time(Seconds) = ?

Max. Session Duration(Seconds) = ?

I have a mixture of 2910s, 2920 and 3800 switches do I need to run "radius-server host <UAM IP> dyn-authorization"  on all of my 80 switches ? as i did not do the install and un certain if that was done ?

Thanks in advance

[NeilR]

Hi Monty,

For authentication lock time, I'm using 90 secs.

The help description of this param:

• Authentication Lock Time: A time range from the time the user is authorized to the accounting start time in which the authenticated user cannot be reauthenticated.

does make obvious as to what it does in IMC, which is the time needed to allow the authentication process to complete so UAM can register the user being logged in.

From my observation, this is the time needed once the switch has sent the request to IMC/Radius, for UAM to decide whether to authenticate, send response back to switch, then get the Radius accounting back from the swtich and count the user as logged in.

You probably want it short as possible, but for users to consistently show as logged in each time. Each form of authentication seems to take a different amount of time, ie mac vs 802.1x, and may vary by switch brand or model, and probably UAM load.

the Max. Session Duration param is acurately described in the help. After however many seconds UAM kills the session. Min is 0 (which is not unlimited, as in some other areas) and max is 315360000 which is equal to 10 years.

I'm using 315360000 because I'm using UAM for authenticating phones, network devices and users, so I want the majority of those to stay on-line for as long as needed - not to expire every day or whatever.

I would experiment - the procurve devices kill the session after time expires, and then reauthenticates it again, so the user is online but UAM doesn't show it. Other devices may work differently.  I want to see all the logged in sessions, so I set the max. I think this scenario is covered in my authentication test results post.

Regarding radius, dyn-auth,

When you add a switch as an access device under User > User Access Policy > Access Device Management > Access Device > Add Access Device, UAM adds the necessary statements to your devices. You will want to do this for any device that you plan to use for access control - other IMC features leverage this information.

Make sure you've configured snmp, telnet or ssh etc access previously. But note that if you alread have radius server specified, this step will NOT remove existing, only add IMC as an additional one. You will want to remove any existing entries.

Then when you deploy AAA configuration in

User > User Access Policy > Access Device Management > Access Device > Deploy Configuration- > AAA Configuration the commands for accounting update interval, mac format, 802.1x on/off, and 802.1x mode (EAP or CHAP ) is set on the switch.

Note that you will probably still need other 802 or mac related port commands for your specific switches.

That is what the

User > User Access Policy > Access Device Management > Access Device > Deploy Configuration- > Commands step is for.  You can use this to deploy port specific settings, or in my case I remove the extra Radius entries I was previuously using. Alternatively you could use a template in Service > Configuration Templates to make these settings first,

Then sync ports, and sync with Platform to insure IMC knows what the configuration is.

Sorry I should sum up better in those longer posts as to what the final conclusion was...

[Monty101]

Hi Neil,

Thankyou for the time and effort you put in in your very detailed response.

I will try your suggestions.

Monty