SSH Timeouts

[James Becker]

Once or twice on most days, we receive a barrage of SSH connection attempts from unpredictable external sources. The barrage reaches the SSH service limit, in effect disabling SSH until the created processes terminate.

When the SSH barrage comes from a given source, it repeats this pattern:
1) A process starts under the TCPIP$SSH username (start time as reported in the process termination record).
2) 2-3 seconds later, a login failure or a breakin detection is logged for that PID.
3) Less than a second later, the same source returns to step 1, creating another process. However...
4) The process created in step 1 terminates 10 minutes later with %SYSTEM-S-NORMAL. That is, it spends over 99% of its life serving no useful purpose, and tying up a connection,

This repeats until we run into the service limit for SSH, which we have at 10, or until the attacker gives up.

In effect, an attacker can use up our available SSH connections in less than 30 seconds, and the denial of service lasts for about 10 minutes.

Can we shorten the 10-minute timeout? That would at least reduce the duration of the denial of service.

When we had a higher service limit, that too would get used up in SSH barrages.

We want to use SSH more broadly (as a replacement for unencrypted telnet & ftp), but the more we depend on it, the more we're vulnerable to this denial of service.

Any suggestions?

$ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 4
on a AlphaServer ES45 Model 2 running OpenVMS V7.3-2

[Andy Bustamante]

Are your allowed connections coming from known IP addresses? If so see the help for TCPIP SET SERVICE /ACCEPT. We only allow connections from known networks that can reach DCL. For example:

TCPIP SET SERVICE SSH /ALLOW = (208.46.36.0:255.255.255.0)

Once you use /allow anything that isn't allowed is automatically rejected. You'll need to disable/enable SSH for the change to work.

[James Becker]

Yup, if we knew in advance where the allowed connections were coming from, we'd be home free. No such luck.

We also don't know in advance where the unwelcome connections would come from. Besides, there are too many sources (160 so far in 2005) for set service /reject ("Maximum is 16").

[Stanley F Quayle]

Welcome to life on the Internet. My border machine gets SSH attempts hundreds of times a day.

You could get a machine whose only purpose in life is to accept the SSH connections. That would allow you to run the connection limit waaay up without messing up your production machine.

Since you want to use SSH to replace FTP, that's perfect -- the client they use can use the encrypted tunnel via port forwarding to hit the production system.

The border machine could be something small like a DS-10L. Last I knew, Island Computer was selling them for ~$1k.

[Willem Grooters]

james,

I'd use /ALLOW if the number of acceptable sources is limited (which is normally the case) since the vast majority of requests will come from unacceptable sources (way above the limit). Not allowed = rejected.

I agree it won't help a lot if valid requests come from a dynamic source (dial-up ISP connection, DSL via most ISP's, different locations...) but it limits the problem.
A border machine, or firewall perhaps, could indeed be the solution. Even a PWS600au or XP1000 would do - sold at Island for (far) less than a DS10L. You can get more RAM in the machine boosting the number of cooncurrent SSH sessions, I bet. It's a slower CPU but I guess most users won't even mark the difference.

Willem

[James Becker]

Those are good points, but what I was actually hoping for was a way to get those processes not to last for 10 minutes. I gather that this isn't configurable.

If the processes terminated upon login failure, my problem would be largely solved. That is, DoS would be achieved only if n attackers all made the attempt at about the same time. But because the processes last for 10 minutes, DoS is achieved if only 1 attacker makes n attempts within a 10-minute period. (n as in set service/limit=n)

If I get another VMS box to serve as an intermediary, I have the same problem, just on a different box.

[Stanley F Quayle]

> If I get another VMS box to serve as an intermediary, I have the same problem, just on a different box.

Right. But you can get a box for people on the 'Net to beat up that's very cheap, with enough resources that you can have thousands of these hanging connections without a problem.

You could even run something like Watcher or Hitman on it and zap connections that aren't pulling I/O.

[James Becker]

I've found a solution that's close to what I was hoping for (and perhaps as close as I'm going to get).

LoginGraceTime is the operating parameter. It's 600 seconds by default. I changed it to 30 seconds. This is in TCPIP$SSH_DEVICE:[TCPIP$SSH.SSH2]SSHD2_CONFIG. FWIW, 30 seconds happens to be the same as my LGI_PWD_TMO setting.

As "luck" would have it, an obliging attacker launched a barrage of SSH connection attempts after I changed LoginGraceTime -- 84 attempts in just under 6 minutes, roughly one attempt every 4.2 seconds.

If LoginGraceTime had been 600, I'd have had 84 concurrent SSH sessions from that one source address. If my connection limit was less than 84 + <# of welcome connections>, there'd have been a denial of service, from just one attacker.

With LoginGraceTime at 30 seconds, I never had more than 8 connections at a time (from that source). There were 7-8 concurrent processes for a period of slightly less than 5 minutes.

Even if the connections had arrived twice as fast, there wouldn't have been more than 16 at a time, for a duration of less than 3 minutes.

I've raised the connection limit, a risk I can bear now that I've shortened LoginGraceLimit. With LoginGraceLimit at 30 seconds, several attackers would have to hit me within the same few minutes for a denial of service. Previously, one attacker could deny service without outside help ;-).

[Eberhard Wacker]

Hi James, very interesting !

Remark: Eco 5 is available since June 2005, it contains SSH service V3.2 instead of V2.? of Eco 4. A real big hop.

For us it means the first time to be able to use SSH on VMS along with our internal security related demands (identifying remote host etc. at login time without using own written scripts).

Cheers
EW

[James Becker]

Just checking in with a follow-up.

The LoginGraceLimit solution (30 seconds vs. 600) is working well enough.

So far, all attackers, without exception, appear to be waiting for the login failure before they attempt another connection. As a result, the connection attempts come every 2-4 seconds. In 30 seconds, therefore, you'd expect up to 15 or so connections.

TCPIP SHOW SERVICE SSH/FULL bears that out. The peak it reports has been 15 + <# of expected sessions>, even when they make hundreds or thousands of attempts.

I'm not entirely safe, of course. If enough attackers all try at the same time, or if they start making rapid-fire attempts without waiting for the login failure, I'll be in trouble again. So far, that hasn't been the pattern.

[Stanley F Quayle]

How about assigning some points to the responses?

Pointer to help on points:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33

Thanks in advance.