ssh - no added security?

[Sune Rastad Bahn]

Mark Hedges wrote:
> > Does it really matter that the install client uses ssh to save
> > log files, and that ssh is used to access the install client
> > from the server?
> >
> > Since the client mounts the filesystem containing its secret
> > keys with nfs, the secret host key and user key pass in the
> > clear from server to client when the client opens them to make a
> > connection.
>
> It seems also that make-fai-nfsroot copies the host keys of the
> installation server to nfsroot rather than generating new keys.
> debootstrap does this.  The copied host keys need to be removed
> and then a `dpkg-reconfigure ssh` will generate new ones for
> the nfsroot environment.
>
> Did the old version do this?  That would mean that until now,
> server host keys have been compromised as well as login keys,
> because the client opens them through the cleartext nfs mount.
>
> I will try to write a patch in the next couple days that fixes
> this problem with make-fai-nfsroot, and the problem of copying
> the correct host keys into the client's known_hosts file that
> Radek Hnilica <ML at hnilica.cz> reported... unless someone already
> did this...?
>
> Then, at least, the only sensitive secret key that will pass
> through the cleartext nfs mount will be the fai user's login key.
>
> --mark--

Dear Mark, 
the question of security during fai installation is a very interesting one.
We have worked a bit on tightning the security of fai but have not had the 
time to submit our patch yet. Our goals was:

 Make sure that the security of the fai-server is not compromised.

As a secondary goal we would try to make sure that the security of a machine 
which is installing from the fai-server is not compromised unnessary during  
installation.

As you also points out, just going from rsh to ssh helps very little in 
securing the system, since the key used has to be available on the nfs mount.
One has to be a little more careful.

Let's analyze the situation a bit. We need the client to be able to tell the 
server when the installation is done. Futhermore we need to move some files 
to the server, and perhaps do some minor changes on the server, such as 
changing tftpboot/bootp files (depending on the setup).

The unmodified setup involves letting the client login password less to a 
special account on the server and do all the necessary things there. The 
problem is that in reality anyone can read the nfsroot and use the 
information to get the access to the fai server (after which a local exploit 
would be enough to get full control). To remedy this situation we made a 
setup where the client has minimal influence on the fai-server. Instead of a 
full login shell, the special account on the fai-server has a special  login 
script designed to do the necessary actions. This means that all the client 
can do is tell the server that it is done. From the client point of view the 
situation is the following:
What I have (via nfs): A private key to trigger the fai-server script. A 
public host key to make sure I connect  to the right host.

Notice that the client doesn't have the public part of the key to contact the 
faiserver. This is kept secret on the server. Actually this is not added 
security since it just plays the same role as the host-key but still, it 
doesn't hurt either. 

An intruder on the network will be able to very easily trigger the script on 
the fai-server, but if the script on the server is well designed this will 
not give any more control to the intruder. If the intruder is more clever he 
can intercept the nfs communication so in theory there will be no way to 
avoid an attack of the client during istallation. As an example, the intruder 
can simply hijack the root nfs-mount already during bootup, giving complete 
control over the client. To avoid such an attack one would have to rethink 
the whole bootup process (including bootp/dhcp and tftp). If you're this 
paranoid, you probably shouldn't install over network anyway!

Having accepted the possibility that the client is taken, puts even more 
emphasize on making sure that a rotten client wont compromise the server.

OK. Lets now look at the situation from the server point of view. 
The special fai account should be locked down as tight as possible, that 
means: 
No valid passwd.
Only access through ssh (with valid key).
No login shell, only a carefully crafted loginscript.
Home directory on nonshared drive.
Proper tight permissions on all relevant files.

Since we dont want the client to be able to control whats going on, the 
copying of files will be initiated by the script, not by the client. Hence 
the server needs to be able to login to the client passwordless. This is done 
by placing the public part of a key in the authorized_keys file on the 
client. The private part is kept secret on the server. This wont compromise 
the safety of the client further, since you need login access to the fai 
account on the server, which you can only get by rooting the server, in which 
case you have easy access to the client anyway.

So the server has the following:
The public part of the client key is in authorized_keys (the private part is 
in the nfsroot).
The private part of the fai-account key (the public part is in authorized_keys 
in the nfsroot).
The host_key of the server (the public part is in the nfsroot)
The host_key of the client (the private part is in the nfsroot)

We believe that our setup is very secure with respect to the fai-server.
Probably the server is more vulnerable due to possible flaws in the various 
services (nfs, tftp,bootp and ssh), than any insecurities introduced by the 
fai setup. At least as long as we believe that the loginscript has no 
security issues :-)

How about the security of the clients then? Well, any client installed 
before an intruder is on the network should in principle be as safe as any 
other linux install. Great care should however be exercised to avoid default 
password etc. In our setup we have completely avoided valid password  on the 
root account and only allow ssh with a valid key. This way we avoid that a 
taken client will allow a brute force attack on the /etc/shadow to reveal the 
rootpasswd on all clients. In general the fact that the clients are more or 
less identical stresses the point that "there is no security through 
obscurity possible". One should therefore make sure that being root on one 
client will NOT make it easier to become root on any other client. This is 
not an easy task, but it is possible.

If an intruder is present on the network, then any new installation is in 
theory compromised. There is therefore only one way to deal with it. Get rid 
of the intruder, and reinstall all machines installed after the intruder got 
in. In practice this means reinstall ALL clients. This is probably the only 
way to make sure the intruder is really gone, and since it is probably 
impossible to figure out when he got in, its better to be safe than sorry 
anyway. Luckily with fai, the reinstallation can be done quite easily. Simply 
turn off power for all client. When all machines are down you can reinstall 
them one at a time or all together as you like by turning on the power.

If you believe that the server is also compromised, you have to reinstall that 
as well, but thats another story...

I hope this long story can help you in securing your system. I also hope to 
get time to create the patch for fai, to allow our setup to be used other 
places, and finally I hope to get time to document the setup more precise 
than I have done here.

Any comments, especially security issues we might have overlooked is very 
welcome :-)

-- 
Sune Rastad Bahn
Systemadministrator
Danmarks Meteorologiske Institut
Lyngbyvej 100
2100 København Ø
 
Direkte tlf. : 39157562
Email: srb at dmi.dk