The delay is likely caused by the server being unable to do a reverse dns lookup for the client machine.  There are a couple of solutions:

1. Add the client’s hostname and IP address to the server’s /etc/hosts file
2. Add the PTR record for the client machine to the DNS server.
3. Add the following line to the server’s /etc/ssh/sshd_config file:

   UseDNS no