Using the default setup in SSH, it usually takes about 30 seconds to 1 minute to get the command prompt after the login credentials were entered. I suspected it could be the DNS resolution issue and based on this webpage, it confirmed my suspicion.
As I do not need the SSHD to get the dns name, I choose to turn off the DNS resolution option.
Steps used to resolve this:-
1. edit /etc/sshd/sshd_config
2. search for UseDNS. If it is commented, un-comment it and set to no.
3. restart sshd daemon.
service sshd restart