It is 2020, and we can say for sure, that ssh-server is still one of the most popular services on Linux systems. During various meetings, I was often asked: How to secure SSH?  Even though this question seems trivial, it is not. There are a lot of things to remember to accomplish well-designed service security. Secure shell is used not only for a remote-shell, per se. Many other technologies depend on it, for various reasons. For example:

  • Git to download and push the code. ​[1]​
  • Ansible to apply tasks. ​[2]​
  • Cloudera for installation procedures. ​[3]​
    (Do not ask us, we do not understand it too.)  

I have decided to answer the above question once and forever, and always refer to this document, trying to keep it up-date and consistent with current security practices. The purpose of this post is to answer the question: How to secure SSH? It is a comprehensive document for infosec and devops teams, that will guide them through all known best-practices and hardening rules.

Server

This paragraph focuses on secure-shell server risks assesment, hardening practices and configuration best-practices. The server configuration is mostly put at /etc/sshd/sshd_config.


Contest the need 

The first and the most basic question the Reader should ask himself: Is there even a need for an OpenSSH server at all, for my specific use-case? At the time of writing this article, it is not uncommon to build fully declarative, golden-image infrastructures. We have a wide set of serverless services and technologies that only run a process inside the container. I would say, for sure, that the majority of the infrastructures we, as @sysdogs, provision for our customers, do not have OpenSSH enabled at all. (You can always contact us, if you want to know, how we do it.)

Another perfect example for disabled ssh-server is a local-workstation.
There is no need for a secure-shell server on a local laptop, for instance.


Management interfaces-only

While reading a variety of resources, like Security Technical Implementation Guidelines, they always mark  access management as the most crucial part, and top priority. It is required to bind the server only to the interface that is trusted.

  • For Enterprise infrastructures virtual-machines can have their own management interface with proper firewalld rules. ​[4]​
  • For virtual private servers, we can align this rule to the trusted subset of public addresses that are whitelisted in the firewall. 
  • In the Cloud environments, we can build a specific security group that defines the access to the server or use other cloud services, like DigitalOcean Firewall. ​​[5]​ ​[6]​

Listing. firewalld trusted zone set on interface eth1.

firewall-cmd --set-default-zone=drop
firewall-cmd --zone=trusted --change-interface=eth1 --permanent
firewall-cmd --zone=trusted --add-service=ssh --permanent

Mandatory Access Control – SELinux

I am the guy, that would do setenforce 1 even if it was already set to 1.
Just in case. To be sure. Being absolutely honest, I strongly believe SELinux is one of the most helpful services in the Linux world to improve and harden your operating system.

I have read a lot of documentations and tutorials that recommend the end-user to disable SELinux or AppArmor. I completely understand their authors. They told you to do that, because it simplifies the process of software installation and removes the burden of hardening. On the other hand, I fully believe it is like getting your house alarm wrongly configured, getting upset because of the false-positives, then disabling it for the good feeling of yourself.

Listing. Enabled SELinux and labeled sshd process.

[[email protected] ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31
[[email protected] ~]#
[[email protected] ~]# netstat -pnltuZ | grep ":22"
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      696/sshd             system_u:system_r:sshd_t:s0-s0:c0.c1023
tcp6       0      0 :::22                   :::*                    LISTEN      696/sshd             system_u:system_r:sshd_t:s0-s0:c0.c1023
[[email protected] ~]#


Use Protocol 2

Sincerely, I have never seen any secure-shell server with enabled Protocol 1. To maintain the state of the art and comprehensiveness of this article, it is important to note it here, that it is insecure and it is not recommended to use Protocol 1.  It has a set of vulnerabilities and problems with security. Protocol 1 support was removed in RHEL7.4.

Listing. sshd configuration parameter for Protocol.

Protocol 2

Use tcp/22

I have seen a lot of write-ups that recommend using a different port than tcp/22. I completely disagree with this argument.  It is not very difficult to scan all possible ports on the target.  Secondly, security by obscurity always leads to something weird. By definition, OpenSSH always presents itself first. Last, but not least; using SELinux, you are forced to run the secure-shell server on tcp/22.

Listing. Port configuration.

Port 22

Listing. sshd listening on tcp/22.

[[email protected] ~]# ss -pnltu | grep 22
tcp     LISTEN   0        128              0.0.0.0:22            0.0.0.0:*       users:(("sshd",pid=696,fd=4))
tcp     LISTEN   0        128                 [::]:22               [::]:*       users:(("sshd",pid=696,fd=6))
[[email protected] ~]#


 Do not use tcpwrapperd

A set of known Linux websites still recommend using tcp-wrappers as another security layer for sshd. I think it is wrong. 

The very first argument for that is that tcpwrappers seems to be a deprecated technology. ​[7]​ It was released over 20 years ago (8 April 1997) and does not seem to be developed anymore. Another factor is that, through this deprecation, maintenance of another security layer, may in paradox add another attack vector for the system and networking.  The modern firewalls, intrusion detection systems and possibilities of isolation may completely replace tcp-wrappers functionalities. In the end, fresh operating systems do not link libwrapd to sshd.

Listing. No tcpwrappersd library linked to sshd binary on Centos/8.

[[email protected] ~]# ldd /sbin/sshd
	linux-vdso.so.1 (0x00007fff47b1a000)
	libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007f107113a000)
	libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f1070f10000)
	libpam.so.0 => /lib64/libpam.so.0 (0x00007f1070d00000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f10709bc000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f10704d9000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f10702d5000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f10700d1000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f106feba000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f106fc91000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f106fa7a000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f106f84f000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f106f5ff000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f106f30f000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f106f0f3000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f106eeef000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f106eb2d000)
	libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f106e927000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f106e707000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f106e4fe000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f106e2d7000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f106e0c0000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f106deba000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007f106dc60000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f106d944000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f106d72c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1071613000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f106d4a8000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f106d297000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f106d093000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f106ce41000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f106cc39000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f106ca18000)
[[email protected] ~]#

Do not accept password-logins

The only allowed authentication method should be Public-Key authentication. Passwords are way easier to break, brute-force and this configuration limit the attack-vector.

Listing. Configuration parameters to disable password logins.

AuthenticationMethods publickey
PasswordAuthentication no
PermitEmptyPasswords no

Do not accept forwarding

X11Forwarding is a Linux mechanism created to expose X11-server into the remote-shell. It very is difficult to imagine the reason to use graphics software on Linux server, per se. That being said, I do only see enabling this as another vector of possible attack, and risk to omit firewalls.

Port-forwarding is the functionality that allows the client to bind local-port to a remote address. It is a very straight and easy way to omit and skip firewall-rules and break compliance.

Agent forwarding is a mechanism of pass

Listing. Configuration parameters to disable forwarding.

X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no

Minimize MaxSessions and MaxAuthTries

The authorised user should not have any difficulties with logging in successfully.. Of course, there are exceptions to that statement, but, in general, it is safe to assume it should be completely sufficient to have two tries during one connection session. In contrast, the attacker trying to apply brute-force attacks needs as much login tries as possible. Therefore, i think we can all agree, minimizing MaxAuthTries is a good way of hardening.

Another useful hardening is to limit possible multiplexed sshd_sessions with MaxSessions.

Listing. MaxSessions and MaxAuthTries configuration parameters.

MaxAuthTries 2
MaxSessions 10

Add metadata-information

This part combines the usage of two separate files, motd and banner. 

Both should be used accordingly. motd is the file that is displayed by Unix after a successful login and before executing a shell. The banner is a file that is visible for each user trying to connect before the authentication process.

Listing. Examples of motd and banner files.

#Unauthorized access to this device is prohibited!#
You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only.
By using this IS (which includes any device attached to this IS), you consent to the following conditions:

- The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations.
- At any time, the USG may inspect and seize data stored on this IS.
- Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose.
- This IS includes security measures (e.g., authentication and access controls) to protect USG interests—not for your personal benefit or privacy.
- Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details.

Do not use HostbasedAuthentication, rhosts

rhosts refers to UNIX-based mechanism of allowing another computer on the same network to login. The rhost file contains hosts, usernames that determine, who can login to a system remotely without putting any password. It can contain positive and negative entries, which refer to either explicitly allowed access or contrary, explicitly denied access. HostbasedAuthentication allows to login from specific host to login and host without other authnetication methods.

Listing. Example of rhosts-file.

hamlet dewey

Listing. sshd configuration parameter to ignore it.

HostbasedAuthentication no
IgnoreRhosts yes

Do not accept root-login

It should not be allowed for the end-user to login to root directly. There are many reasons for that. For example, one is  that it limits the auditing possibilities.  It is way easier to audit and do the forensics when the user has to log in and then escalate its privileges to the superuser from there using tools like sudo.

Listing. Disable root login.

PermitRootLogin no

Configure KeepAliveSession

Inactive sessions should be terminated. It limits persistence, therefore at least partially the persistance part of MITRE ATT&CK ​[8]​ matrix harder to apply, it is also also a good lesson for forgetful users.

Listing. Configuration parameters for KeepAlive.

ClientAliveInterval 5m
ClientAliveCountMax 2

Use second-factor authentication

Since OpenSSH6.3 it is possible to use two-factor authentication, and I would strongly recommend to use it. We can use software OTP, like Google Authenticator ​[9]​,  or  hardware keys, like Yubikey ​[10]​ or Google Security Key ​[11]​.


Use a strong set of algorithms

The default setting for the current encryption and hash algorithms is not really the best option out there. We can refer to the various standards like PCI DSS ​[12]​ , FIPS-140-3 ​[13]​.  It is important to remember that the secure-shell server encrypts the connection through the standard public-key-infrastructure. Said that, the same problems with Logjam attack still apply to this service, so increasing moduli param in Diffie-Hellman algorithm is required. ​[14–16]​

Listing. Cryptography configuration parameters.

KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]

Protect server-keys

Host keys should be accessible by the root user only. Once deployed, it is recommended that Host keys should be registered in an automated inventory system. Server-keys should be also configured to be monitored by host intrusion detection systems, like Auditbeat ​[17]​ or AIDE ​[18]​.


Use port-knocking

The paranoids can use port knocking. It is an automated way to open  external ports enacted by attempting to connect to a prespecified set of closed ports. When the Client applies the correct sequence in a specified timeframe, firewall rules are dynamically modified to allow the host to attempt connection to a specific port.


Build a Certificate Authority

The paranoids can create a full certificate authority for the whole inventory. ​[19]​

Listing. Certificate authority configuration parameters in sshd_config.

TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Client

This paragraph focuses on secure-shell client configuration practices and hardening rules.


Physical access to workstation


Hash known_hosts file

Hashing the known_hosts file prevents leakage and information disclosure. The following examples present non-hashed and hashed known_hosts files.

Listing. An example of non-hashed entry.

255.255.255.255 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==

Listing. An example of hashed entry.

|1|iAsVNM5h7KXr9LM59l6UXSdjLpk=|IWK/TOxygz+vQNt9R600PNG4/ts= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMH96GPuCQuE6JIWqmuzClWiecxwc7orVmisNPQyN0eW7K0Jj3kJ/GsuxPr3AujjiJPQHO3M8m4Oqd+/0p254/Q=

Protect private-keys with passphrase

SSH keys should be protected with a strong passphrase; they should be never copied from one system to another using disks, USB drives and tokens.

They should have very, very strict permissions as well.

Listing. Read-only mode for all ssh-keys.

umask 077
chmod 0400 ~/.ssh/*

Unpickable

Agent-forwarding

Disable agent-forwarding. Since OpenSSH7.3 it is possible to jump through a server in a way better-automated fashion, using ProxyJump. ​[20]​

Listing. Disabled agent-forwarding on client-side.

Host *
    ForwardAgent no

Automation

The most important part of my work is automation. I try to automate everything. You can apply automation for security too.

Listing. An example of Lynis scan.

[[email protected] lynis]# ./lynis audit system
[...]
[+] SSH Support
------------------------------------
  - Checking running SSH daemon                               [ FOUND ]
    - Searching SSH configuration                             [ FOUND ]
    - OpenSSH option: AllowTcpForwarding                      [ SUGGESTION ]
    - OpenSSH option: ClientAliveCountMax                     [ SUGGESTION ]
    - OpenSSH option: ClientAliveInterval                     [ OK ]
    - OpenSSH option: Compression                             [ SUGGESTION ]
    - OpenSSH option: FingerprintHash                         [ OK ]
    - OpenSSH option: GatewayPorts                            [ OK ]
    - OpenSSH option: IgnoreRhosts                            [ OK ]
    - OpenSSH option: LoginGraceTime                          [ OK ]
    - OpenSSH option: LogLevel                                [ SUGGESTION ]
    - OpenSSH option: MaxAuthTries                            [ SUGGESTION ]
    - OpenSSH option: MaxSessions                             [ SUGGESTION ]
    - OpenSSH option: PermitRootLogin                         [ SUGGESTION ]
    - OpenSSH option: PermitUserEnvironment                   [ OK ]
    - OpenSSH option: PermitTunnel                            [ OK ]
    - OpenSSH option: Port                                    [ SUGGESTION ]
    - OpenSSH option: PrintLastLog                            [ OK ]
    - OpenSSH option: StrictModes                             [ OK ]
    - OpenSSH option: TCPKeepAlive                            [ SUGGESTION ]
    - OpenSSH option: UseDNS                                  [ OK ]
    - OpenSSH option: X11Forwarding                           [ SUGGESTION ]
    - OpenSSH option: AllowAgentForwarding                    [ SUGGESTION ]
    - OpenSSH option: AllowUsers                              [ NOT FOUND ]
    - OpenSSH option: AllowGroups                             [ NOT FOUND ]
[...]

You did it. 💀 Thank you. 🥃
If you need some help with your infrastructure, contact us.

References

  1. [1]
    (n.d.) Git on the Server – Generating Your SSH Public Key. Git.
  2. [2]
    (n.d.) Connection methods and details. Ansible.
  3. [3]
    (n.d.) Connecting to Cloudera Manager. Cloudera.
  4. [4]
    (n.d.) USING ZONES TO MANAGE INCOMING TRAFFIC DEPENDING ON SOURCE. RedHat.
  5. [5]
    (n.d.) How to create firewall? DigitalOcean.
  6. [6]
    (n.d.) Security groups for your VPC. Amazon Web Services.
  7. [7]
    (n.d.) Deprecate TCP wrappers. Fedora Project.
  8. [8]
    (n.d.) MITRE ATT&CK®. MITRE ATT&CK®.
  9. [9]
    (n.d.) Google Authenticator. Wikipedia.
  10. [10]
    (n.d.) Yubikey Offer. Yubico.
  11. [11]
    (n.d.) Titan Security Key. Google.
  12. [12]
    (n.d.) DOCUMENT LIBRARY. PCI-DSS.
  13. [13]
    (n.d.) FIPS-140-3. NIST.
  14. [14]
    (n.d.) OpenSSH. Mozilla Security.
  15. [15]
    (n.d.) Weak Diffie-Hellman and the Logjam Attack. Weakdh.
  16. [16]
    David Adrian, Karthikeyan Bhargavan, Zakir Durumeric, Pierrick Gaudry, Matthew Green, J. Alex Halderman, Nadia Heninger, Drew Springall, Emmanuel Thomé, Luke Valenta, Benjamin VanderSloot, Eric Wustrow, Santiago Zanella-Béguelink Paul Zimmermann (n.d.) Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice. WekaDH.Org.
  17. [17]
    (n.d.) File Integrity Module. Auditbeat.
  18. [18]
    (n.d.) AIDE. Github.
  19. [19]
    (n.d.) Scalable and secure access with SSH. Facebook Engineering.
  20. [20]
    (n.d.) SSH to remote hosts though a proxy or bastion with ProxyJump. RedHat.
About the author

GDPR