How to secure SSH?

2020-08-26|By Kamil Zabielski|Security

It is 2020, and we can say for sure, that ssh-server

is still one of the most popular services on Linux systems. During various meetings, I was often asked: How to secure SSH? Even though this question seems trivial, it is not. There are a lot of things to remember to accomplish well-designed service security. Secure shell is used not only for a remote-shell, per se. Many other technologies depend on it, for various reasons.

I have decided to answer the above question once and forever, and maintain the freshness of this answer, so that you can always refer to this document, as an up-to-date, and consistent with current security practices, knowledgebase. This post is an comprehensive document for infosec and devops teams, that will guide them through all known best-practices and hardening rules.

Contemplate the needs

The first and the most basic question the Reader should ask himself: Is there even a need for an OpenSSH server at all, for my specific use-case? At the time of writing this article, it is not uncommon to build fully declarative, golden-image infrastructures. We have a wide set of serverless services and technologies that only run a process inside the container. I would say, for sure, that the majority of the infrastructures we, as @sysdogs, provision for our customers, do not have OpenSSH enabled at all. (You can always contact us, if you want to know, how we do it.) Another perfect example for disabled ssh-server is a local workstation. There is practically no purpose in a secure-shell server on a local laptop.

Server

Management interfaces only

While reading dedicated instructions and resources, like Security Technical Implementation Guidelines, you'll notice that they always mark access management as the most crucial part of the process, and top priority. And they are right. It is very, very important to bind the server only to the interface that is trusted.

  • In Enterprise infrastructures, virtual machines can have their own management interfaces, with proper firewalld rules enacted.
  • For virtual private servers, we can align this rule to the trusted subset of public addresses that are whitelisted in the firewall.
  • In the Cloud environments, we can either build a specific security group that defines the access to the server, or use other cloud services, like DigitalOcean Firewall, Cloudflare, or any other.

Example set of firewall security rules

firewall-cmd --set-default-zone=drop
firewall-cmd --zone=trusted --change-interface=eth1 --permanent
firewall-cmd --zone=trusted --add-service=ssh --permanent

Mandatory Access Control - SELinux

I am the guy, that would do setenforce 1 even if it was already set to 1. Just in case. To be sure. Being absolutely honest, I strongly believe SELinux is one of the most helpful services in the Linux world, when you want to improve and harden your operating system.

I've read a lot of documentation and tutorials that recommend the end-user to disable SELinux or AppArmor. I completely understand their authors. They told you to do that, because it simplifies the process of software installation and fades out the burden of hardening. On the other hand, I fully believe it is like getting your house alarm wrongly configured, getting upset because of the false-positives, then disabling it, just to feel fine with yourself.

Checking SELinux status

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31
#

Checking TCP:22 port status

# netstat -pnltuZ | grep ":22"
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      696/sshd             system_u:system_r:sshd_t:s0-s0:c0.c1023
tcp6       0      0 :::22                   :::*                    LISTEN      696/sshd             system_u:system_r:sshd_t:s0-s0:c0.c1023
#

Use Protocol 2

To be honest, I have never seen any secure shell server with enabled Protocol 1, or at least, never seen such one after i've done my work on it. To maintain the state of the art and comprehensiveness of this article, it is important to note it here, that it is insecure and it is not recommended to use Protocol 1. It has a set of vulnerabilities and problems with security. Protocol 1 support was removed in RHEL7.4

.

Protocol setting in `sshd` config

Protocol 2

Use tcp/22

I have seen a lot of write-ups that recommend using a different port than tcp/22

. I do whole-heartedly disagree with this argument. It is not very difficult to scan all possible ports on the target. Additionally, security by obscurity always leads to something weird and twisted. By definition, OpenSSH always presents itself first. Last, but not least; If you're using SELinux, you're forced to run the secure-shell server on tcp/22.

Ports configuration in `sshd` config

Port 22

Checking `tcp/22` port status

# ss -pnltu | grep 22
tcp     LISTEN   0        128              0.0.0.0:22            0.0.0.0:*       users:(("sshd",pid=696,fd=4))
tcp     LISTEN   0        128                 [::]:22               [::]:*       users:(("sshd",pid=696,fd=6))
#

Do not use tcpwrapperd

A set of known Linux websites still recommend using tcpwrapperd as another security layer for sshd

. I think you shouldn't.

The very first argument for that, is that tcpwrapperd is mostly deprecated technology

. It was released over 20 years ago (8 April 1997) and does not seem to be developed anymore. Another factor is that, through this deprecation, maintenance of another security layer, may in paradox add another attack vector for the system and networking. The modern firewalls, intrusion detection systems and possibilities of isolation may completely replace tcpwrapperd functionalities. In the end, fresh operating systems do not link libwrapd to sshd.

`sshd` dependency list

# ldd /sbin/sshd
  linux-vdso.so.1 (0x00007fff47b1a000)
  libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007f107113a000)
  libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f1070f10000)
  libpam.so.0 => /lib64/libpam.so.0 (0x00007f1070d00000)
  libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f10709bc000)
  libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f10704d9000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007f10702d5000)
  libutil.so.1 => /lib64/libutil.so.1 (0x00007f10700d1000)
  libz.so.1 => /lib64/libz.so.1 (0x00007f106feba000)
  libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f106fc91000)
  libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f106fa7a000)
  libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f106f84f000)
  libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f106f5ff000)
  libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f106f30f000)
  libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f106f0f3000)
  libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f106eeef000)
  libc.so.6 => /lib64/libc.so.6 (0x00007f106eb2d000)
  libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f106e927000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f106e707000)
  librt.so.1 => /lib64/librt.so.1 (0x00007f106e4fe000)
  liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f106e2d7000)
  liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f106e0c0000)
  libcap.so.2 => /lib64/libcap.so.2 (0x00007f106deba000)
  libmount.so.1 => /lib64/libmount.so.1 (0x00007f106dc60000)
  libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f106d944000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f106d72c000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f1071613000)
  libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f106d4a8000)
  libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f106d297000)
  libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f106d093000)
  libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f106ce41000)
  libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f106cc39000)
  libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f106ca18000)
#

Do not accept password logins

The only allowed authentication method should be Public-Key authentication

. Passwords are way easier to break, brute-force and this configuration limits the attack-vector.

Authentication methods in `sshd` config

AuthenticationMethods publickey
PasswordAuthentication no
PermitEmptyPasswords no

Do not accept forwarding

X11Forwarding is a Linux mechanism created to expose X11 into the remote shell. It is very difficult to imagine reasons for using graphical software on Linux server, per se. That being said, I do only see enabling this as another vector of possible attack, and risk to omit firewalls. Port forwarding is the functionality that allows the client to bind local-port to a remote address. It is a very straight and easy way to omit and skip firewall-rules and break compliance.

Explicitly disabled X11Forwarding and TcpForwarding

X11Forwarding no
AllowTcpForwarding no

Minimize MaxSessions and MaxAuthTries

The authorized user shouldn't have any difficulties with logging in successfully... Of course, there are exceptions to that statement, but in general, it is safe to assume it should be completely sufficient to have two, max three tries during one connection session. In contrast, the attacker trying to apply brute-force attacks needs as much login tries as possible. Therefore, I think we can all agree, minimizing MaxAuthTries is a good way of hardening. Another useful hardening trick, is to limit possible multiplexed sshd_sessions with MaxSessions.

Limiting the amount of max login tries and concurrent sessions

MaxAuthTries 2
MaxSessions 10

Add metadata information

This part combines the usage of two separate files: motd and banner. Both should be used accordingly. motd is the file that is displayed by Unix after a successful login, yet before executing the shell. Banner is a file that is visible for each user trying to connect before the authentication process.

Example of `motd` file

#Unauthorized access to this device is prohibited!#

Example of `banner` file

You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only.
By using this IS (which includes any device attached to this IS), you consent to the following conditions:

- The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations.
- At any time, the USG may inspect and seize data stored on this IS.
- Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose.
- This IS includes security measures (e.g. authentication and access controls) to protect USG interests — not for your personal benefit or privacy.
- Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details.

Do not use HostbasedAuthentication, rhosts

rhosts

refers to UNIX-based mechanism of allowing another computer on the same network to login. The rhosts file contains hosts and usernames that determine, who can login to a system remotely, without putting any password. It can contain positive and negative entries, which refer to either explicitly allowed access or contrary, explicitly denied access. HostbasedAuthentication allows to bypass authentication when you're logging in from declared host.

Example of `rhosts` file content

hamlet dewey

Disabling `hostauth` and `rhosts` authentication

HostbasedAuthentication no
IgnoreRhosts yes

Do not accept root login

It should not be permitted for the end-user to login to root directly. There are many reasons for that. For example, one is that it limits the auditing possibilities. It is way easier to audit and do the forensics when the user has to log in and then escalate his privileges to the superuser from there using tools like sudo.

Disable root-logins

PermitRootLogin no

Configure KeepAliveSession

Inactive sessions should be terminated. It limits persistence, therefore at least partially rendering the "persistence part" of ATT&CK

matrix harder to apply, it is also also a good lesson for forgetful users.

Terminate stale sessions

ClientAliveInterval 5m
ClientAliveCountMax 2

Use two-factor authentication

Since OpenSSH 6.3 it is possible to use two-factor authentication, and I would strongly recommend you to use it. There are many providers available, both software, like Google Authenticator, or hardware keys, like Yubikey or Google Security Key.

Use a strong set of algorithms

The default setting for the current encryption and hash algorithms is not really the best option out there. We can refer to the various standards like PCI DSS, FIPS-140-3. It is important to remember that the secure shell server encrypts the connection through the standard public-key-infrastructure. Said that, the same problems with Logjam attack still apply to this service

, so increasing moduli param in Diffie-Hellman algorithm is required.

Suggested set of Algorithms, ciphers and MACs

KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]

Protect server keys

Host keys should be accessible by the root user only. Once deployed, it is recommended that Host keys should be registered in an automated inventory system. Server keys should also be monitored by host intrusion detection systems, like Auditbeat

or AIDE.

Use port knocking

The paranoics can use port knocking

. It is an automated way to open external ports enacted by attempting to connect to a specified set of closed ports. When the Client applies the correct sequence in a specified timeframe, firewall rules are dynamically modified to allow the host to connect to a specific port.

Build a Certificate Authority

Even more paranoic... paranoics can create a full certificate authority for the whole inventory.

Attaching your own certificate authority

TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Client

Physical access to workstation

Hash known_hosts file

Hashing the known_hosts file prevents leakage and information disclosure. The following examples present non-hashed and hashed known_hosts files.

Non-hashed known_hosts content

255.255.255.255 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==

Hashed known_hosts content

|1|iAsVNM5h7KXr9LM59l6UXSdjLpk=|IWK/TOxygz+vQNt9R600PNG4/ts= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMH96GPuCQuE6JIWqmuzClWiecxwc7orVmisNPQyN0eW7K0Jj3kJ/GsuxPr3AujjiJPQHO3M8m4Oqd+/0p254/Q=

Protect private keys with passphrase

Private keys should be protected with a strong passphrase; they should never be copied from one system to another using disks, USB drives or tokens. They should have very, very strict permissions applied as well.

Umask configuration

umask 077
chmod 0400 ~/.ssh/*

Agent forwarding

Disable agent forwarding. Since OpenSSH 7.3 it is possible to jump through servers much, much more comfortably and in a way better automated manner using ProxyJump.

Disabling agent forwarding

Host *
    ForwardAgent no

Automation

The most important part of my work is automation. I try to automate everything. You can apply automation for security too - for example, like this:

An example of Lynis tests

# ./lynis audit system
[...]
[+] SSH Support
------------------------------------
  - Checking running SSH daemon                               [ FOUND ]
    - Searching SSH configuration                             [ FOUND ]
    - OpenSSH option: AllowTcpForwarding                      [ SUGGESTION ]
    - OpenSSH option: ClientAliveCountMax                     [ SUGGESTION ]
    - OpenSSH option: ClientAliveInterval                     [ OK ]
    - OpenSSH option: Compression                             [ SUGGESTION ]
    - OpenSSH option: FingerprintHash                         [ OK ]
    - OpenSSH option: GatewayPorts                            [ OK ]
    - OpenSSH option: IgnoreRhosts                            [ OK ]
    - OpenSSH option: LoginGraceTime                          [ OK ]
    - OpenSSH option: LogLevel                                [ SUGGESTION ]
    - OpenSSH option: MaxAuthTries                            [ SUGGESTION ]
    - OpenSSH option: MaxSessions                             [ SUGGESTION ]
    - OpenSSH option: PermitRootLogin                         [ SUGGESTION ]
    - OpenSSH option: PermitUserEnvironment                   [ OK ]
    - OpenSSH option: PermitTunnel                            [ OK ]
    - OpenSSH option: Port                                    [ SUGGESTION ]
    - OpenSSH option: PrintLastLog                            [ OK ]
    - OpenSSH option: StrictModes                             [ OK ]
    - OpenSSH option: TCPKeepAlive                            [ SUGGESTION ]
    - OpenSSH option: UseDNS                                  [ OK ]
    - OpenSSH option: X11Forwarding                           [ SUGGESTION ]
    - OpenSSH option: AllowAgentForwarding                    [ SUGGESTION ]
    - OpenSSH option: AllowUsers                              [ NOT FOUND ]
    - OpenSSH option: AllowGroups                             [ NOT FOUND ]
[...]

References

LinkedInLinkedInLinkedIn
Kamil Zabielski photo

About the author

Kamil Zabielski