It is 2020, and we can say for sure, that ssh-server
is still one of the most popular services on Linux systems. During various meetings, I was often asked: How to secure SSH? Even though this question seems trivial, it is not. There are a lot of things to remember to accomplish well-designed service security. Secure shell is used not only for a remote-shell, per se. Many other technologies depend on it, for various reasons.I have decided to answer the above question once and forever, and maintain the freshness of this answer, so that you can always refer to this document, as an up-to-date, and consistent with current security practices, knowledgebase. This post is an comprehensive document for infosec and devops teams, that will guide them through all known best-practices and hardening rules.
Contemplate the needs
The first and the most basic question the Reader should ask himself: Is there even a need for an OpenSSH server at all, for my specific use-case? At the time of writing this article, it is not uncommon to build fully declarative, golden-image infrastructures. We have a wide set of serverless services and technologies that only run a process inside the container. I would say, for sure, that the majority of the infrastructures we, as @sysdogs, provision for our customers, do not have OpenSSH enabled at all. (You can always contact us, if you want to know, how we do it.) Another perfect example for disabled ssh-server is a local workstation. There is practically no purpose in a secure-shell server on a local laptop.
Server
Management interfaces only
While reading dedicated instructions and resources, like Security Technical Implementation Guidelines, you'll notice that they always mark access management as the most crucial part of the process, and top priority. And they are right. It is very, very important to bind the server only to the interface that is trusted.
- In Enterprise infrastructures, virtual machines can have their own management interfaces, with proper firewalld rules enacted.
- For virtual private servers, we can align this rule to the trusted subset of public addresses that are whitelisted in the firewall.
- In the Cloud environments, we can either build a specific security group that defines the access to the server, or use other cloud services, like DigitalOcean Firewall, Cloudflare, or any other.
Example set of firewall security rules
firewall-cmd --set-default-zone=drop
firewall-cmd --zone=trusted --change-interface=eth1 --permanent
firewall-cmd --zone=trusted --add-service=ssh --permanent
Mandatory Access Control - SELinux
I am the guy, that would do setenforce 1
even if it was already set to 1. Just in case. To be sure. Being absolutely honest, I strongly believe SELinux is one of the most helpful services in the Linux world, when you want to improve and harden your operating system.
I've read a lot of documentation and tutorials that recommend the end-user to disable SELinux or AppArmor. I completely understand their authors. They told you to do that, because it simplifies the process of software installation and fades out the burden of hardening. On the other hand, I fully believe it is like getting your house alarm wrongly configured, getting upset because of the false-positives, then disabling it, just to feel fine with yourself.
Checking SELinux status
# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 31
#
Checking TCP:22 port status
# netstat -pnltuZ | grep ":22"
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 696/sshd system_u:system_r:sshd_t:s0-s0:c0.c1023
tcp6 0 0 :::22 :::* LISTEN 696/sshd system_u:system_r:sshd_t:s0-s0:c0.c1023
#
Use Protocol 2
To be honest, I have never seen any secure shell server with enabled Protocol 1
, or at least, never seen such one after i've done my work on it. To maintain the state of the art and comprehensiveness of this article, it is important to note it here, that it is insecure and it is not recommended to use Protocol 1
. It has a set of vulnerabilities and problems with security. Protocol 1
support was removed in RHEL7.4
Protocol setting in `sshd` config
Protocol 2
Use tcp/22
I have seen a lot of write-ups that recommend using a different port than tcp/22
Ports configuration in `sshd` config
Port 22
Checking `tcp/22` port status
# ss -pnltu | grep 22
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=696,fd=4))
tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=696,fd=6))
#
Do not use tcpwrapperd
A set of known Linux websites still recommend using tcpwrapperd
as another security layer for sshd
The very first argument for that, is that tcpwrapperd
is mostly deprecated technology
tcpwrapperd
functionalities. In the end, fresh operating systems do not link libwrapd
to sshd
.`sshd` dependency list
# ldd /sbin/sshd
linux-vdso.so.1 (0x00007fff47b1a000)
libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007f107113a000)
libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f1070f10000)
libpam.so.0 => /lib64/libpam.so.0 (0x00007f1070d00000)
libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f10709bc000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f10704d9000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f10702d5000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f10700d1000)
libz.so.1 => /lib64/libz.so.1 (0x00007f106feba000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f106fc91000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f106fa7a000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f106f84f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f106f5ff000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f106f30f000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f106f0f3000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f106eeef000)
libc.so.6 => /lib64/libc.so.6 (0x00007f106eb2d000)
libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f106e927000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f106e707000)
librt.so.1 => /lib64/librt.so.1 (0x00007f106e4fe000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f106e2d7000)
liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f106e0c0000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007f106deba000)
libmount.so.1 => /lib64/libmount.so.1 (0x00007f106dc60000)
libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f106d944000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f106d72c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1071613000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f106d4a8000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f106d297000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f106d093000)
libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f106ce41000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f106cc39000)
libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f106ca18000)
#
Do not accept password logins
The only allowed authentication method should be Public-Key authentication
. Passwords are way easier to break, brute-force and this configuration limits the attack-vector.Authentication methods in `sshd` config
AuthenticationMethods publickey
PasswordAuthentication no
PermitEmptyPasswords no
Do not accept forwarding
X11Forwarding
is a Linux mechanism created to expose X11
into the remote shell. It is very difficult to imagine reasons for using graphical software on Linux server, per se. That being said, I do only see enabling this as another vector of possible attack, and risk to omit firewalls. Port forwarding is the functionality that allows the client to bind local-port to a remote address. It is a very straight and easy way to omit and skip firewall-rules and break compliance.
Explicitly disabled X11Forwarding and TcpForwarding
X11Forwarding no
AllowTcpForwarding no
Minimize MaxSessions
and MaxAuthTries
The authorized user shouldn't have any difficulties with logging in successfully... Of course, there are exceptions to that statement, but in general, it is safe to assume it should be completely sufficient to have two, max three tries during one connection session. In contrast, the attacker trying to apply brute-force attacks needs as much login tries as possible. Therefore, I think we can all agree, minimizing MaxAuthTries
is a good way of hardening. Another useful hardening trick, is to limit possible multiplexed sshd_sessions
with MaxSessions
.
Limiting the amount of max login tries and concurrent sessions
MaxAuthTries 2
MaxSessions 10
Add metadata information
This part combines the usage of two separate files: motd
and banner. Both should be used accordingly. motd
is the file that is displayed by Unix after a successful login, yet before executing the shell. Banner is a file that is visible for each user trying to connect before the authentication process.
Example of `motd` file
#Unauthorized access to this device is prohibited!#
Example of `banner` file
You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only.
By using this IS (which includes any device attached to this IS), you consent to the following conditions:
- The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations.
- At any time, the USG may inspect and seize data stored on this IS.
- Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose.
- This IS includes security measures (e.g. authentication and access controls) to protect USG interests — not for your personal benefit or privacy.
- Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details.
Do not use HostbasedAuthentication
, rhosts
rhosts
UNIX
-based mechanism of allowing another computer on the same network to login. The rhosts
file contains hosts and usernames that determine, who can login to a system remotely, without putting any password. It can contain positive and negative entries, which refer to either explicitly allowed access or contrary, explicitly denied access. HostbasedAuthentication
allows to bypass authentication when you're logging in from declared host.Example of `rhosts` file content
hamlet dewey
Disabling `hostauth` and `rhosts` authentication
HostbasedAuthentication no
IgnoreRhosts yes
Do not accept root login
It should not be permitted for the end-user to login to root directly. There are many reasons for that. For example, one is that it limits the auditing possibilities. It is way easier to audit and do the forensics when the user has to log in and then escalate his privileges to the superuser from there using tools like sudo
.
Disable root-logins
PermitRootLogin no
Configure KeepAliveSession
Inactive sessions should be terminated. It limits persistence, therefore at least partially rendering the "persistence part" of ATT&CK
Terminate stale sessions
ClientAliveInterval 5m
ClientAliveCountMax 2
Use two-factor authentication
Since OpenSSH 6.3 it is possible to use two-factor authentication, and I would strongly recommend you to use it. There are many providers available, both software, like Google Authenticator, or hardware keys, like Yubikey or Google Security Key.
Use a strong set of algorithms
The default setting for the current encryption and hash algorithms is not really the best option out there. We can refer to the various standards like PCI DSS, FIPS-140-3. It is important to remember that the secure shell server encrypts the connection through the standard public-key-infrastructure. Said that, the same problems with Logjam attack still apply to this service
, so increasing moduli param in Diffie-Hellman algorithm is required.Suggested set of Algorithms, ciphers and MACs
KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
Protect server keys
Host keys should be accessible by the root user only. Once deployed, it is recommended that Host keys should be registered in an automated inventory system. Server keys should also be monitored by host intrusion detection systems, like Auditbeat
or AIDE.Use port knocking
The paranoics can use port knocking
. It is an automated way to open external ports enacted by attempting to connect to a specified set of closed ports. When the Client applies the correct sequence in a specified timeframe, firewall rules are dynamically modified to allow the host to connect to a specific port.Build a Certificate Authority
Even more paranoic... paranoics can create a full certificate authority for the whole inventory.
Attaching your own certificate authority
TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
Client
Hash known_hosts
file
Hashing the known_hosts
file prevents leakage and information disclosure. The following examples present non-hashed and hashed known_hosts
files.
Non-hashed known_hosts content
255.255.255.255 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
Hashed known_hosts content
|1|iAsVNM5h7KXr9LM59l6UXSdjLpk=|IWK/TOxygz+vQNt9R600PNG4/ts= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMH96GPuCQuE6JIWqmuzClWiecxwc7orVmisNPQyN0eW7K0Jj3kJ/GsuxPr3AujjiJPQHO3M8m4Oqd+/0p254/Q=
Protect private keys with passphrase
Private keys should be protected with a strong passphrase; they should never be copied from one system to another using disks, USB drives or tokens. They should have very, very strict permissions applied as well.
Umask configuration
umask 077
chmod 0400 ~/.ssh/*
Agent forwarding
Disable agent forwarding. Since OpenSSH 7.3
it is possible to jump through servers much, much more comfortably and in a way better automated manner using ProxyJump
.
Disabling agent forwarding
Host *
ForwardAgent no
Automation
The most important part of my work is automation. I try to automate everything. You can apply automation for security too - for example, like this:
An example of Lynis tests
# ./lynis audit system
[...]
[+] SSH Support
------------------------------------
- Checking running SSH daemon [ FOUND ]
- Searching SSH configuration [ FOUND ]
- OpenSSH option: AllowTcpForwarding [ SUGGESTION ]
- OpenSSH option: ClientAliveCountMax [ SUGGESTION ]
- OpenSSH option: ClientAliveInterval [ OK ]
- OpenSSH option: Compression [ SUGGESTION ]
- OpenSSH option: FingerprintHash [ OK ]
- OpenSSH option: GatewayPorts [ OK ]
- OpenSSH option: IgnoreRhosts [ OK ]
- OpenSSH option: LoginGraceTime [ OK ]
- OpenSSH option: LogLevel [ SUGGESTION ]
- OpenSSH option: MaxAuthTries [ SUGGESTION ]
- OpenSSH option: MaxSessions [ SUGGESTION ]
- OpenSSH option: PermitRootLogin [ SUGGESTION ]
- OpenSSH option: PermitUserEnvironment [ OK ]
- OpenSSH option: PermitTunnel [ OK ]
- OpenSSH option: Port [ SUGGESTION ]
- OpenSSH option: PrintLastLog [ OK ]
- OpenSSH option: StrictModes [ OK ]
- OpenSSH option: TCPKeepAlive [ SUGGESTION ]
- OpenSSH option: UseDNS [ OK ]
- OpenSSH option: X11Forwarding [ SUGGESTION ]
- OpenSSH option: AllowAgentForwarding [ SUGGESTION ]
- OpenSSH option: AllowUsers [ NOT FOUND ]
- OpenSSH option: AllowGroups [ NOT FOUND ]
[...]
References
- SSH - Official Website. (n. d.). https://www.ssh.com/ssh/ (accessed August 26, 2020).
- The server-side SSH-1 protocol removal from RHEL 7.4 - RedHat Official Website. (n. d.). https://access.redhat.com/articles/3022681 (accessed August 26, 2020).
- Argument for moving SSH off port 22 - Isc.sans.edu. (n. d.). https://isc.sans.edu/forums/diary/The+argument+for+moving+SSH+off+port+22/19143/ (accessed August 26, 2020).
- How and why to change default SSH port - Linuxhint. (n. d.). https://linuxhint.com/change_default_ssh_port/ (accessed August 26, 2020).
- TCP Wrapper for added security on SSH - Akadia. (n. d.). https://www.akadia.com/services/ssh_tcp_wrapper.html (accessed August 26, 2020).
- How to secure network services using TCP Wrappers in Linux - Tecmint. (n. d.). https://www.tecmint.com/secure-linux-tcp-wrappers-hosts-allow-deny-restrict-access (accessed August 26, 2020).
- Fedora Project - Deprecate TCP Wrappers. (n. d.). https://fedoraproject.org/wiki/Changes/Deprecate_TCP_wrappers (accessed August 26, 2020).
- Deprecate TCP Wrappers - RedHat Issue Tracker. (n. d.). https://bugzilla.redhat.com/show_bug.cgi?id=1495181 (accessed August 26, 2020).
- Why authentication using SSH Public Key is better than using password, and how do they work? - Runcloud Blog. (n. d.). https://blog.runcloud.io/why-authentication-using-ssh-public-key-is-better-than-using-password-and-how-do-they-work (accessed August 26, 2020).
- What is rhosts? - Lifewire. (n. d.). https://www.lifewire.com/what-is-rhosts-2195896 (accessed August 26, 2020).
- MITRE Att&ck - Official Website. (n. d.). https://attack.mitre.org (accessed August 26, 2020).
- Weak Diffie-Hellman and the Logjam Attack - Official Website. (n. d.). https://weakdh.org (accessed August 26, 2020).
- Elastic Auditbeat - Official Website. (n. d.). https://www.elastic.co/beats/auditbeat (accessed August 26, 2020).
- AIDE on Github. (n. d.). https://aide.github.io (accessed August 26, 2020).
- Port Knocking - Wikipedia. (n. d.). https://en.wikipedia.org/wiki/Port_knocking (accessed August 26, 2020).