This post was originally published by me on binx.io, it has been slightly adapted to better fit my personal site.
Of all the sessions I attended at re:Invent 2019, the one I found most interesting was about the Security Benefits of the Nitro architecture.
Although it might sound weird to hear a Cloud Consultant discuss hardware details, what makes it worth going down to this level is, that the Nitro architecture showcases in what ways AWS is innovating in the field of virtualization. I guess that because of this, Werner himself highlighted these features in his keynote.
Privileged Virtual Machines
The Xen hypervisor, which AWS originally used and that still powers some EC2 instance classes, has the concept of a dom0. Each host has one privileged virtual machine, dom0, that is allowed to interact with real hardware; this allows the hypervisor to remain small by offloading work to dom0.
This privileged virtual machine runs a Linux operating system that manages (disk and network) devices and exposes them to the non-privileged guest virtual machines. In dom0 AWS runs, among others, services supporting EBS, VPC networking, and CloudWatch.
The privileged dom0 virtual machine has a large attack surface as it shares hardware with the guest virtual machines and runs a fully-fledged operating system.
The Nitro System
What AWS calls the Nitro system is a collection of custom build devices that take most of the work that normally happens in dom0 to support the virtual machines. Not only does offloading this work to the Nitro system leave more capacity for the guests (about 10% of EC2 host resources are regained), it also makes everything much more secure.
It consists of a couple of PCI-card like devices, in addition to this a special security chip is embedded on the motherboard. While implementing the Nitro system, AWS paid attention to a new security paradigm.
(Image taken from the re:Inforce 2019 slides of this talk.)
New Security Paradigm
When building the Nitro system, AWS paid special attention to the following security precautions:
- where the Nitro components run an operating system, remote login (SSH) is disabled
- the Nitro components are interconnected using a private network
- only the Nitro components are allowed connect to the AWS network, direct access to this network is not allowed from the bare-metal and guest virtual machines
- the Nitro components and hypervisor are controlled by sending requests to their APIs, never will these components initiate outgoing traffic themselves; outgoing traffic is a sign of trouble
- the Nitro components are live-updatable, with barely noticeable impact for the virtual machines
PCI-card like Nitro System Devices
The functionality of the Nitro system is provided by various PCI-card like devices. This design is inspired by microservice software architectures.
Some types of devices include:
- Nitro Controller
- Nitro EBS subsystem
- Nitro Ephemeral storage subsystem
- Nitro VPC subsystem
Nitro Controller
The Nitro controller manages the host, hypervisor, and the other Nitro devices in the system.
The Nitro controller:
- validates the motherboard’s firmware(s) against known-good cryptographic hashes
- manages the (de)provisioning of EBS and Local Storage volumes and ENI devices
- manages the hypervisor to (de)provision virtual machines
- manages the hypervisor to attach/detach block and network devices to/from virtual machines
- gathers metrics for CloudWatch
Nitro EBS subsystem
The Nitro EBS subsystem provides EBS volumes as NVMe devices to the motherboard.
All encryption happens on the Nitro device, encryption keys are managed by KMS, encryption keys can not be accessed by the EC2 host.
Nitro Ephemeral Storage subsystem
The Nitro local storage subsystem provides ephemeral volumes as NVMe devices to the motherboard.
All encryption happens on the Nitro device, encryption keys are generated on, and never leave, the device.
After use, the volumes are cryptographically wiped, i.e., the keys are removed, the actual data in encrypted form remains on the physical disks. Before use, the first and last bytes of the volumes are zeroed to prevent confusing the guest operating system (as the device would still be filled with “random” data from previous uses).
Nitro VPC subsystem
The Nitro VPC subsystem provides network interface controller devices to the motherboard.
The VPC stack runs on the Nitro system; only the Nitro system has access to the private AWS network, the EC2 host and guests can only access the network via the Nitro system.
All traffic between Nitro powered instances is transparently encrypted on the Nitro system, traffic to non-Nitro instances is not encrypted as this would impact the performance.
Security Chip Embedded on the Motherboard
The Nitro security chip guards the firmware of the motherboard in two ways:
- it denies any updates of the motherboard firmware
- it pauses the motherboard boot process at an early stage to allow the firmware to be cryptographically checked by the Nitro Controller
What We Learned from this re:Invent Session
The Security Benefits of the Nitro architecture session taught us:
- about 10% of host performance is regained by offloading work to the Nitro system
- the Nitro system is modelled after microservice architectures
- the Nitro system improves security by isolating management and encryption from the guest machines
- the Nitro system improves security by adopting a new paradigm