20 years on AWS and never not my job

20 Years on AWS and Never Not My Job

I created my first AWS account at 10:31 PM on April 10th, 2006. I had seen the announcement of Amazon S3 and had been thinking vaguely about the problem of secure backups — even though I didn't start Tarsnap until several months later — and the idea of an online storage service appealed to me. The fact that it was a web service made it even more appealing; I had been building web services since 1998, when I decided that coordinating a world-record-setting computation of Pi over HTTP would be easier than doing it over email.

While I created my AWS account because I was interested in Amazon S3, that was not in fact immediately available to me: In the early days of AWS, you had to specifically ask for each new service to be enabled for your account. My new AWS account did come with two services enabled by default, though — Amazon Simple Queue Service, which most people know as "the first AWS service", and Amazon E-Commerce Service, an API which allowed Amazon affiliates to access Amazon.com's product catalogue — which was the real first AWS service, but which most people have never heard of and which has been quietly scrubbed from AWS history.

It didn't take long before I started complaining about things. By this point I was the FreeBSD Security Officer, so my first interest with anything in the cloud was security. AWS requests are signed with API keys providing both authentication and integrity protection — confirming not only that the user was authorized, but also that the request hadn't been tampered with. There is, however, no corresponding signature on AWS responses — and at this time it was still very common to make AWS requests over HTTP rather than HTTPS, so the possibility of response tampering was very real. I don't recall if anyone from Amazon showed any interest when I posted about this on the (long-disappeared) AWS Developer Forums, but I still think it would be a good thing to have: With requests going over TLS it is obviously less critical now, but end-to-end signing is always going to be better than transport-layer security.

Of course, as soon as Amazon EC2 launched I had a new target: I wanted to run FreeBSD on it! I reached out to Jeff Barr via his blog and he put me in touch with people inside Amazon, and in early 2007 I had my first Amazon NDA. (Funny story, in 2007 Amazon was still using fax machines — but I didn't have a fax machine, so my first briefing was delayed while I snail-mailed a wet-ink signature down to Seattle.) Among the features I was briefed on was "Custom Kernels"; much like how AWS Lambda works today, Amazon EC2 launched without any "bring your own kernel" support. Obviously, to bring FreeBSD support to EC2 I was going to need to use this functionality, and it launched in November 2007 when Amazon EC2 gained the ability to run Red Hat; soon after that announcement went out, my FreeBSD account was allowlisted for the internal "publish Amazon Kernel Images" API.

But I didn't wait for this functionality to be offered before providing more feedback about Amazon EC2. In March 2007 I expressed concerns to an Amazonian about the security of Xen — it was at the time still quite a new system and Amazon was the first to be deploying it in truly hostile environments — and encouraged them to hire someone to do a thorough security audit of the code. When the Amazonian I was speaking to admitted that they didn't know who to engage for this, I thought about the people I had worked with in my time as FreeBSD Security Officer and recommended Tavis Ormandy to them. Later that year, Tavis was credited with reporting two vulnerabilities in Xen (CVE-2007-1320 and CVE-2007-1321); whether there is any connection between those events, I do not know.

I also mentioned — in fact in one of Jeff Barr's AWS user meetups in Second Life — that I wanted a way for an EC2 instance to be launched with a read-only root disk and a guaranteed state wipe of all memory on reboot, in order to allow an instance to be "reset" into a known-good state; my intended use case for this was building FreeBSD packages, which inherently involves running untrusted (or at least not-very-trusted) code. The initial response from Amazonians was a bit confused (why not just mount the filesystem read-only) but when I explained that my concern was about defending against attackers who had local kernel exploits, they understood the use case. I was very excited when EC2 Instance Attestation launched 18 years later.

I ended 2007 with a blog post which I was told was quite widely read within Amazon: Amazon, Web Services, and Sesame Street. In that post, I complained about the problem of Eventual Consistency and argued for a marginally stronger model: Eventually Known Consistency, which still takes the "A" route out of the CAP theorem, but exposes enough internal state that users can also get "C" in the happy path. Amazon S3 eventually flipped from being optimized for Availability to being optimized for Consistency (while still having extremely high Availability), and of course DynamoDB is famous for giving users the choice between Eventual or Strongly consistent reads; but I still think the model of Eventually Known Consistency is the better theoretical model even if it is harder for users to reason about.

In early 2008, Kip Macy got FreeBSD working on Xen with PAE — while FreeBSD was one of the first operating systems to run on Xen, it didn't support PAE and I was at the time not competent to write such low-level kernel code, so despite being the driving force behind FreeBSD/EC2 efforts I had to rely on more experienced developers to write the kernel code at the time. I was perfectly comfortable with userland code though — so when Amazon sent me internal "AMI tools" code (necessary for using non-public APIs), I spent a couple weeks porting it to run on FreeBSD. Protip: While I'm generally a tools-not-policy guy, if you find yourself writing Ruby scripts which construct and run bash scripts, you might want to reconsider your choice of languages.

Unfortunately even once I got FreeBSD packaged up into an AKI (Amazon Kernel Image) and AMI (Amazon Machine Image) it wouldn't boot in EC2; after exchanging dozens of emails with Cape Town, we determined that this was due to EC2 using Xen 3.0, which had a bug preventing it from supporting recursive page tables — a cute optimization that FreeBSD's VM code used. The problem was fixed in Xen 3.1, but Xen didn't have stable ABIs at that point, so upgrading EC2 to run on Xen 3.1 would have broken existing AMIs; while it was unfortunate for FreeBSD, Amazon made the obvious choice here by sticking with Xen 3.0 in order to support existing customers.

In March 2008, I received one of those emails which only really seems notable in hindsight:

Hi Colin,

This is Matt Garman from the EC2 team at Amazon.  [...]

Matt was inviting me to join the private Alpha of "Elastic Block Storage" (now generally known as "Elastic Block Store" — I'm not sure if Matt got the name wrong or if the name changed). While I was excited about the new functionality, as I explained to Matt the best time to talk to me about a new service is before building it. I come from a background of mathematics and theory; I can provide far more useful feedback on a design document than from alpha-test access.

By April 2008 I had Tarsnap in private beta and I was working on its accounting code — using Amazon SimpleDB as a storage back-end to record usage and account balances. This of course meant that I had to read the API documentation and write code for signing SimpleDB requests — back then it was necessary, but I still write my own AWS interface code rather than using any of their SDKs — and a detail of the signing scheme caught my eye: The canonicalization scheme had collisions. I didn't have any contacts on the SimpleDB team — and Amazon did not at the time have any "report security issues here" contacts — so on May 1st I sent an email to Jeff Barr starting with the line "Could you forward this onto someone from the SimpleDB team?"

While the issue wasn't fixed until December, Amazon did a good job of handling this — and stayed in contact with me throughout. They asked me to review their proposed "signature version 2" scheme; fixed their documentation when I pointed out an ambiguity; corrected what I euphemistically referred to as a "very weird design decision"; and allowlisted my account so I could test my code (which I had written against their documentation) against their API back-end. (I wrote more about this in my blog post AWS signature version 1 is insecure.)

In June 2008 I noticed that NextToken values — returned by SimpleDB when a query returns too many results and then passed back to SimpleDB to get more results — were simply base64-encoded serialized Java objects. This was inherently poor security hygiene: Cookies like that should be encrypted (to avoid leaking internal details) and signed (to protect against tampering). I didn't know how robust Amazon's Java object deserializer was, but this seemed like something which could be a problem (and should have been fixed regardless, as a poor design decision even if not exploitable), so I reported it to one of the people I was now in contact with on the SimpleDB team... and heard nothing back. Six months later, when a (perhaps more security minded) engineer I had been working with on the signing issue said "let me know if you find more security problems; since we don't yet have a security response page up, just email me" I re-reported the same issue and he wrote it up internally. (Even after this I still never received any response, mind you.)

Later in 2008, after Tarsnap was in public beta (but before it had much traction) — and after considerable prompting from Jeff Barr — I considered the possibility of working for Amazon. I had a phone interview with Al Vermeulen and slightly too late learned an important lesson: In a 45 minute interview, spending 30 minutes debating the merits of exceptions with an author of The Elements of Java Style is probably not the best use of time. I still firmly believe that I was correct — exceptions are an inherently poor way of handling errors because they make it easier to write bugs which won't be immediately obvious on casual code inspection — but I also know that it isn't necessary to correct everyone who is wrong.

Finally in November 2008, I drove down to Seattle for an AWS Start-up Tour event and met Amazonians in person for the first time; for me, the highlight of the trip was meeting the engineer I had been working with on the request signing vulnerability. We had a lengthy discussion about security, and in particular my desire for constrained AWS access keys: I was concerned about keys granting access to an entire account and the exposure it would create if they were leaked. I argued for cryptographically derived keys (e.g. hashing the master secret with "service=SimpleDB" to get a SimpleDB-only access key) while he preferred a ruleset-based design, which was more flexible but concerned me on grounds of complexity. Ultimately, I was entirely unsurprised when I was invited to join a private beta of IAM in January 2010 — and also somewhat amused when SigV4 launched in 2012 using derived keys.

For most of 2009 I was busy with growing Tarsnap. The EC2 team set up some Xen 3.1 hosts for testing and by mid-January I was able to launch and SSH into FreeBSD; but since EC2 had no concrete plans to upgrade away from Xen 3.0, the FreeBSD/EC2 project as a whole was still blocked. I did however notice and report a problem with the EC2 firewall: The default ruleset blocked ICMP, including Destination Unreachable (Fragmentation Required) messages — thereby breaking Path MTU Discovery. In December 2009 a manager in EC2 agreed with my proposed solution (adding a rule to the default ruleset) and wrote "I'll let you know as soon as I have an implementation plan in place and am confident it will happen soon". This was ultimately fixed in 2012, soon after I raised the issue publicly.

By the start of 2010, with EC2 still stuck on an ancient version of Xen, I was starting to despair of ever getting FreeBSD running, so I turned to the next best option: NetBSD, which famously runs on anything. It only took me a week — and a few round trip emails to Cape Town to ask for console logs — to create a NetBSD AMI which could boot, mount its root filesystem, configure the network, and launch sshd. While Amazon was a bit wary about me announcing this publicly — they quite reasonably didn't want me to say anything which could be construed as making a promise on their behalf — they agreed that I could discuss the work with developers outside the NDA, and the NetBSD team were excited to hear about the progress... although a bit confused as to why Amazon was still using paravirtualized Xen rather than HVM.

The lack of HVM continued to be a sore point — especially as I knew EC2 provided Xen/HVM for Windows instances — but in July 2010 Amazon launched "Cluster Compute" instances which supported HVM even for "Linux" images. I wasn't able to boot FreeBSD on these immediately — while HVM solved the paging table problem, there were still driver issues to address — but this gave me some hope for progress, so when Matt Garman mentioned they were "thinking about" making HVM more broadly available I immediately wrote back to encourage such thoughts; by this point it was clear that PV was a technological dead end, and I didn't want Amazon to be stuck on the wrong technology for any longer than necessary.

The first real breakthrough however came with the launch of the new t1.micro instance type in September. While it wasn't publicly announced at the time, this new instance family ran on Xen 3.4.2 — which lacked the bug which made it impossible to run FreeBSD. By mid-November I was able to SSH into a FreeBSD/EC2 t1.micro instance, and on December 13, 2010, I announced that FreeBSD was now available for EC2 t1.micro instances.

Once I'd gotten that far, things suddenly got easier. Amazon now had customers using FreeBSD — and they wanted more FreeBSD. A Solutions Architect put me in touch with a FreeBSD user who wanted support for larger instances, and they paid me for the time it took to get FreeBSD working on Cluster Compute instances; then it was pointed out to me that EC2 didn't really know which OS we were running, and I proceeded to make FreeBSD available on all 64-bit instance types via defenestration. Obviously this meant paying the "windows tax" to run FreeBSD — which Amazon was not very happy about! — but even with the added cost it filled an essential customer need. (This hack finally ceased to be necessary in July 2014, when T2 filled out the stable of instance types which supported running "Linux" on HVM.)

2012 was an exciting year. In April, I had the classic greybeard experience of debugging a network fault; I found that a significant proportion of my S3 requests to a particular endpoint were failing with peculiar errors, including SignatureDoesNotMatch failures. These error responses from Amazon S3 helpfully contained the StringToSign, and I could see that these did not match what I was sending to S3. I had enough errors to identify the error as a "stuck bit"; so I pulled out traceroute — this was pre-SRD so my packets were traversing a consistent path across the datacenter — and then proceeded to send a few million pings to each host along the path. The Amazonians on the AWS Developer Forums were somewhat bemused when I posted to report that a specific router had a hardware failure... and even more surprised when they were able to confirm the failure and replace the faulty hardware a few days later.

The highlight of 2012 however was the first re:Invent — which was short of technical content and had a horrible tshirt-to-suit ratio, but did give me the opportunity to talk to a number of Amazonians face to face. On one memorable occasion, after attending an Intel talk about "virtual machine security" (delivered by a VP who, in response to my questioning, professed to have no knowledge of "side channel attacks" or how they could affect virtual machines) I turned up at the EC2 booth in the expo hall to rant... and by complete accident ended up talking to a Principal engineer. I talked about my work exploiting HyperThreading to steal RSA keys, and explained that, while the precise exploit I'd found had been patched, I was absolutely certain there were many more ways that information could leak between two threads sharing a core. I ended with a strong recommendation: Based on my expertise in the field I would never run two EC2 instances in parallel on two threads of the same core. Years later, I was told that this recommendation was why so many EC2 instance families jumped straight to two vCPUs ("large") and skipped the "medium" size.

Time passed. With FreeBSD fundamentally working, I turned to the "nice to haves": merging my FreeBSD patches, simplifying the security update path (including automatically installing updates on first boot), and resizing the root filesystem on first boot. In April 2015, I finished integrating the FreeBSD/EC2 AMI build process into the FreeBSD src tree and handed off image builds to the FreeBSD release engineering team — moving FreeBSD/EC2 across the symbolic threshold from a "Colin" project to "official FreeBSD". I was still the de facto owner of the platform, mind you — but at least I wasn't responsible for running all of the builds.

In October 2016, I took a closer look at IAM Roles for Amazon EC2, which had launched in mid-2012. The more I thought about it, the more concerned I got; exposing credentials via the IMDS — an interface which runs over unauthenticated HTTP and which warned in its documentation against storing "sensitive data, such as passwords" — seemed like a recipe for accidental foot-shooting. I wrote a blog post "EC2's most dangerous feature" raising this concern (and others, such as overly broad IAM policies), but saw no response from Amazon... that is, not until July 2019, when Capital One was breached by exploiting the precise risk I had described, resulting in 106 million customers' information being stolen. In November 2019, I had a phone call with an Amazon engineer to discuss their plans for addressing the issue, and two weeks later, IMDSv2 launched — a useful improvement (especially given the urgency after the Capital One breach) but in my view just a mitigation of one particular exploit path rather than addressing the fundamental problem that credentials were being exposed via an interface which was entirely unsuitable for that purpose.

In May 2019, I was invited to join the AWS Heroes program, which recognizes non-Amazonians who make significant contributions to AWS. (The running joke among Heroes is that a Hero is someone who works for Amazon but doesn't get paid by Amazon.) The program is heavily weighted towards people who help developers learn how to use AWS (via blog posts, YouTube videos, workshops, et cetera), so I was something of an outlier; indeed, I was told that when I was nominated they weren't quite sure what to make of me, but since I had been nominated by a Distinguished Engineer and a Senior Principal Engineer, they felt they couldn't say no.

In March 2021, EC2 added support for booting x86 instances using UEFI; a "BootMode" parameter could be specified while registering an image to declare whether it should be booted using legacy BIOS or modern UEFI. For FreeBSD this was great news: Switching to UEFI mode dramatically sped up the boot process — performing loader I/O in 16-bit mode required bouncing data through a small buffer and cost us an extra 7 seconds of boot time. The only problem was that while all x86 instance types supported legacy BIOS booting, not all instance types supported UEFI — so I had to decide whether to degrade the experience for a small number of users to provide a significant speedup to most users. In June, I requested a BootMode=polyglot setting which would indicate that the image was able to boot either way (which, in fact, FreeBSD images already could) and instructed EC2 to pick the appropriate boot mode based on the instance. In March 2023, this landed as "BootMode=uefi-preferred", which I had to admit was a friendlier, albeit less geeky, name for it.

One of the most important things about the AWS Heroes program is the briefings Heroes get, especially at the annual "Heroes Summit". In August 2023, we had a presentation about Seekable OCI, and looking at the design I said to myself "hold on, they're missing something here": The speaker made security claims which were true under most circumstances, but did not hold in one particular use case. I wrote to the AWS Security team (unlike in 2008, there was now a well-staffed team with clear instructions on how to get in touch) saying, in part, "I'm not sure if this is them not understanding about [type of attack] or if it's just an issue of confused marketing, but I feel like someone needs to have a conversation with them". My sense was that this could probably be addressed with clear documentation saying "don't do this really weird thing which you probably weren't planning on doing anyway", but since I wasn't particularly familiar with the service I didn't want to make assumptions about how it was being used. After a few email round trips I was assured that the problem had been corrected internally and that the fix would be merged to the public GitHub repository soon. I accepted these assurances — over the years I've developed a good relationship with AWS Security people and trust them to handle such matters — and put it out of my mind.

In December 2023, however, I was talking to some Amazonians at re:Invent and was reminded of the issue. I hadn't heard anything further, which surprised me given that fixing this in code (rather than in documentation) would be fairly intrusive. I asked them to check up on the issue and they promised to report back to me in January, but they never did, and again I stopped thinking about it. The following re:Invent though, in December 2024, I met a Principal Engineer working on OCI and mentioned the issue to him — "hey, whatever happened with this issue?" — but he wasn't aware of it. In January 2025, I raised it again with a Security Engineer; he found the original ticket from 2023 and talked to the team, who pointed at a git commit which they thought fixed it.

The issue had not, in fact, been fixed: The 2023 commit prevented the problem from being triggered by accidental data corruption, but did nothing to prevent a deliberate attack. Once I pointed this out, things got moving quickly; I had a Zoom call with the engineering team a few days later, and by the end of February the problematic feature had been disabled for most customers pending a "major revision".

The largest change in my 20 years of working with Amazon started out as something entirely internal to FreeBSD. In September 2020, the FreeBSD Release Engineering Lead, Glen Barber, asked me if I could take on the role of Deputy Release Engineer — in other words, Hot Spare Release Engineer. As the owner of the FreeBSD/EC2 platform, I had been working with the Release Engineering team for many years, and Glen felt that I was the ideal candidate: reliable, trusted within the project, and familiar enough with release engineering processes to take over if he should happen to "get hit by a bus". While I made a point of learning as much as I could about how Glen managed FreeBSD releases, like most hot spares I never expected to be promoted.

Unfortunately, in late 2022 Glen was hospitalized with pneumonia, and while he recovered enough to leave the hospital a few months later, it became clear that the long-term effects of his hospitalization made it inadvisable for him to continue as release engineer; so on November 17, 2023, Glen decided to step back from the role and I took over as FreeBSD Release Engineering Lead. I like to think that I've done a good job since then — running weekly snapshot builds, tightening schedules, establishing a predictable and more rapid release cadence, and managing four releases a year — but my volunteer hours weren't unlimited, and it became clear that my release engineering commitments were making it impossible to keep up with EC2 support as well as I would have liked.

In April 2024 I confided in an Amazonian that I was "not really doing a good job of owning FreeBSD/EC2 right now" and asked if he could find some funding to support my work, on the theory that at a certain point time and dollars are fungible. He set to work, and within a couple weeks the core details had been sorted out; I received sponsorship from Amazon via GitHub Sponsors for 10 hours per week for a year and addressed a large number of outstanding issues. After a six month hiatus — most of which I spent working full time, unpaid, on FreeBSD 15.0 release engineering — I've now started a second 12-month term of sponsorship.

While I like to think that I've made important contributions to AWS over the past 20 years, it's important to note that this is by no means my work alone. I've had to remind Amazonians on occasion that I do not have direct access to internal AWS systems, but several Amazonians have stepped in as "remote hands" to file tickets, find internal contacts, inspect API logs, and obtain technical documentation for me. Even when people — including very senior engineers — have explicitly offered to help, I'm conscious of their time and call upon them as little as I can; but the fact is that I would not have been able to do even a fraction of what I've accomplished without their help.

blog comments powered by Disqus