Avoiding Header Method Override Cache Poisoning
Configuring CloudFront in front of Google Cloud Storage
On April 15, 2021, a security researcher reached out to Pendo’s security team regarding a potential vulnerability in cdn.pendo.io. The proof of concept used the HTTP header X-HTTP-Method-Override to poison the cache, causing the cache to return a blank page to future clients for the lifetime of the cache entry.
A security researcher, who has asked to remain anonymous, described the issue as they observed it (personal communication, April 15, 2021):
“When sending a request to any asset located on https://cdn.pendo.io/ that includes an HTTP header X-HTTP-Method-Override with the value HEAD, the request gets passed to the back-end (unless the cache prevents it) and then triggers a response based on the value of that header which gets stored in the cache. By pointing a request to any endpoint located on https://cdn.pendo.io/ and providing that header an attacker can store an invalid response in the cache that has no body, which in the browser will be seen as a completely blank page.”
What is a CDN?
First, let’s start by explaining what a Content Delivery Network (CDN) is, and why it is a useful resource for improving performance for Pendo’s software.
A Content Delivery Network (CDN) is a globally distributed network of proxy servers that allows a company like Pendo to provide highly available web-based services that are distributed in such a way to be geographically closer to (and thus faster to retrieve data for) the service’s end users. Using a CDN is a crucial step in delivering Pendo to its customers as quickly and efficiently as possible.
Companies use CDNs for many things. The main use is to store a cached version of some of the resources of a website’s content in multiple geographic locations all over the world. This improves the retrieval of that data based on the distance the server hosting the cached version is from the requestor. The closer in physical location the data is, the less distance it has to travel across the network to arrive in the requestor’s browser. CDNs can also be used to deliver dynamic content unique to the requestor as well, but that’s beyond the scope of this post.
What is a cache?
A cache stores data so that future requests for that data can be more quickly retrieved. The cache could be a query or computation that has been previously executed, or it could be data that has been retrieved in a previous request from another source. Caches may be very small, but can help to improve the performance of a website significantly.
Cache poisoning
Cache poisoning attacks occur when an attacker exploits the behavior of a vulnerable web server and cache to serve a malicious HTTP response in such a way that the HTTP response is served up to other users. The malicious HTTP response can, in some cases, lead to XSS vulnerabilities, JavaScript injection attacks, open redirects, and many other vulnerabilities. (However, in this case, the attack was limited to returning blank output; an attacker was not afforded any opportunity to add malicious data.)
In order to execute a cache poisoning attack, an attacker must first identify an HTTP response from the back-end server that can be evoked and that will contain some type of dangerous payload. Then, the attacker must ensure that the HTTP response containing the dangerous payload will in fact be cached and served to other users.
For more information on cache poisoning, visit Portswigger’s Web Security Academy tutorial on web cache poisoning.
HTTP Header Override Cache Poisoning
Now that we have established what CDN, cache, and cache poisoning attacks are, let’s dig into the specific security vulnerability that our gracious security researcher submitted.
The researcher chose a random asset that they believed was not regularly used, so that they did not need to wait for the cache to expire. They thus expected this asset not to be currently cached, which means that the cache is ready to store a new HTTP response in its buffer. The researcher then constructed an HTTP request with the header X-HTTP-Method-Override: HEAD. After that request succeeded, the researcher confirmed that a normal GET without the X-HTTP-Method-Override header returned an empty body. If any Pendo customer had been actively using this asset, they would have received an empty body until the cache expired.
Below is the HTTP request the researcher submitted as a proof of concept:
GET /agent/releases/2.58.0/pendo.preview.min.js HTTP/1.1 Host: cdn.pendo.io User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0 Accept: */* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate x-http-method-override: HEAD Connection: close
Because this type of attack denies access to the intended content to the intended users, it is known more formally as a Cache Poisoned Denial of Service (or CPDoS). Such an attack can render resources stored in the cache completely unavailable for users until the cache has been cleared. For services that require high availability, this could cause significant frustration for its customers.
For more information on X-HTTP-Method-Override, check out this documentation from IBM.
Why Pendo was Vulnerable to CPDoS
Pendo uses Google Cloud Storage (GCS) for storage, and Amazon’s CloudFront as our primary CDN for the majority of our customers. CloudFront forwarded the malicious header to the source (GCS, in this case) as-is. GCS, which honors the X-HTTP-Method-Override header in order to support very old web browsers, responded with the correct result for a HEAD response, which provides only the metadata about the resource with an empty body. The CloudFront CDN then cached that empty response, and would provide the empty body in response to a normal GET request without the malicious X-HTTP-Method-Override header until the cache expired several minutes later.
Where was the breakdown? Amazon and Google both have resolved X-HTTP-Method-Override CPDoS attacks within their own cloud products. However, they have chosen different mechanisms to protect against it. Amazon has chosen to address this by not supporting X-HTTP-Method-Override in their S3 storage service, so CloudFront caches are not poisoned by providing this header in requests for assets stored in S3. Google, by contrast, does support X-HTTP-Method-Override in GCS, and instead resolved the vulnerability within Google Cloud CDN. Both mechanisms effectively prevent a CPDoS attack based on X-HTTP-Method-Override when only one cloud provider is being used. Therefore, either using Google Cloud CDN to cache assets stored in GCS, or CloudFront to cache assets stored in S3, will not be subject to X-HTTP-Method-Override CPDoS. However, in a hybrid cloud deployment like Pendo’s, the different defenses left a hole.
The Solution
Pendo’s Security and Operations teams, with assistance from AWS support, worked together to quickly find a solution to this problem. Amazon suggested that Pendo should use AWS WAF (Web Application Firewall) or Lambda@Edge to block any request that contains a non-standard header, such as X-HTTP-Method-Override, or remove the header from the request. A WAF would incur additional costs, as would a Lambda@Edge function, but the Lambda@Edge costs were likely lower.
We estimated that our costs to implement a Lambda@Edge function would be in the ballpark of $0.60 per 1M requests + $0.00005001 for every GB-second. Our Operations team tested the Lambda function 4 times, at an average runtime of 3.6ms, which came out to .002 GB-seconds. On the high side, we estimate that we would have 2M cache misses per day; thus, the total cost would be approximately $1.25 per day.
Below is the Lambda@Edge function that we successfully tested:
def lambda_handler(event, context):
request = event['Records'][0]['cf']['request']
request['headers'].pop('x-http-method-override', None)
return request
However, while testing this, we explored another option which allowed us to entirely ignore the header for the purposes of caching. We used the AWS “whitelist” feature to include the X-HTTP-Method-Override header in the basis for identifying cached data. With that in place, the 0-length content returned for X-HTTP-Method-Override: HEAD is not used to answer a normal request, defeating the cache poisoning attack. This is the better solution for our case, primarily because it does not incur any additional latency, but also because it is entirely free of additional cost. We used the following steps to “whitelist” the header:
-
- Sign in to the AWS Management Console and navigate to the CloudFront service.
- Select the ID of a distribution.
- To edit the default behavior settings, choose the Behaviors tab, select the behavior and click on edit. This will need to be done for each Behavior (path) defined.
- On the behavior settings page, scroll down to “Cache Based on Selected Request Headers” and select “Whitelist” from the dropdown.
- In the ‘Whitelist header’ textbox, enter X-HTTP-Method-Override and click on the “Add Custom >>” button.
- Click ‘Yes, edit’ at the bottom of the page and wait till the changes are propagated across all edges.
When a header is whitelisted in a distribution’s cache policy or origin request policy, CloudFront ensures that a different cache key is generated for each combination of the headers. Therefore, a request for an object with the X-HTTP-Method-Override header and without the header would be treated as different content and cached separately.
Our product never uses X-HTTP-Method-Override, so it does not matter if a request providing X-HTTP-Method-Override: HEAD returns a zero-length body. The only problem was with the side effect of caching that zero-length body and providing it to normal users. This use of “whitelist” prevents an attacker from causing the zero-length content to be supplied to normal users. The fact that the attacker still sees a zero-length response when they try to create a denial-of-service attack does not hurt Pendo’s customers at all.
Final Thoughts
Because CloudFront is widely used in front of hybrid environments, others may be exposed to the same class of attack. More generally, if you use CloudFront to cache any HTTP responses for which X-HTTP-Method-Override is honored, whether in GCS or in your own web services, you will want to add a similar CloudFront configuration.
Pendo has chosen to replace “whitelist” and “blacklist” in our own product with terms that more clearly describe the relevant function, such as “allow list” and “exclude list”, which do not use color to imply acceptance or rejection. However, we have used “whitelist” in this article because as of the time of publication, the AWS CloudFront settings use this language to describe the necessary feature.
We would be remiss if we did not thank the security researcher who brought this issue to our attention. While Pendo does not run a public bug bounty program, we follow the principles of responsible disclosure and promptly investigate and, as necessary, remediate any potential vulnerabilities that are reported to [email protected]. Although the individual requested that they remain anonymous, we wanted to publicly thank them for contacting us, and providing us with useful information to correct this interesting edge case.
About the Authors
Galen Johnson is a Sr. Production Operations Engineer at Pendo where he helps support their Cloud Infrastructure. He is a 25+ year veteran in the IT industry. He spent his formative years working for start-ups and consulting. Prior to joining Pendo, he spent 16 years at SAS where he helped architect and support their hosting solution. He also architected the deployment and managed the SAS OnDemand for Academics learning environment. He spends a lot of his free time gardening, simple woodworking and being a mediocre Fortnite player.
Michael K Johnson (mkj) is an Engineering Director at Pendo. His early background was dedicated to Linux, including operating system and kernel development at Red Hat. He has been responsible for operating system security, reproducible software build processes, system design and architecture, open source policies and processes, and scalable software development processes. At Pendo, Michael leads the teams primarily responsible for web event collection, horizontally scalable event ingestion and processing, and high-performance ad-hoc batch analytics—all while walking at his treadmill desk. In his spare time, he enjoys making things in many ways, including 3D printing, machining, and various forms of CNC.
Melodie Moorefield-Wilson is Lead Product Security Engineer at Pendo. With over a decade of experience in software, she has had various roles in FinTech, BioTech, and Healthcare. When time allows, you might find her joining in on Capture the Flag competitions, or playing pickleball with her family.