In the last few years, several Red Hat customers have asked how to add aWeb Application Firewall (WAF) to the OpenShift ingress to protect all externally facing applications.
A WAF is a Layer 7 capability that protects applications against some types of web-based attacks, including but not limited toCross Site Request Forgery (CRSF),Cross-Site Scripting (XSS) andSQL injection (for a more comprehensive list of all known web based attacks, seehere).Unfortunately, OpenShift does not have these capabilities included within the default ingress router, and as a result, alternate solutions must be pursued.
But now, with the advent of theGateway API and anew ingress stack for OpenShift, things are changing. In this article, we will demonstrate how you can protect your applications with a WAF.
OpenShift, Gateway API and WASM
Note: As of this writing, the OpenShift Gateway API implementation is in a dev preview support status and the functionality may change in the future, including the underlying technologies.
OpenShift, as a product, will adopt the new Gateway API standard. This feature is in dev preview behind a feature gate and can be activated as described inthis article.
For the purpose of this article, we do not need to analyze all of the new features of the Gateway API (a good summary for that can be foundhere). However, the feature that we are most concerned with is that the reverse proxy implementation of the Gateway API for OpenShift isEnvoy. In fact, by enabling the Gateway API feature, aclusterwideOpenShift Service Mesh is currently installed automatically.
By having the power of theIstio at our disposal, we can easily inject new behavior in Envoy usingWebAssembly (WASM) packaged plugins.WASM is a technology that compiles code such that it runs safely in a lightweight stack-based virtual machine.WASM was originally created to inject behavior in the browser to support sophisticated websites, but it turned out to be ideal for all kinds of plugin systems.To be compatible with Envoy, a WASM program must implement theproxy-wasm Application Binary Interface (ABI). This interface specification ensures that Envoy understands how to call the WASM program.
ModSecurity, OWASP WAF core rule set and Coraza
ModSecurity is a mature and stableApache module that adds WAF capabilities to an Apache instance. ModSecurity has been adopted over time by other http servers and proxies products, such asngnix via an appropriateconnector.ModSecurity has recently been donated toOpen Web Application Security Project (OWASP) and under this organization, will be maintained as an open source project with open and transparent governance.
One of the features of ModSecurity is that it can be configured to check for a set of rules via aDomain Specific Language (DSL) calledSecLang. The OWASP organization maintains a default set of rules that cover the most common attack types. This set of rules is referred to as the OWASP core rule set.When configuring a WAF based on ModSecurity, it is a good practice to start with the OWASP core rule set and then tweak the ruleset based on one’s needs.
Coraza is a port of ModSecurity to the golang language (ModSecurity is originally written in C) with the objective of making ModSecurity more adopted by enterprises.Coraza Proxy WASM is an adaptation of Coraza to the proxy-wasm-ABI.
That being said, by leveraging these solutions, it should start to be clear how we are going to enhance OpenShift ingress with a WAF.
Putting it all together
The following diagram depicts the approach for injecting Coraza in the Envoy proxy which manages the ingress traffic:
In the diagram above, we can see in yellow, the CRs that we created, and in green, pods or groups of pods that are running. You can find this implementation in a declarative GitOps format within thisrepository.
At the bottom of the diagram, we can find theGatewayClass CR. When configured appropriately, it triggers the deployment of an ServiceMesh (Istio) control plane. Moving up higher in the diagram, we can see theGateway CR. This manifest creates an Istio ingress gateway pod. When the Gateway CR is deployed the OpenShift Ingress Operator creates a wildcard DNS entry for this ingress endpoint. Additionally, we created awildcard certificate usingcert-manager to enable TLS for all the subdomains managed by this endpoint.
The last piece of configuration that the administrator needs to do is to inject the Coraza WAF code as a WASM plugin. This can be achieved with aWasmPlugin CR. Notice that this manifest references an OCI artifact containing the WASM code.
On the tenant side, we can deploy a simple application that demonstrates the use of a WAF now that the capability has been enabled. The example application can be foundhere. The key component in this application is theHttpRoute. By specifying theparentRefs section, we request the newly created Gateway to serve this route.
Now that we have enabled the Gateway API within OpenShift and configured the cluster to support our use case, and deployed the sample application, we can finally test and validate the functionality.
Let’s first get the current cluster base domain:
export cluster_base_domain=$(oc get ingress.config.openshift.io cluster --template={{.spec.domain}} | sed -e "s/^apps.//")
Then we can issue a request:
curl -I https://nginx-echo-headers.gwapi.${cluster_base_domain}HTTP/2 200server: istio-envoydate: Fri, 10 May 2024 18:23:50 GMTcontent-type: text/plainx-envoy-upstream-service-time: 2Now, let’s make a request that looks a little more suspicious:
curl -I "https://nginx-echo-headers.gwapi.${cluster_base_domain}/?arg=<script>alert(0)</script>"HTTP/2 403date: Fri, 10 May 2024 18:25:21 GMTserver: istio-envoyThe request was denied with a 403 error.
Within the Istio ingress gateway pod log, messages similar to the following will be displayed:
024-05-10T18:25:21.318531Z critical envoy wasm wasm log openshift-ingress.coraza-waf: [client "100.64.0.5"] Coraza: Warning. XSS Attack Detected via libinjection [file "@owasp_crs/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "7663"] [id "941100"] [rev ""] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS_GET:arg: <script>alert(0)</script>"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "10.129.2.11"] [uri "/?arg=<script>alert(0)</script>"] [unique_id "cmQXIPQegrompxpYJnD"]126…Load test
By now, we know that the OWASP core rule set is being applied by our WAF./ But, was there a performance hit against the application? Let’s try to measure the delta with and without Coraza enabled.
To accomplish this task, we need to deploy anothergateway without the plugin.
To generate load, we are going to usecassowary, a cross platform load testing tool. Here is our baseline, without Coraza:
podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi-no-coraza.${cluster_base_domain} -c 10 -n 1500 -d 30 --disable-keep-aliveStarting Load Test with 1500 requests using 10 concurrent users99% |███████████████████████████████████████ | [29s:0s] 30.056327229sTCP Connect.....................: Avg/mean=17.74ms Median=17.00ms p(95)=22.00msServer Processing...............: Avg/mean=19.43ms Median=19.00ms p(95)=25.00msContent Transfer................: Avg/mean=0.00ms Median=0.00ms p(95)=0.00msSummary:Total Req.......................: 1500Failed Req......................: 0DNS Lookup......................: 1.00msReq/s...........................: 49.91Now, let’s see how the same load performs with Coraza enabled:
podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi.${cluster_base_domain} -c 10 -n 1500 -d 30 --disable-keep-aliveStarting Load Test with 1500 requests using 10 concurrent users99% |███████████████████████████████████████ | [29s:0s] 30.075051016sTCP Connect.....................: Avg/mean=18.14ms Median=18.00ms p(95)=23.00msServer Processing...............: Avg/mean=33.46ms Median=33.00ms p(95)=40.00msEnvoyContent Transfer................: Avg/mean=0.00ms Median=0.00ms p(95)=0.00msSummary:Total Req.......................: 1500Failed Req......................: 0DNS Lookup......................: 1.00msReq/s...........................: 49.88We can see that for this simple use case, the latency introduced by the WAF was around 14ms (33ms-19ms).
Also, we noticed that on the resource consumption side, the ingress gateway pod with Coraza (top in the picture below) consumes approximately one order of magnitude (x10) more memory and CPU.
Conclusions
In this article, we demonstrated an approach to add a WAF to our OpenShift ingress stack using the new Gateway API and injection via a WASM plugin. We used Coraza as the WAF implementation and the OWASP core rule set as our WAF policies. When attempting to deploy these assets in your own environment, please keep in mind that the Gateway API is still in dev preview in OpenShift and that the Coraza WASM plugin is not currently supported by Red Hat.
In addition to WAF and the core rule set, OWASP recommends sanitizing the HTTP headers following theOWASP Secure Headers Project “best practices”. These best practices can be tested at develop time, but one can also decide to sanitize the headers on the fly at run time. We built a custom WASM plugin to demonstrate this use case. The plugin can be found inthis repository. This sample plugin attempts to satisfy most recommendations from the publishedHTTP Security Response Headers Cheat Sheet.
In our opinion, WASM is a promising approach for enhancing the behavior of the Envoy reverse proxy especially when Envoy is controlled by Istio (multiple plugins can be attached to the same Envoy instance). We also recommend investigating projectKuadrant in detail which uses WASM to add authz/authn and rate-limiting capabilities (and more) to an Envoy proxy.
