Domain Name System (DNS) Improving Technical Security Mutually Agreed Norms for Routing Security (MANRS)

APNIC Labs/CloudFlare DNS Outage: Hijack or Mistake?

At 29-05-2018 08:09:45 UTC, BGPMon (A very well known BGP monitoring system to detect prefix hijacks, route leaks and instability) detected a possible BGP hijack of prefix. Cloudflare Inc has been announcing this prefix from AS 13335 since 1st April 2018 after signing an initial 5-year research agreement with APNIC Research and Development (Labs) to offer DNS services.

Shanghai Anchang Network Security Technology Co., Ltd. (AS58879) started announcing at 08:09:45 UTC, which is normally announced by Cloudflare (AS13335). The possible hijack lasted only for less than 2min. The last announcement of was made at 08:10:27 UTC. The BGPlay screenshot of is given below:

Anchang Network (AS58879) peers with China Telecom (AS4809), PCCW Global (AS3491), Cogent Communications (AS174), NTT America, Inc. (AS2914), LG DACOM Corporation (AS3786), KINX (AS9286) and Hurricane Electric LLC (AS6939). Unfortunately, Hurricane Electric (AS6939) allowed the announcement of originating from Anchang Network (AS58879). Apparently, all other peers blocked this announcement. NTT (AS2914) and Cogent (AS174) are also MANRS Participants and actively filter prefixes.

Dan Goodin (Security Editor at Ars Technica, who extensively covers malware, computer espionage, botnets, and hardware hacking) reached out to Cloudflare about this possible hijack and received following statement from Cloudflare PR stating that they are ruling out any malicious intent and there was no drop in customer traffic and it was fixed quickly, but also blamed Hurricane Electric (AS6939) for the leaked route.

Considering this just a configuration mistake which was rectified quickly and didn’t cause any reported damage but it doesn’t solve the problem and there is a possibility that someone with a bad intent can do a lot of harm like the way it was done during Amazon Route 53 hijack, unless we take appropriate steps towards a secure and resilient internet.

Once again, this attack would have been easily avoided if proper prefix filtering was done by Hurricane Electric. As discussed in the previous blog, MANRS can be part of the solution here. Mutually Agreed Norms for Routing Security (MANRS) calls for four simple, but concrete actions ALL network operators should take to reduce the most common routing threats. The first is filtering, which prevents the propagation of incorrect routing information (others are anti-spoofing, address validation, and global coordination.).

So, what are you waiting for? Be part of the solution and help protect the core. Join MANRS.

Deploy360 Domain Name System (DNS) IETF Improving Technical Security Transport Layer Security (TLS)

Cloudflare launches DNS service with privacy, TLS and more

There was an important development this month with the launch of Cloudflare’s new DNS resolver service. This is a significant development for several reasons, but in particular it supports the new DNS-over-TLS and DNS-over-HTTPS protocols that allow for confidential DNS querying and response.


Before we get to that though, Cloudflare joins Google’s Public DNS that uses and Quad9 DNS that uses, by implementing as a memorable IP address for accessing its new DNS service. IP addresses are generally not as memorable as domain names, but you need access to a DNS server before you can resolve domain names to IP addresses, so configuring numbers is a necessity. And whilst a memorable IP address might be cool, it’s also proved important recently when DNS resolvers have been blocked or taken down, requiring devices to be pointed elsewhere.

The address is part of the – public IP address range actually allocated to APNIC, one of the five Regional Internet Registries, but it has been randomly used as an address for so many things (e.g. for proxies), that it was overwhelmed with garbage traffic every time it was announced in the global routing system. The Cloudflare network is sufficiently scaled that it can cope with this traffic, so an agreement was established to allow APNIC Labs to analyse traffic to this address range in return for Cloudflare being able to use for its DNS resolver project.

Under the agreement, APNIC also allowed Cloudflare to use the address as the second IPv4 address for their DNS services, from a separate address range allocated for research. IPv6 addresses are also available, although at 2606:4700:4700::1111 and 2606:4700:4700::1001 they are not nearly so memorable.

Improving DNS Performance

It’s also important to have a fast and reliable DNS resolver service. Every device connecting to the Internet needs a recursive DNS resolver to translate names into addresses, which are usually provided by your ISP who’ll automatically configure your devices to point to them. However, ISP-provided resolvers can often be slow and unreliable (a problem I’ve encountered recently with an ISP that I use, and there’s some interesting comparison testing in this in this blog post), quite aside from the fact that locally-based servers have been blocked or censored by some national governments in response to civil unrest or politically embarrassing incidents. The likes of Cloudflare, Google, Quad9 and others provide alternatives, allowing users to choose better performing services with enhanced features rather than just having to rely on the default service offered by your ISP.

DNS Privacy

And this leads us to the most innovative feature of Cloudflare’s DNS service. The DNS was never designed with privacy or security in mind, and whilst DNSSEC adds verification to the information returned by DNS queries, traffic is still sent unencrypted which allows anyone monitoring your network connection to see what you’re looking up and therefore visiting.

A couple of different solutions to this problem have recently been developed by the IETF, which we’ve discussed in previous blogs. The DPRIVE Working Group has developed DNS-over-(D)TLS, whilst the DOH Working Group has been working on DNS-over-HTTPS. The different protocols have different merits and try to address particular legacy behaviour of the DNS protocol (e.g. limitations on UDP packet size and the need for fast responses), but both approaches encrypt DNS queries and responses to provide confidentiality of these transactions.

So far so good, but there’s also a need for both resolvers and clients to be upgraded to support these protocols. Up until now, whilst some test servers have been set-up by various organisations, Google has been the only significant provider supporting DNS-over-HTTPS through their Google Public DNS and Android operating system. As a result, there’s been a certain reticence for other providers and vendors to build in support for protocols that rely on a competitor, but hopefully as more alternative resolver services are established, we’ll see increasing usage of these DNS privacy protocols.


It should be pointed out that both DNS-over-HTTPS and DNS-over-TLS only encrypt DNS communications between stub-resolver (on the client device) and recursive resolver (e.g. Cloudflare They do not currently encrypt the communications between the recursive resolver and authoritative DNS servers when resolving queries, so the provider of the resolver needs to be trusted as it still has the potential to monitor and log transactions. Neither does either protocol ensure the integrity of information returned from an authoritative server, which is why DNSSEC is also required to cryptographically assert the DNS entries are correct. They are nonetheless important components in improving the security and confidentiality of the DNS.


If you’re interested in how to set-up a recursive DNS resolver and/or stub-resolver on a client device, you can find a lot of information over on the DNS Privacy website. Deploy360 also published a blog post last year on how we set-up DNS-over-TLS in the Go6lab which is well worth reading.


UPDATE: We should also note that Cloudflare’s service also supports “Query Name (QNAME) Minimization”, as defined in RFC 7816. They include a reference to this at the end of their “Nitty Gritty Details” page:

Cloudflare minimizes privacy leakage by only sending minimal query name to authoritative DNS servers. For example, if a client is looking for, the only part of the query discloses to .com is that we want to know who’s responsible for and the zone internals stay hidden.

Kudos to Cloudflare for implementing this higher level of privacy protection in their service.

Thanks to Stéphane Bortzmeyer for pointing out on Twitter that Cloudflare had implemented QNAME minimization. We had missed the small mention in Cloudflare’s technical blog post and weren’t aware they had included this.


Deploy360 Domain Name System Security Extensions (DNSSEC) IETF

Main IETF Website Returns To Being DNSSEC Signed Via CloudFlare

Good news this week for DNSSEC and content-distribution-networks (CDNs)! Last year the Internet Engineering Task Force (IETF) decided to move the main IETF web site over to a CDN to speed up access to IETF web pages for people trying to reach them all over the world.   While this sped up access to the IETF’s content, it unfortunately meant that the main IETF website had to lose its DNSSEC signature because the CDN vendor, CloudFlare, did not yet support DNSSEC.  (I’d note that this was only the main IETF web site – other IETF web sites such as the datatracker and tools sites continued to be DNSSEC-signed.)

Those of us advocating for DNSSEC were naturally disappointed by this move last year, but we understood the need and also understood that CloudFlare was committed to bringing DNSSEC to their customers – and indeed we’ve been writing about CloudFlare’s journey towards DNSSEC.

So this week we were very pleased to see this announcement by IETF Chair Jari Arkko:

Some time ago we moved the static parts of the IETF web page to a CDN service. While this provided a significant improvements for page load times and retained our ability to serve the pages over IPv6, we were unable to provide DNSSEC for the web pages that were being served by the CDN.

Our CDN vendor, Cloudfare, however, has worked in the background to enable DNSSEC for their customers. They have now come back with a system that we have enabled for the IETF web pages. (Thank you Cloudfare, this was important!)

We plan to keep the new arrangement on at for a while before finally moving to this arrangement on Testing the new arrangement on would be appreciated!

Jari Arkko, IETF Chair

As noted, the main IETF website is NOT yet DNSSEC-signed at the regular “” but is instead available with a DNSSEC signature at while everything is tested out.

Regardless, this is great news for DNSSEC, for the IETF … and also as a demonstration that CloudFlare’s implementation is obviously getting that much closer to being available!

Please do check out and give it any kind of DNSSEC-related tests that you can!

IETF web site

And if you haven’t gotten started with DNSSEC yet, please visit our Start Here page to find out how you can begin!

Deploy360 Domain Name System (DNS) Domain Name System Security Extensions (DNSSEC)

CloudFlare Wants To Update DNS Registration Model To Automate DNSSEC

CloudFlare logoOver on the CloudFlare blog today, Olafur Gudmundsson wrote a lengthy post titled “Updating the DNS Registration Model to Keep Pace with Today’s Internet” where he outlines a critical challenge that CloudFlare has run into on their path to implementing DNSSEC for their customers.

Essentially, the issue is this – on the signing side of DNSSEC, the process works like this:

  1. A “DNS Operator” may host your DNS records and sign them with DNSSEC keys.  As part of doing this, they will generate a “Delegation Signer” or “DS” record that must be provided to the parent zone (typically a top-level domain (TLD)) to complete the “global chain of trust”.
  2. The DNS Operator has to communicate this DS record to the Registrar for the domain.
  3. The Registrar then provides this DS record to the Registry that operates the TLD.

This needs to be done initially when the domain is first signed with DNSSEC – and then the process needs to be performed every time the Key Signing Key (KSK) for the domain is rolled over.  Typically this might be done once each year but could be done more or less frequently.  The key point is that every time there is a key rollover, the new DS record must be communicated up to the TLD.

Here’s one way that I show the process graphically:

DNSSEC Signing Steps

Notice the role of the Registrar here. They are in the middle of the process.

And THAT is CloudFlare’s problem.  They say they are hosting 2 million domains for customers.  In order for CloudFlare to automate DNSSEC signing to be as simple as a clicking/tapping a button in their user interface (as they have done for IPv6), they need to be able to interact easily with the registries for all those domains – and in the current system that means interacting with all the registrars!  Making it more challenging, some registrars have a clue about DNSSEC – and many others still don’t.

It’s a challenging issue.  As Olafur notes, there are now DNS records such as CDS and CDNSKEY, defined in RFC 7344, that can help with this, but they will require registrars to do some work to look for those records. But there are larger issues here that get into business processes, too.  For instance, many registrars are also DNS operators who will gladly host your DNS records for you for a fee – they have very little incentive to help make it easy for other DNS operators to host your domain.   There are a number of other issues.

Olafur began talking about this back at IETF 91 in Hawaii and this will be a big panel discussion at next week’s DNSSEC Workshop at ICANN 52 in Singapore (which will be streamed live and also recorded).

There is also a public mailing list set up for anyone who is interested in helping work on this issue.  You can join the effort and subscribe at:

This work will be ongoing for quite some time and probably wind up in the DNSOP Working Group within the IETF.  It’s a critically important challenge we need to address to bring further automation to DNSSEC deployment and help many more people secure their domains.

Your feedback on all of this is definitely welcome!  Please do leave a comment here… or on Olafur’s blog post… or on social media… or contact Olafur directly.

And… if you want to get started with DNSSEC, please do visit our Start Here page to begin!

Deploy360 Domain Name System Security Extensions (DNSSEC)

CloudFlare Seeks Help Testing Their DNSSEC Implementation

CloudFlare logoThis morning the team over at CloudFlare announced they have begun their DNSSEC implementation and asked for help in testing what they have done so far.

They also include a note at the bottom indicating that people interested in participating in their public beta program should contact them.

As we’ve written about several times in the past, CloudFlare continues to move ahead towards their goal of making DNSSEC available throughout their content distribution network (CDN) / DNS hosting service.  They’d originally had a goal of rolling it out by the end of 2014 but ran into some challenges that they are still working through.  As they note, they are getting closer.

This is great news to see – and we look forward to seeing their public beta program move ahead!  If you use CloudFlare, or just want to help them out, please do check out their blog post.

And if you want to get started with DNSSEC or DANE yourself, please visit our Start Here page to find resources to help you begin.

Deploy360 Domain Name System Security Extensions (DNSSEC)

CloudFlare Writes About DNSSEC Complexities And Considerations

CloudFlare logoThe folks over at CloudFlare published another great article earlier this week, “DNSSEC: Complexities and Considerations” that dives into more detail about some of the challenges of implementing DNSSEC.  Specifically, author Nick Sullivan explores the:

  • Exposure of DNS zone content through zone-walking
  • DNSSEC key management
  • DNS reflection/amplification attacks

He dives into the topics in great detail and explains what CloudFlare is planning to do to address each of these issues.  I strongly encourage you to check it out!

And then if you want to start implementing DNSSEC or DANE within your own environment, please visit our Start Here page to get started!

Deploy360 Domain Name System Security Extensions (DNSSEC) Tutorials

CloudFlare Publishes Excellent Introduction To DNSSEC

CloudFlare logoThe team over at CloudFlare published an excellent introduction to DNSSEC today that is well worth a read.  CloudFlare has developed a reputation for writing blog posts that provide a solid level of technical depth and this one certainly does.  Nick Sullivan starts by walking through the basics of DNS and including some packet captures and nice illustrations. Then he gets into man-in-the-middle (MITM) attacks and provides a great graphic that very succinctly shows a MITM attack against DNS:

CloudFlare MITM example

Even better, Sullivan nicely explains the “Kaminsky Attack” and the situation that makes the attack possible.    He then plunges into DNSSEC, explains RRsets and RRSIGs, ZSKs and KSKs, and touches on the value of NSEC/NSEC3 to prove that records don’t exist.

All in all it is an excellent introduction and we’re very pleased to see CloudFlare publishing this piece.  Thanks to Nick Sullivan and his team for getting this out there!

As we’ve written about before, CloudFlare has been saying since the ICANN 50 DNSSEC Workshop back in July that they would have DNSSEC available for their customers by the end of 2014.  Their post today says “in the next six months”… but we’ll hope it comes in on the sooner side of that. 🙂  It was also great to see the official announcement that CloudFlare has hired Olafur Gudmundsson, one of the developers of the first DNSSEC implementation many, many years ago and currently one of the co-chairs of the DANE Working Group within the IETF.  We’ve been working with Olafur over the past few years through our partnership with Shinkuro, Inc., where he worked before, and we’re delighted that he’s now working on DNSSEC at CloudFlare.

All great to see – and this will only help get DNSSEC much more widely deployed!

If you want to get started with DNSSEC today, please visit our Start Here page to find resources targeted at your role or type of organization. Help us make the Internet more secure today!

P.S. Have you checked out our new DNSSEC Fact Sheet in English, French and Spanish?

Deploy360 Domain Name System Security Extensions (DNSSEC)

CloudFlare Re-affirms Goal of DNSSEC Support By End of 2014

CloudFlare logoOver on ThreatPost, Dennis Fisher wrote about “Small Signs Of Progress On DNSSEC” reporting on a presentation by CloudFlare’s Nick Sullivan at the Virus Bulletin conference in Seattle this week.  The article didn’t go deeply into DNSSEC (as our tutorial pages do) but did have this point which is key to me:

Sullivan said CloudFlare, one of the larger DNS providers in the world, plans to deploy DNSSEC on its network by the end of the year.

To no surprise, this reaffirms what CloudFlare’s John Graham-Cumming stated back in June at the ICANN 50 DNSSEC Workshop in London where he presented a set of slides that are available for download.  From what Graham-Cumming said in London, the intent was to make DNSSEC available to customers with as simple a switch as CloudFlare has done today with IPv6.

I highlight this because the content distribution networks (CDNs), of which CloudFlare is an example, are one of the major stumbling blocks for many companies to be able to sign their domains with DNSSEC.  Typically this is because of either:

1. The CDN vendor is also providing the DNS hosting for the domain (so that they can use DNS for load balancing and distribution to CDN edge servers) and would therefore be the one to do the DNSSEC signing of the zone; or

2. The CDN vendor is hosting the website via a CNAME, with the issue then that the company can sign their domain, but when DNSSEC validation hits the CNAME it has to restart, and typically the site referenced in the CNAME will not be signed because it is hosted on the CDN.

As John Graham-Cumming presented in his slides, there definitely ARE challenges related to DNSSEC-signing for CDNs and vendors providing global load balancing.  BUT… we as an industry have to figure out solutions so that we can get domains signed that are hosted by CDN vendors.

We’re thrilled that CloudFlare is again indicating that they will enable DNSSEC by the end of 2014 to provide a higher level of trust and security for their customers. We’re looking forward to seeing the nice spike in signed domains that should come from CloudFlare doing this.  And… we do hope to see the other major CDN vendors offering this soon, too!  Working together we can make the DNS part of Internet communication that much more secure!

P.S. Want to get started with DNSSEC?  Visit our Start Here page to find resources targeted for your role or type of organization.

Deploy360 Transport Layer Security (TLS)

CloudFlare Releases Open Source CFSSL, a TLS/SSL Toolkit

CloudFlare logoYesterday the folks over at CloudFlare introduced their “CFSSL” toolkit for working with TLS (SSL) certificates. Their blog post explains what CFSSL is all about, and they have also made the code available along with further documentation on Github:

This is interesting to me for a couple of reasons.  First, their blog post has some excellent diagrams outlining the challenges with ensuring that a TLS certificate is able to be validated by a web browser.  The author Nick Sullivan points out that different browsers trust different numbers of Certificate Authorities (CAs) – and that older browsers may not trust newer CA certificates.  He outlines the need to create “certificate bundles” that include multiple TLS chains.  The key point of all of this is to make it so that your TLS certificate is accessible to the widest range of browsers and systems.

As a tutorial alone, the post is a good read.

It also highlights the complexity (some might say “brokenness”!) of the current CA system and why many folks are looking for mechanisms to add more trust into the system (the DANE protocol being one of those potential mechanisms).

The post also explains their CFSSL tool which is available for anyone to use.  While it is not exactly a TLS library, like some of the other tools we’ve highlighted in our TLS for Applications area, the source code is available and some developers may find it of use.  I found it interesting that the tool could also be used to create your own CA and generate your own certificates.  This might be useful for people looking to do additional testing or to run their own CA for their own purposes.

Regardless of what you may do with the toolkit, kudos to CloudFlare for making it available under a permissive open source license and for providing the documentation they do.  I hope it will help some folks out there make the Internet more secure!

Deploy360 IPv6

CloudFlare Enabling IPv6 For All Customers

CloudFlare logoBuried in a post last month about Martin Levy joining CloudFlare was this gem:

CloudFlare is enabling IPv6 by default for ALL of their customers!

If you are not aware of CloudFlare, the are a “content delivery network” (or “content distribution network”… either way it is “CDN”) that takes your website and makes it available through their large network.  A CDN can help you accelerate the speed at which users access your content. They also can help with performance issues, protection from DDoS attacks and many other website concerns.

CDNs also, as I documented in a video a while back, can be an easy path to making your web content available over IPv6.  In my own personal case, I have a couple of sites on a hosting provider that has only IPv4.  Given that I don’t have the time to move them to a hosting provider that provides IPv6, I’ve set both sites up to go through a CDN that automagically makes them available over IPv6.  We maintain a list of CDN providers we are aware of who support IPv6.

But back to CloudFlare…  a few years ago they implemented a setting for “Automatic IPv6”.  All you had to do was toggle that from “Off” to “On” and… ta da… your content would be available over IPv6.  Now, as Martin Levy writes on CloudFlare’s blog:

Many customers have flipped the switch to enable IPv6. That’s good; but it’s time to make the default setting “IPv6 on.” In this day and age this is a very safe thing to do. Over the next few weeks CloudFlare is going to make the default for new customer be “IPv6 on.” No need to flip that switch to be enabled for the whole Internet (that’s IPv4 and IPv6).

In the upcoming weeks CloudFlare will enable IPv6 for existing customers in a staggered release. CloudFlare takes the delivery of each and every bit very serious and you can be assured that every person at the company is involved in making this operation is successful. Yes there will be the option to turn off IPv6; but we strongly believe that at this point there’s little need for that option to be exercised. 

So IPv6 will be on by default for all new customers – and all old customers will be migrated to having that setting enabled.

The results are already being seen in some of the available IPv6 statistics sites.  Eric Vyncke noticed the uptick in his chart of the % of top web sites available over IPv6 and in a posting to the IPv6 group on Facebook attributed that growth to CloudFlare:


Regardless of whether CloudFlare drove that specific growth, the fact is that have a CDN provider enable IPv6 by default for all customers is a great step forward!  Now we just need all the other CDNs to do the same thing and we’ll go a long way toward having a significant amount of the Web’s content available over IPv6.

How about you?  Have you enabled you web content to be available over IPv6 yet?  If not, how can we help you?  Please do check out our IPv6 resources and let us know what other help you need.