Deploy360 Transport Layer Security (TLS)

DNS privacy in new Android 9

I recently enrolled in the Android developer preview programme and got hold of the Android P (9 beta) OTA image for my Nokia 7 Plus phone, and while discovering what’s new, I found a new advanced option under network settings called ‘Private DNS’ that got my attention. This led to me finding an article from Erik Kline describing this new feature in Android 9, which to my surprise supports DNS-over-TLS (RFC 7858).

Last year we wrote about the experiments in the Go6lab with DNS-over-TLS where we set up a recursive DNS resolver listening on port 853 and serving DNS answers to queries encrypted with TLS. This setup was useful if your local DNS resolver was Unbound or Stubby, and since then I’ve been using Stubby as my local DNS client on MacOS with the Unbound DNS server at the Go6lab ( as a recursive resolver for encrypted DNS queries without any issues.

So armed with the information from Erik, I decided to test out the Android implementation.

First thing was to turn on the setting and test it with the ‘’ server which worked fine. Enabling ‘log-queries’ on the Unbound server quickly revealed that DNS queries are reaching the server and being properly resolved. Since ‘’ port 853 is openly reachable from the Internet – you can test it as well.

Encouraged with this success, I thought of adding this functionality to the Go6lab recursive resolvers that I use on a daily basis, but this time it needs to listen on the standard DNS port 53 and as well as port 853 for DNS-over-TLS queries. Since we are running recursive resolvers in Go6lab on Unbound software, this was not a difficult task.

I had to install certbot, obtain a Let’s Encrypt certificate for the host, and add 5 new lines of configuration to unbound.conf…

interface: ::0@853
interface: ::0@53

ssl-service-key: “/etc/letsencrypt/live/”
ssl-service-pem: “/etc/letsencrypt/live/”
ssl-port: 853

To make sure it’s also listening on port 53, I added the two lines ending with @53 as a precaution.

After rebooting the Unbound server, setting the ‘Private DNS’ setting on my phone to ‘ and briefly setting the ‘log-queries’ setting to ‘yes’, the first queries started to appear in the query log.

I was not sure whether queries were coming in over port 53 or 853, and therefore whether they were encrypted or not, so I used tcpdump to capture the traffic from the IPv6 address of my phone.

As we can see, traffic is flowing to and from port 853, indicating that our DNS traffic is being encrypted. Examining the tcpdump capture file with Wireshark reveals this is actually TLS 1.2 traffic and that the content of the packets is encrypted.

Most excellent I thought, but the problem is that with a pre-set ‘Private DNS’ server I would have issues when outside of the Go6lab network (or without a VPN connection back into it) as our DNS recursive resolvers are not reachable from outside to prevent DDOS attacks.

Well there’s a setting in ‘Private DNS’ that says ‘Automatic’ which is pretty opportunistic, and after switching this on, the phone tries to send the DNS query to port 853 and falls back to port 53 if there’s no answer. Observing this through tcpdump confirmed the automatic setting does exactly what is expected – it figures out whether DNS-over-TLS is working on our recursive resolvers and starts using it. This option doesn’t validate server names or hashes, but it makes sure it can negotiate TLS 1.2 (and probably also validates the certificate chain that’s provided). However, in ‘Strict Mode’, the system does additionally validate the hostname where this is entered.

The great aspect of automatic mode is that it does not lock you into one pre-set DNS resolving server, but uses whatever is provided when connecting to a different network. And it’s been confirmed by the user of a Pixel phone that this option is also present in the release version of Android 9 Pie, so this functionality will probably stick around in future versions of Android.

Implementation of this new privacy and security feature by the good people at Android is a big boost for making encryption the norm on the Internet, and opens the opportunity for network operators, DNS operators, VPN providers and everyone else running recursive DNS resolvers to quite easily add this functionality. Whoever connects to your network with Android 9 will automatically start using encrypted DNS service and will make sniffing of the queries at the transport layer a little bit more difficult for eavesdroppers.

With Unbound this is pretty simple – and in my next blog post we’ll be looking into how to enable DNS-over-TLS in Bind and some other DNS servers.

There are already few public recursive DNS resolvers currently supporting DNS-over-TLS on port 853 – Cloudflare on, Quad9 on, and also CleanBrowsing. But we’re also appealing to network operators and anyone who operates a recursive resolver to please add this functionality to your server. It won’t hurt as regular DNS clients will not even notice it, but for those whose operating systems can take advantage of port 853, it will benefit them a lot.

For more information about why DNS-over-TLS is important, the DNS Privacy Project has a good problem statement on its website, along with an explanation of the technical solutions.

Further Information

Deploy360 Domain Name System (DNS) Domain Name System Security Extensions (DNSSEC) Transport Layer Security (TLS)

Monitoring your DANE deployment

In a previous series of articles (part 1, part 2) we described how to install and use DANE for verifying your email and web server certificates through the DNS. In this article, I discuss how to monitor whether your TLSA records still match the certificates used for your services.

The aim is help people doing something similar, as this monitoring system saved the DANE deployment in the Go6lab from being broken many times, especially at the very beginning of deployment when the automated systems we built didn’t work completely correctly all the time 🙂

If we’re using self-signed certificates for our services and we manually change the underlying key, then we’ll probably also change the values of the TLSA records. If we forget to change them or are using, for example, Let’s Encrypt that automatically renews the certificate every 90 days, then we need to have some automatic DANE monitoring system that informs us of misalignment between the certificate being used and the TLSA record.

For this reason there are more and more online tools to check individual services by entering the URL and port, and with a press of a button tell if things are still working fine. A few examples of such tools include:

These tools allow you to quickly check your services, but you need to manually enter their details and check them one-by-one. In the Go6lab we use TLSA records to secure many services, and since we’re using Let’e Encrypt certificates that are automatically renewed, we had to build a tool that checks for DANE validity and even dynamically change the TLSA records in our DNS to match the new certificate.

Using the default mode of certbot, one of the Let’s Encrypt applications to fetch and renew certificates, the underlying key of certificate is also changed whenever we renew. In this case, the proper process is to renew the key, use a tool to generate a TLSA record from our certificate using the Command Line Interface (e.g. tlsagen), use the dynamic DNS updater (nsupdate) to add the new TLSA record, wait for the DNSSEC process to finish, sign and distribute the zone, then restart the service to commence using the new certificate and then remove the old TLSA record from our zone.

Let’s focus here on the DANE monitoring tools that we are using for remote DANE monitoring in the Go6lab:

  • For checking HTTPS we are using ldns-dane, part of LDNS toolkit from NLNETLabs.
  • For checking SMTP we are using a tool named “danecheck“, built by Viktor Dukhovni.

LDNS toolkit from NLNETLabs is pretty straightforward, and you can install it with the apt or yum package installers if you happen to use a Linux system (just search for “ldns”). However, if you have an SNI-based https domain service (multiple virtual hosts using the same IP address), then please note that Server Name Indication (SNI) was introduced in ldns-dane in version 1.6.17. At the time of writing, the majority of apt and yum repositories offer version 1.6.16 and with that version checking of the SNI domains will fail despite the fact that your TLSA might be correct.

The tool for testing DANE is called ldns-dane and usage is pretty simple:

[root@mx dane-monitor]# /usr/local/bin/ldns-dane verify 443 dane-validated successfully
 2001:67c:27e4::52 dane-validated successfully

The tool will send an DNS query for the TLSA record for, in our case, ‘’, and will also query for A and AAAA records associated with ‘’. It will then communicate with the IP addresses, ask for the certificate and compare the two of them.

If your output does not look similar to the above, something is wrong with your TLSA records or DNSSEC signatures.

If you decide to download and build LDNS toolkit from the NLNETLabs website, please note that version 1.17 requires OpenSSL 1.1. If you use version 1.0, then stick with LDNS version 1.16.17.

Viktor Dukhovni’s danecheck tool is built a little bit differently as it requires a working Haskell GHC plus Stack toolchain. These can be downloaded from the Haskell platform website and are also available as packages for various operating systems.

Using the tool is quite simple, you specify just a domain name that you wish to test (in most cases):

root@openNMS:/# danecheck IN DS 15361 8 1 104e2a57cd968ec0bb283872c7f546d9b174
1839 ; AD=1 NoError IN DS 15361 8 2 f2cf0d4b46d3be6ccd25516cafc33fb5105f
28c3a9a82d782afe2ff1017bb208 ; AD=1 NoError IN DNSKEY 256 3 8 AwEAAc48Qs/Ew8nipd9vYGlk+NzXqE/llT
lqWNmYJeuQ0WwbSwu04Kuco9H5E0tdx26jbzaRffsxRPB/Kr0ydVqTU84fJnc= ; 
AD=1 NoError IN DNSKEY 257 3 8 AwEAAcgqIy9CjcyqWY+sJpNk9hC7CAUozO
O2VBU2Y16bHKOQsck4WyPovc0J7zdUX7QqIQod0DzB2qbdf1sXo4+T24yJzvU= ;
AD=1 NoError IN DNSKEY 257 3 8 AwEAAcmW5udATR8Gmpeqyv4WYVGvr3gE5s
juQgFotDTaxX3yR/y7hPCOq02EtQD2kMqgsOHK4nyqx79l1sME1CZ8aDSFBgc= ; 
AD=1 NoError IN MX 10 ; AD=1 NoError IN MX 20 ; AD=1 NoError IN A ; AD=1 NoError IN AAAA 2001:67c:27e4::beee ; AD=1 NoError IN TLSA 3 1 1 39726a2fe2bb052cf00e6b95
a8385f7446c3ea76d015fd0644085d6e9f3e8853 ; AD=1 NoError[]: pass: TLSA match: depth = 0, 
name =
 name =
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-10-29T12:33:26Z
 notAfter = 2018-01-27T12:33:26Z
 Subject CommonName =
 pkey sha256 [matched] <- 3 1 1 39726a2fe2bb052cf00e6b95a8385f7
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2
4a9db406fee12d2cc90180517616e8a18 IN A ; AD=1 NoError IN AAAA 2001:67c:27e4::61 ; AD=1 NoError IN TLSA 3 1 1 43fc43f31135d9542658b87356d
129ccfdd654f996499ac97cd6d34b2b8f9668 ; AD=1 NoError[]: pass: TLSA match: depth = 0, 
name =
 name =
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-11-18T22:45:21Z
 notAfter = 2018-02-16T22:45:21Z
 Subject CommonName =
 pkey sha256 [matched] <- 3 1 1 43fc43f31135d9542658b87356d129c
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2

If everything works as it should, then you should see similar output to the above.

Note the lines with a string of “AD=1 NoError” in it, that can be can be used in a script to automatically determine whether anything is wrong. For Go6lab purposes, we test each domain and “grep” for string “AD=” to extract messages from the tool’s output, and “grep -v” string “AD=1 NoError” to eliminate everything that seems OK and valid. If there is anything after the “grep” processing, omething is wrong with the domain and an e-mail can be sent flagging that.

Here’s an example how we monitor our DANE SMTP environment using a simple bash script:

domains=`cat $file`
for domain in $domains; do
  echo "Checking SMTP DANE for $domain"
  /usr/local/bin/danecheck "$domain" | grep "AD=" | grep -v NoError \ 
  > /root/dane-smtp-monitor/"$domain"
    if [ -s /root/dane-smtp-monitor/"$domain" ]
      echo "DANE for $domain is broken"
      mail -s "DANE TLSA config for $domain is broken!!!" \ 
      < /root/dane-smtp-monitor/"$domain"
      echo "All good for $domain ..."

This shows how you can monitor all your domains that are entered line-by-line in the specified domains.txt file. You can also use the same script to monitor your https services, but of course it will need to be modified as the output of the tool is different, and you would probably want to use “grep -v “dane-validated successfully”” as an indicator that all is well and functioning.

DANE and DNSSEC are very nice add-ons for security and provide better verification of certificates and services. But if they’re not working properly then you’re not doing yourselves a favour, hence the importance of monitoring to enable action to be taken if something fails.

Jan Žorž