Deploy360 Domain Name System (DNS) Domain Name System Security Extensions (DNSSEC) Transport Layer Security (TLS)

Monitoring your DANE deployment

In a previous series of articles (part 1, part 2) we described how to install and use DANE for verifying your email and web server certificates through the DNS. In this article, I discuss how to monitor whether your TLSA records still match the certificates used for your services.

The aim is help people doing something similar, as this monitoring system saved the DANE deployment in the Go6lab from being broken many times, especially at the very beginning of deployment when the automated systems we built didn’t work completely correctly all the time 🙂

If we’re using self-signed certificates for our services and we manually change the underlying key, then we’ll probably also change the values of the TLSA records. If we forget to change them or are using, for example, Let’s Encrypt that automatically renews the certificate every 90 days, then we need to have some automatic DANE monitoring system that informs us of misalignment between the certificate being used and the TLSA record.

For this reason there are more and more online tools to check individual services by entering the URL and port, and with a press of a button tell if things are still working fine. A few examples of such tools include:

These tools allow you to quickly check your services, but you need to manually enter their details and check them one-by-one. In the Go6lab we use TLSA records to secure many services, and since we’re using Let’e Encrypt certificates that are automatically renewed, we had to build a tool that checks for DANE validity and even dynamically change the TLSA records in our DNS to match the new certificate.

Using the default mode of certbot, one of the Let’s Encrypt applications to fetch and renew certificates, the underlying key of certificate is also changed whenever we renew. In this case, the proper process is to renew the key, use a tool to generate a TLSA record from our certificate using the Command Line Interface (e.g. tlsagen), use the dynamic DNS updater (nsupdate) to add the new TLSA record, wait for the DNSSEC process to finish, sign and distribute the zone, then restart the service to commence using the new certificate and then remove the old TLSA record from our zone.

Let’s focus here on the DANE monitoring tools that we are using for remote DANE monitoring in the Go6lab:

  • For checking HTTPS we are using ldns-dane, part of LDNS toolkit from NLNETLabs.
  • For checking SMTP we are using a tool named “danecheck“, built by Viktor Dukhovni.

LDNS toolkit from NLNETLabs is pretty straightforward, and you can install it with the apt or yum package installers if you happen to use a Linux system (just search for “ldns”). However, if you have an SNI-based https domain service (multiple virtual hosts using the same IP address), then please note that Server Name Indication (SNI) was introduced in ldns-dane in version 1.6.17. At the time of writing, the majority of apt and yum repositories offer version 1.6.16 and with that version checking of the SNI domains will fail despite the fact that your TLSA might be correct.

The tool for testing DANE is called ldns-dane and usage is pretty simple:

[root@mx dane-monitor]# /usr/local/bin/ldns-dane verify 443 dane-validated successfully
 2001:67c:27e4::52 dane-validated successfully

The tool will send an DNS query for the TLSA record for, in our case, ‘’, and will also query for A and AAAA records associated with ‘’. It will then communicate with the IP addresses, ask for the certificate and compare the two of them.

If your output does not look similar to the above, something is wrong with your TLSA records or DNSSEC signatures.

If you decide to download and build LDNS toolkit from the NLNETLabs website, please note that version 1.17 requires OpenSSL 1.1. If you use version 1.0, then stick with LDNS version 1.16.17.

Viktor Dukhovni’s danecheck tool is built a little bit differently as it requires a working Haskell GHC plus Stack toolchain. These can be downloaded from the Haskell platform website and are also available as packages for various operating systems.

Using the tool is quite simple, you specify just a domain name that you wish to test (in most cases):

root@openNMS:/# danecheck IN DS 15361 8 1 104e2a57cd968ec0bb283872c7f546d9b174
1839 ; AD=1 NoError IN DS 15361 8 2 f2cf0d4b46d3be6ccd25516cafc33fb5105f
28c3a9a82d782afe2ff1017bb208 ; AD=1 NoError IN DNSKEY 256 3 8 AwEAAc48Qs/Ew8nipd9vYGlk+NzXqE/llT
lqWNmYJeuQ0WwbSwu04Kuco9H5E0tdx26jbzaRffsxRPB/Kr0ydVqTU84fJnc= ; 
AD=1 NoError IN DNSKEY 257 3 8 AwEAAcgqIy9CjcyqWY+sJpNk9hC7CAUozO
O2VBU2Y16bHKOQsck4WyPovc0J7zdUX7QqIQod0DzB2qbdf1sXo4+T24yJzvU= ;
AD=1 NoError IN DNSKEY 257 3 8 AwEAAcmW5udATR8Gmpeqyv4WYVGvr3gE5s
juQgFotDTaxX3yR/y7hPCOq02EtQD2kMqgsOHK4nyqx79l1sME1CZ8aDSFBgc= ; 
AD=1 NoError IN MX 10 ; AD=1 NoError IN MX 20 ; AD=1 NoError IN A ; AD=1 NoError IN AAAA 2001:67c:27e4::beee ; AD=1 NoError IN TLSA 3 1 1 39726a2fe2bb052cf00e6b95
a8385f7446c3ea76d015fd0644085d6e9f3e8853 ; AD=1 NoError[]: pass: TLSA match: depth = 0, 
name =
 name =
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-10-29T12:33:26Z
 notAfter = 2018-01-27T12:33:26Z
 Subject CommonName =
 pkey sha256 [matched] <- 3 1 1 39726a2fe2bb052cf00e6b95a8385f7
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2
4a9db406fee12d2cc90180517616e8a18 IN A ; AD=1 NoError IN AAAA 2001:67c:27e4::61 ; AD=1 NoError IN TLSA 3 1 1 43fc43f31135d9542658b87356d
129ccfdd654f996499ac97cd6d34b2b8f9668 ; AD=1 NoError[]: pass: TLSA match: depth = 0, 
name =
 name =
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-11-18T22:45:21Z
 notAfter = 2018-02-16T22:45:21Z
 Subject CommonName =
 pkey sha256 [matched] <- 3 1 1 43fc43f31135d9542658b87356d129c
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2

If everything works as it should, then you should see similar output to the above.

Note the lines with a string of “AD=1 NoError” in it, that can be can be used in a script to automatically determine whether anything is wrong. For Go6lab purposes, we test each domain and “grep” for string “AD=” to extract messages from the tool’s output, and “grep -v” string “AD=1 NoError” to eliminate everything that seems OK and valid. If there is anything after the “grep” processing, omething is wrong with the domain and an e-mail can be sent flagging that.

Here’s an example how we monitor our DANE SMTP environment using a simple bash script:

domains=`cat $file`
for domain in $domains; do
  echo "Checking SMTP DANE for $domain"
  /usr/local/bin/danecheck "$domain" | grep "AD=" | grep -v NoError \ 
  > /root/dane-smtp-monitor/"$domain"
    if [ -s /root/dane-smtp-monitor/"$domain" ]
      echo "DANE for $domain is broken"
      mail -s "DANE TLSA config for $domain is broken!!!" \ 
      < /root/dane-smtp-monitor/"$domain"
      echo "All good for $domain ..."

This shows how you can monitor all your domains that are entered line-by-line in the specified domains.txt file. You can also use the same script to monitor your https services, but of course it will need to be modified as the output of the tool is different, and you would probably want to use “grep -v “dane-validated successfully”” as an indicator that all is well and functioning.

DANE and DNSSEC are very nice add-ons for security and provide better verification of certificates and services. But if they’re not working properly then you’re not doing yourselves a favour, hence the importance of monitoring to enable action to be taken if something fails.

Jan Žorž



Deploy360 IETF IPv6

Deploy360@IETF99, Day 5: Kdo se moc ptá, moc se dozví

There’s a couple of sessions of interest on the last day of IETF 99 before we say na shledanou to the City of a Hundred Spires.

Both sessions are running in parallel on the Friday morning starting at 09.30 CEST/UTC+2. ACME will continue to discuss the ACME specification, as well as the addition of CAA checking for compliance with CA/B Forum guidelines. There’s also new drafts specifying how to issue certificates for telephone numbers, how to issue certificates for VoIP service providers to Secure Telephony Identity, and ACME extensions to enable the issuance of short-term and automatically renewed certificates, certificates for e-mail recipients that want to use S/MIME, and certificates for use by TLS e-mail services.

NOTE: If you are unable to attend IETF 99 in person, there are multiple ways to participate remotely.

Alternatively you can check out LPWAN that’s working on enabling IPv6 connectivity with very low wireless transmission rates between battery-powered devices spread across multiple kilometres. This will be discussing five drafts related to IPv6 header fragmentation and compression, as well as ICMPv6 usage over LPWANs.

That brings this IETF to an end, so it’s goodbye from us in Prague. Many thanks for reading along this week… please do read our other IETF 99-related posts … and we’ll see you at IETF 100 on 12-17 November 2017 in Singapore!

Relevant Working Groups

Deploy360 Transport Layer Security (TLS)

TLS…? Yes!

TLS badgeWe’re currently making an effort to build out our TLS for Applications resources, and we’re pleased to announce some updates.

First up, we’ve added a TLS Basics page which answers some of the questions about what TLS is, how it works, and why you should deploy it. If you need more detail, it’s worth checking out the SSL/TLS: What’s Under the Hood? paper from the SANS Institute.

We’ve also added links to a couple of other research papers from UC Berkeley on some TLS related issues, which can be found on our new TLS Reports page.

There are several tools available for checking a server’s TLS protocol support, and those that we’re aware of can be found on our TLS Tools page. We’re always happy to add other useful tools if you let us know about them.

And in case we haven’t mentioned it before ;-), Let’s Encrypt is offering free digital certificates with the view to increasing overall deployment of TLS. So there’s now even fewer excuses for not make the Internet more secure!

Finally, we’re also interested to hear from anyone who has implemented TLS and can share their experience as a case study, or who’d be willing to share other configuration and/or deployment information that others might find useful. If so, don’t hesitate to contact us.

Deploy360 Transport Layer Security (TLS)

Let’s Encrypt Today

letsencryptLet’s Encrypt enters public beta today, which means anyone can now sign-up for free digital certificates. Let’s Encrypt is a new trusted Certificate Authority (CA) that aims to bring down the costs of configuring secure servers in order to increase overall deployment of TLS.

This initiative offers more than just free certificates though, as it also supports automation to make the business of obtaining and managing certificates significantly less complex, whilst encouraging more frequent (90 day) renewal to limit damage from key compromise and mis-issuance. This is achieved through the Automated Certificate Management Environment (ACME) which offers a standards-based REST API allowing the agent software to authenticate that a server controls a domain, request a certificate, and then install it on a server without human intervention.

Over 26,000 certificates were issued during the closed beta trial, so the system has already been extensively tested in the wild. The CA is supported by fourteen sponsoring organisations with an interest in promoting encryption as the norm.

For more information on TLS, please do look at our Start Here page to understand how you can get started transitioning your networks, devices and applications!

Building Trust Encryption Improving Technical Security Open Internet Standards

Let’s Encrypt Enters Public Beta to Increase Encryption on the Internet

The Internet Society believes that encryption should be the norm for all Internet traffic. There are several efforts underway to enhance trust in the confidentiality of communication, including Let’s Encrypt, which attempts to bring down the price of configuring secure and authenticated web servers to increase overall deployment. Today, 3 December 2015, Let’s Encrypt enters Public Beta.

That means that you no longer need an invitation to get free certificates from Let’s Encrypt. From their announcement today:

“It’s time for the Web to take a big step forward in terms of security and privacy. We want to see HTTPS become the default. Let’s Encrypt was built to enable that by making it as easy as possible to get and manage certificates.”

We have more work to do before we’re comfortable dropping the beta label entirely, particularly on the client experience. Automation is a cornerstone of our strategy, and we need to make sure that the client works smoothly and reliably on a wide range of platforms. We’ll be monitoring feedback from users closely, and making improvements as quickly as possible.”

We encourage you to check out Let’s Encrypt and encrypt your own web servers. As we said in our previous post about the initiative, we are excited about the possibilities initiatives like Let’s Encrypt can bring, specifically for those that need to configure secure web services at scale.

We hope that we see even more innovative projects like this, and even more security on the Internet!

Deploy360 Domain Name System Security Extensions (DNSSEC) Transport Layer Security (TLS)

Let's Encrypt certificates tested in go6lab

letsencryptSince the announcement of the Let’s Encrypt project, we’ve been enthusiastic about the idea of having an open and transparent Certification Authority (CA) that could show other CA’s that it’s possible to run a certificates business and keep it free for everyone!

So, after the Closed Beta programme was announced, we managed to get involved and secure our web servers with their certificates. Firstly, we had a look at their ACME client that’s needed if you want one of their certificates, as the only way to request a domain certificate is through interaction with a back-end API.

Firstly you need to know that a separate certificate is required for every domain and subdomain you want to secure. For example you need separate certificates for,,, etc…

When you install the ACME tool on your system, you have a few options – or you can use Apache mode that configures your Apache web server automatically, talks to the Let’s Encrypt system to obtain an “auth token”, and publishes it on your website. It then tells the remote system to check for that file and whether it has the right content, and if that check goes through, the system will transfer the signed certificates and install them on your system. In case you are not running Apache (and for now just on Debian systems), you have other options such as Standalone mode that does a similar thing. However, since it sets up its own web server on port 80 in order to authenticate, you need to shut down your primary web server for a period of time which is not entirely ideal.

We chose Webroot mode that does the same authentication as the other methods, except that you need tell the ACME client where your webroot directory is. Then when you request a certificate for a domain, it obtains the authentication file from the remote system and puts it in your web server root directory so the system can access it and verify that you really control the server for that domain.

Here’s a quick how-to from their beta announcements post:

If you’re running Apache on a recent Debian-based OS, you can try the Apache plugin, which automates both obtaining and installing certs:

  ./letsencrypt-auto --apache --server 

To obtain a cert using a “standalone” webserver (you may need to temporarily stop your exising webserver) and

  ./letsencrypt-auto certonly -a standalone 
  -d -d 
  --server --agree-dev-preview

To obtain a cert using the “webroot” plugin, which can work with the webroot of any webserver software:

  ./letsencrypt-auto certonly -a webroot --webroot-path /var/www/example 
  -d -d 
  --server --agree-dev-preview

Note: Currently the webroot plugin can only obtain certs for several domains simultaneously if they share a webroot.

To receive instructions for the (fairly complex) process of obtaining a cert from Let’s Encrypt by manually providing proof you control a domain:

  ./letsencrypt-auto certonly -a manual -d 
  --server --agree-dev-preview

So, after experimenting a bit with the whole thing, the command that worked for us was:

  ./letsencrypt-auto certonly --debug --webroot --webroot-path /var/www/html 
  -d --server  
  /etc/init.d/httpd restart

Why is –debug there? Because their client does not (sort of) support Python 2.6 anymore. This is the default on CentOS 6.x systems and upgrading it creates a high probability that you’ll break many things on your server. So if you end up having to deal with Python 2.6, the letsencrypt client will not do anything other than try to update the dependancies on your system and exit with a message that Python 2.6 is not supported. If you really mean it, you should add that –debug switch to the script. Well okay, so be it, debug it is 😉 secured secured

After obtaining the initial certificate, I had to fix the content on the server as some pictures were referenced with http:// at the beginning of the URL and the verification system began complaining that some content was not secured. Well, now it is and the lock on the left of URL glows green.

So, the first step went quite smoothly and was not that hard at all, but with the understanding that the certificate was only valid for 90 days, we wanted to make the renewal automatic. However, how can we make it automatic if the client requires dialogs for entering email during the initial procedure?

Well, we tried to run it again with same parameters, but it started asking questions and after some research we found out that adding –renew to the script automagically skips all the questions and becomes perfect for automation. So, now we have our cron system ready to renew that certificate on the first day of every month at 00:55, fingers crossed 😉

During the process we encountered some small issues and questions that we’re still awaiting an answer to. One of the issues was that you can’t currently obtain a certificate for a server on an IPv6-only network as Let’s Encrypt seems to have some issues with the data centers where they have their system installed, although they say they’re working to rectify this.

Let’s Encrypt uses DNSSEC validating resolvers (unbound) and this ensures a quite high level of protection during the authentication process for DNSSEC signed zones. For non-signed zones, they are querying geographically sparse DNS resolvers and comparing the results – if results are not consistent, an attack is happening and validation fails. They also plan to rotate the resolving servers, so somebody could not build a sparse enough attack to poison all their DNS servers.

What we’d also hope is that Let’s Encrypt will put a big sign on on every corner of their webpage saying “Please sign your domain with DNSSEC and make the verification for certificate delivery more resilient to attacks, and by the way, your domain and services would be much more secure this way…” in order to encourage people to deploy DNSSEC. This is in the interests of everyone, and we already showed how deploying DNSSEC, signing domains and doing a KSK rollover is not difficult.

Our Conclusion: Let’s Encrypt is a great idea for running a CA that issues free certificates, and their certificate just works. The tools and client do need to a bit of polishing and made a bit more automatic so that every sysadmin will be able to install it with apt-get/yum, run it, and then state which domain they wish to secure.

Our next test will be to obtain a Let’s Encrypt certificate for our mail server sitting on Theoretically it should work, but let’s see next week when that subdomain will be whitelisted on the Let’s Encrypt system.

Important notice: The Let’s Encrypt Open Beta program starts up on 3 December 2015 and then more domains will be whitelisted in a much shorter time. Don’t be afraid to try it out!

Encryption Improving Technical Security Technology

Internet Society Supports the Let’s Encrypt Initiative to Increase End-to-End Encryption

The Internet Society sees encryption as a new norm to enhance the security of, and thereby trust in, the Internet.

The community is establishing that norm: Encrypted end-to-end (server to browser) web traffic is on the rise, allowing secure communication that cannot be monitored by 3rd parties. However, to deploy encryption for web services, you need encryption protocols to be implemented, an authentication infrastructure, and to tie this all together in a way that is workable for end-users and for those that provide web services.

There are various threads of work that contribute to enhancing trust in the confidentiality of communication. For end-users, these enhancements are shipped with updates of their browser. For the maintainers of web servers, configuring secure communication is tedious and error prone, and an expensive process.

The Let’s Encrypt initiative addresses that aspect. From its website, Let’s Encrypt is “a free, automated, and open certificate authority (CA), run for the public’s benefit.”

Let’s Encrypt sets out to bring down the price of configuring secure and authenticated web servers in order to get more web servers to deploy secured transport for services.

The innovation of Let’s Encrypt is not that it offers free certificates (which is something that is done by other market parties) but that it has developed a protocol that allows for driving down operational overhead costs associated with certificate configuration. Using an open-sourced toolset and a protocol currently being standardized in the ACME working group in the IETF, it provides a toolchain that aspires to be ‘shoot-and-forget’ with two commands. This is very appealing to anybody who has ever configured a certificate in a web server. With Let’s Encrypt, all an operator needs to know is one command — letsencrypt — and all the certificate generation and signing magic will happen automagically.

Let’s Encrypt is a fully certified Certificate Authority infrastructure that will be trusted by all browsers. Currently it is issuing certificates to beta program participants.

We would like to see this initiative flourish and in order to support its initial growth and help it scale from a few thousand beta users to millions of production users, we joined the initiative as a gold sponsor for 2015-2016.

We are excited about the possibilities an initiative like Let’s Encrypt can bring, specifically for those that need to configure secure web services at scale. Check out the website and follow along as it gets up and running over the next few months!

Editorial Note: The Let’s Encrypt initiative also published a blog post today about the Internet Society’s sponsorship.

UPDATE – 3 Dec 2015 – Olaf published a follow-up post about the Let’s Encrypt initiative entering the public beta stage where invitations are no longer needed to participate.