Deploy360 Events

Other IPv6 Stuff at APRICOT 2017

We already covered the IPv6 Deployment session during APRICOT 2017, but there were several other IPv6-related sessions that are worth mentioning.

First thing to mention is that there was a tutorial on deployment of IPv6 in a production mobile network. This was a 1.5-hour session led by Jeff Schmidt (Telstra) and provided some insights into what the business and technical considerations were, and what ended-up actually being deployed on their network.

Jon Brewer (NSRC) is someone who always managed to conjure up interesting talks at APRICOT Conferences, and his tutorial on the Internet-of-Things was no exception. Whilst IoT is not specifically IPv6, it’s likely that IPv6 will be required to facilitate the plethora of devices expected in future.

This tutorial discussed core concepts and the types of applications that are enabled by IoT technologies, before covering radio and network protocols for low-power WANs such as the 802 series, 3GPP and LoRaWAN. Higher-level protocols used in IoT are also discussed including CoAP, MQTT, REST and Websockets.

On a related note, there was also a presentation by Jeff Apcar (Cisco) on the Low Power Wide Area (LPWA). This covered this new area of communications where networks of sensors with limited power availability need to be connected across both urban and rural areas; some over long range wireless networks with substantial RF interference.

Last, but not least, there was the IPv6 Readiness Measurement session. This is an initiative of TWNIC (the Taiwan National Internet Registry) aims to encourage organisations working on IPv6 deployment to share their IPv6 measurement methods and results.

The IPv6 situation in India as presented by Ajai Kumar (NIXI) is becoming interesting, with different measurements calculating IPv6 deployment somewhere between 20 and 40% which makes India the economy with the highest rate of deployment in the Asia-Pacific region. Even more encouragingly, some large organisations including the State Bank of India have even higher rates of deployment, most of the major IXPs are IPv6-enabled, whilst the .in ccTLD also support IPv6.

The Asia-Pacific economy with the second-highest level of IPv6 deployment is Japan, whose situation was presented by Tomohiro Fujisaki (NTT). IPv6 deployment continues to increase and currently sits somewhere just over 20%. However, three major cellular operators have announced they will commence IPv6 services in 2017, whilst a number of fixed-line ISPs have already started to offer commercial ISP services. Some government services are now available via IPv6, although support by the major Japanese content providers still needs to improve.

Things are a bit less positive in Korea, Taiwan and Vietnam, although there was substantial progress with IPv6 deployment in Vietnam during 2016, with FPT Telecom becoming the first ISP in the country to offer IPv6 to customers (which we previously discussed). The Taiwanese government has also deployed IPv6 in most of its agencies, has started to offer IPv6 on its public Wi-Fi service (iTaiwan), whilst around 20% of the traffic of TANet (the NREN) is using IPv6.

One of the reasons for limited IPv6 uptake in Korea appears because it has one of the highest user bases in the world, and therefore obtained a large amount of IPv4 resources early on. IPv6 deployment only really started 3 years ago, and then mostly on cellular networks, so that combined with the lack of local content available via IPv6 has provided limited incentives for ISPs to provide this to their customers.

However, regardless of where you are, we encourage you to consider deploying IPv6, so please check our Start Here page for more information!

Deploy360 Events

The Network Forensics problem of IPv4

Although not directly on the subject IPv6, we absolutely need to draw your attention to a great presentation from Geoff Huston (APNIC) on Forensic Tracing in the Internet during APRICOT 2017. This relates to the pervasive use of Carrier Grade NATs as a means of extending the useable life of IPv4 on the Internet, and the implications for metadata record keeping and tracing users.

As we know, the pools of IPv4 addresses are close to depletion, but around 90% of the Internet is still only accessible via IPv4. As a result, Carrier-Grade NAT (CGN) has been widely implemented whereby private IPv4 address space is used in conjunction with a limited number of public IPv4 addresses in order to conserve public IPv4 address space. In other words, many customers are sharing a single public IPv4 address that will usually also change over a given time period.

If you therefore wish to trace from where traffic has originated from, then you need to maintain an extensive logging system keeping records on source IP addresses, source port addresses, along with dates/times. CGN bindings are formed for every unique TCP and UDP session, which can mean 150-450 bytes per connection and 33-216,000 connections per subscriber each day, resulting in the need to log 5-96 MB of data. For 1 million subscribers, this will generate up to 1 PB of data per month!

It’s becoming ever more complex to handle this information, and even if it’s possible to maintain comprehensive records, subscribers are also likely to be operating NATs and the trace will stop at these edge points. Bear in mind that some operators are also running out of private IPv4 address space on individual subnets, and are therefore needing to implement layers of CGNs.

Furthermore, it’s becoming increasingly difficult to analyse traffic flows as users and applications resort to encryption, sessions are split over multiple paths and access technologies (e.g. cellular, wifi), and even over a combination of IPv4 and IPv6.

So whilst Law Enforcement Agencies have traditionally focused on the network as the point of interception and tracing, and have introduced laws to mandate ever more extensive logging, the reality is that IPv4 addresses are increasing losing coherent meaning in terms of end party identification.

This might be interpreted that the choice is between ever more complicated and expensive record keeping systems, or transitioning to IPv6. Of course, some may see obfuscation through IPv4 as a positive benefit, but the fact remains that IPv4 is increasingly less scalable and becoming more complex to manage. IPv6 brings many other advantages with it, and confidentiality can still be maintained by using platforms and applications that support this.

You can watch Geoff’s presentation during the Network Security session on YouTube.

And if you’re interested in deploying IPv6 after this, then please see our Start Here page for more information!

Deploy360 Domain Name System Security Extensions (DNSSEC) Events

The Case for DNSSEC, DANE & Root Key Rollover @ APRICOT 2017

Our colleague Jan Žorž from the Deploy360 team was asked to present during the DNS/DNSSEC sessions during APRICOT 2017 last week, and this provides us with the opportunity to highlight a few of the other presentations there.

If you need to make the technical and/or business case for deploying DNSSEC, then you can do worse than check out the excellent presentation from Jim Reid (RTFM). In particular, he dispels some of the myths surrounding the size of zone files, CPU loads, traffic levels and key rollover, whilst covering some of the tools that can be used to manage and debug DNSSEC installations. It’s pretty in-depth and covers quite a few aspects of  running DNSSEC, but it’s maybe worth highlighting some of the guides and tools mentioned.

For those looking for guidance on issues such as key selection and signature duration,  NIST report 800-81-2 is recommended, whilst RFC 6841 is suggested as a starting point for a DNSSEC policy/practice statement. When it comes to troubleshooting though, then DNSSEC drill from NLnet Labs can be used to validate domains, show which key algorithms and lengths are used, pinpoint stale keys and signatures, and identify DS record and KSK mismatches. ISC also has a similar tool called delv included with BIND 9.10 and above distributions, whilst DNSVIZ also offers a web-based visualisation tool that allows you to type in a domain name and check whether it validates correctly, although it uses its own resolver.

Jim said one of big drivers of DNSSEC could be DANE, especially with the emergence of Let’s Encrypt and the IETF’s ACME standards that are promoting the use of TLS. DANE can address the issue of third-party trust as it allows digital certificates to be DNS and signed with DNSSEC, enabling end users to validate that the correct certificate is being used.

This led into Jan’s presentation where he explained how he’d deployed and tested DANE in the Go6lab. As well as demonstrating that deployment can be straightforward, it also showed that nearly 70% of all attempted SMTP sessions with the top million Alexa domains were encrypted with TLS. 41% of these used a certificate from a trusted CA, 17% used an untrusted certificate, whilst 11% was opportunist and unsigned, but it does illustrate the potential opportunities for DANE. The other important point to make is that DANE is particularly useful for non-web applications such as SMTP (e-mail) and XMPP (Jabber) where it is difficult to visually identify the validity of a certificate.

On a practical level, DNSSEC has recently been implemented in Vietnam, and Nguyen Trung Keen (VNNIC) provided an overview of DNSSEC deployment on .vn. This had been planned for two years, with the necessary infrastructure being installed at two separate sites. The key signing ceremony had taken place on 20 December 2016, with the DS record for .vn being activated in the root zone on 31 December 2016.

The next stages were sign the second level domains (e.g.,, as well as open a testbed to allow registrars to add and update signed domains for their customers. A training programme was also planned during 2017 to encourage ISPs, DNS hosting providers and other DNS owners to deploy DNSSEC. So the message is that DNSSEC is becoming increasingly available and you need to think about implementing it.

And just in case you haven’t yet heard the news, Rick Lamb (ICANN) discussed ICANN’s plans to roll the Root Zone DNSSEC Key Signing Key during the second-half of 2017. ICANN in its role as the IANA Functions Operator, manages the Key Signing Key (KSK) used to sign the Root Zone Signing Key (ZSK) which is managed by VeriSign and changed every quarter. The current KSK has been used since the Root Zone was first signed in 2010, it’s good practice to change keys, and ICANN wish to try this exercise under normal rather than emergency conditions (such as a key compromise).

The rollover plan is outlined in five documents, with a view to generating the new KSK in the 4th quarter of 2016, adding it to the root zone on 11 July 2017 (in parallel with the old KSK), and then signing the DNSKEY RRset starting from 11 October 2017. The old KSK will be revoked on 11 January 2018, but will remain in the root zone.

The DNS/DNSSEC session was held in two parts, and both are available on YouTube. Whilst not directly DNSSEC-related, you might also want to check out the other presentation from Jim Reid (RTFM) on Using ~300 Billion DNS Queries to Analyse the TLD Name Collision Problem in Session 2.

If this inspires you to deploy DNSSEC, then please see our Start Here page for more information!

Deploy360 Events

NAT64check & More IPv6 @ APRICOT 2017

Some of the Deploy360 team were presenting at APRICOT 2017 last week, and there were other interesting presentations that are worth highlighting. Whilst we’ll cover the dedicated session on IPv6 deployment in this post, we’ll also do a round-up of some of the other IPv6-relevant presentations and tutorials in subsequent posts.

Our colleague Jan Žorž had been asked to chair the APNIC Executive Committee election process, so it meant a mad dash from there to open the IPv6 deployment session with his presentation on the NAT64/DNS64 testing being undertaken by the Go6lab. This debuted at AfriNIC-25 towards the end of last year, but NAT64check has already proved to be an extremely useful tool.

As many mobile operators are moving to IPv6 only which is incompatible with IPv4 on the wire, it’s necessary to employ transition mechanisms such as 464XLAT or NAT64. The NAT64check testbed was therefore developed by Go6lab and SJM Steffann with support from the Internet Society so that operators, service providers, and hardware and software vendors can see how their networks and systems work in these environments.

When using NAT64 there are many things that need to be checked to ensure they work correctly, so NAT64check was developed so that websites can be checked for consistency over IPv4, IPv6-only and NAT64, as well to compare responsiveness using the different protocols. This allows network and system administrators to easily identify anything is ‘broken’ and to pinpoint where the problems are occurring, thus allowing any non-IPv6 compatible elements on the website to be fixed.

If you go to the NAT64check website, you can find a list of which websites that users have checked, and how they display using the different protocols. The tool does a comparison of the images returned, and attempts to identify any broken elements, along with providing information on the responsiveness of the respective connections. And one interesting observation that seems to backup those by APNIC Labs, is that IPv6 connections are typically more responsive if they can be established in the first place. Further evidence that IPv6 does not result in a degraded experience if set-up correctly.

This was followed by a presentation from Abdul Awal (BDREN) on IPv6 deployment in BDREN, the Bangladesh National Research and Education Network, as well as the wider challenges of deploying IPv6 in Bangladesh. There is not currently much deployment of IPv6 in the country, and this has not been helped by a lack of support for AAAA glue records in the .bd ccTLD. However, BDREN has been leading the IPv6 deployment efforts having implemented dual-stack in 2016, and is currently promoting IPv6 on its connected campuses. As a result, the websites of some universities and educational institutes are now available via IPv6, and dual-stack has been enabled at one campus.

More generally, some of the ISPs in Bangladesh had enabled IPv6 on their backbones, but are still not providing IPv6 to their end users citing a lack of demand. It needs impetus from another direction, and whilst directives existed from the Bangladesh Telecommunications Regulatory Commission (BTRC), the government could help by mandating the use of IPv6 in their e-governance platforms, when procuring IT systems, and by establishing a National IPv6 Task Force to help raise awareness and push deployment of IPv6.

Sami Salih (Al Jouf University) then presented IPv6 developments in Sudan and on SudREN, the national research and education network. Sudan had a national IPv6 migration plan and 26 training workshops involving more than 1,000 participants had been organised between 2011 and 2015, as well the SDv6 Task Force being formed. Twelve /32s have been allocated to Sudan by AfriNIC.

SudREN is also leading the IPv6 deployment initiative and has enabled IPv6 on 50% of the core routers, 10% of the routers of their member institutions, and on 100% of all servers. There is a native IPv6 link to the rest of the Internet via Sudatel, with a backup link via Hurricane Electric. Every member institute has been allocated a /48 with a /4o reserved for future demand, which means that SudREN had now assigned 80% of its two /32 blocks. As a result, more than 25% of SudREN are now IPv6 capable, which compares very favourably to other parts of the world.

Rounding off the session was Dat Nguyen Thanh (FPT Telecom) who provided a commercial perspective on IPv6 deployment at FPT. FPT is one of the major ISPs in Vietnam and Cambodia with over 1.6 million subscribers, and has been rolling out IPv6 in its core network. Over half their subscribers are now IPv6 enabled and total traffic amounted to 477 Gb/s, and their method for rolling out IPv6 on CPEs was discussed in detail.

This is a session worth following, and if you missed it live, it’s available on YouTube to watch.

If this inspires you to transition your networks, devices and applications to IPv6, then please see our Start Here page for more information!


Deploy360 @ APRICOT 2017

The Deploy360 team will be busy next week at the APRICOT 2017 Conference in Ho Chi Minh City, Vietnam. Aftab Siddiqui and Jan Žorž will be co-chairing the Asia-Pacific BCOP BoF, Jan will be giving two presentations on DNSSEC/DANE and IPv6, whilst Aftab will moderate a panel about the APNIC transfer market. Our colleague Andrei Robachevsky will also be talking about the MANRS initiative.

First up on the agenda is the Best Current Operational Practices (BCOP) BoF being organised by the Deploy360 team on Monday, 27 February 2017 (17.30 to 19.00 UTC+7) in Ballroom 1&2 of the Sheraton Saigon Hotel. We already blogged about this, but this is part of our BCOP initiative to encourage the development of regional BCOP initiatives to document operational knowledge and practices based on the experience of network engineers.

The following day (28 February) between 14.30 and 15.30 UTC+7 in VIP 3&4, Jan will talk during the DNS/DNSSEC session about his experiences setting up DNSSEC with Let’s Encrypt and DANE. He’ll also present the results of his mail server testing where he asked the question – which remote servers have DNSSEC-signed domains, which are TLS capable,  and how many are using DANE.

On Wednesday (1 March) between 09.00 and 10.30 UTC+7 in Ballroom 3, Aftab will be moderating the APNIC Panel on Navigating the IPv4 Transfer Market. With 4 out of 4 RIRs reaching IPv4 exhaustion, activity in the IPv4 transfer market has been growing, and this Q&A session aims to guide network operators through the transfer market process. The other panellists include David Huberman (Oracle), Amy Potter (Hilco Streambank), Peter Himmesch (Addex) and George Kuo (APNIC).

In the afternoon between 16.00 and 17.00 UTC+7 in Ballroom 1 & 2, Andrei will be presenting Two Years of Good MANRS during the Network Security (2) sessionMutually Agreed Norms for Routing Security (MANRS) is an initiative of network operators to establish a recognised industry supported baseline for implementation of essential security measures.

Jan then has another presentation on Thursday (2 March) between 09.00 and 10.30 UTC+7 in Ballroom 3 during the IPv6 Deployment session, where he’ll talk about the NAT64/DNS64 experiments undertaken by Go6lab and IPv6-lab. As many mobile operators are moving to IPv6 only, it’s necessary to employ transition mechanisms such as 464XLAT or NAT64. The Go6lab NAT64/DNS64 testbed has therefore been established so that operators, service providers, and hardware and software vendors can see how their solutions work in these environments. This is a really interesting presentation, so be sure to catch this session if you’re involved with IPv6 deployment.

The Internet Society is also organising a couple of other sessions. The Community Wireless Mesh Networks (CWMN) BoF will be held on Tuesday, 28 February 2017 between 17.30 and 19.00 UTC+7 in VIP 3 & 4, and will be chaired by Kanchana Kanchanasut (Internet Education and Research Laboratory). CWMNs are an alternative way of providing affordable access to the Internet through community participation, and the aim of this session is to bring together practitioners with experience in planning and deployment to expand knowledge, share lessons learnt, and encourage others to get involved.

Last but certainly not least, the ISOC@APRICOT session will be held on Wednesday, 1 March 2017 between 17.00 and 17.30 UTC+7 in the Danang Room. This is an opportunity to come and meet the team and hear about what the Internet Society is doing, but we’re always happy to talk any time about technology and how we can make the Internet a better place.

Further Information


BCOP BoF to be held at APRICOT 2017

The Deploy360 team will be organising a BoF on Best Current Operational Practices (BCOP) during APRICOT 2017 in Ho Chi Minh City, Vietnam. This is being held on Monday, 27 February 2017 (17.30 to 19.00 UTC+7) in Ballroom 1&2 of the Sheraton Saigon Hotel, and will be co-chaired by Aftab Siddiqui and Jan Žorž.

This is part of Deploy360’s BCOP initiative to encourage the development of regional BCOP initiatives to document operational knowledge and practices based on the experience of network engineers. This knowledge is often exchanged in a casual manner between groups of people in hallways or in social settings, or where it is discussed online, in a variety of forums with archives in different formats, if archived at all.

BCOPs aim to be living documents that attempt to encapsulate best practices as agreed by experts in their fields, and reviewed by the global networking community. There are active initiatives in Europe and North America in particular, as well as Africa and Latin America, so this BoF is an attempt to re-initiate efforts in the Asia-Pacific region with support from APNIC.

Deploy360 has also recently updated its BCOP page to reflect all the BCOP documents that we know about around the world. If we’ve missed something, please contact us at and we’ll be pleased to add it.

More Information:

We encourage network operators to document how they deploy new technologies including IPv6DNSSECTLS and routing resilience/security, so their experiences may assist others in doing so. We’ll be organising other BCOP sessions at RIPE 74 & 75, and in Africa and Latin America this year, so stayed tuned for more information.