Deploy360 Internet of Things (IoT) IPv6

11th Slovenian IPv6 Summit

13475071_10154872777836002_3153864564864735524_oThe 11th Slovenian IPv6 Summit organised by Go6, ARNES and LTFE was held on 21 June 2016 at the Brdo Technology Park in Ljubljana. This event co-sponsored by the Internet Society and chaired by Deploy360’s Jan Žorž featured a high quality programme of speakers which attracted an international audience of around 120 participants.

The opening keynote was provided by Lousewies van der Laan who’s a Director on the ICANN Board and a former Vice-President of the Alliance of Liberals and Democrats for Europe (ALDE). She discussed how the Internet was a rare successful example of a functioning global multi-stakeholder community, but there needed to be a sustained campaign from the Internet community to engage with governments to preserve this in future. Like it or not, government policy had a very real impact on Internet operators, and policy makers needed to be able to make informed decisions. Unfortunately, this often required a repetitive process to reinforce the message, but ultimately would lead towards a constructive dialogue.

Lousewies van der LaanThis led into the Internet-of-Things and IPv6 talk from Patrik Fältström (Netnod) who discussed the paradigm shifts in connectivity models since the 1980s, and how end-to-end communication between devices used to be the norm. IoT is essentially a return to that situation, although he tended to define IoT as where one node acted autonomously; either as a sensor communicating data or a node acting upon a command. However, unlike in the past, the open and ubiquitous nature of the Internet brought new challenges such as security, privacy, and interoperability standards, as well as regulatory and economic issues.

Market forces also had strong interest in service oriented vertical integration because the profit margins tended to be in ongoing integrated service provision rather than selling low-cost interoperable devices. This not only led to a proliferation of ‘standards’, but did not encourage use of IPv6 as NAT’ed IPv4 environments could be used to lock users into particular services.

IoT devices needed to be designed so they could continue to operate if the service they were originally intended to work with disappeared, and in a way they could be upgraded once the original vendor stopped supporting them. IPv6 also played a key role as this facilitated universal reachability and reduced dependence on individual vendor solutions.

Patrik FalstromPublic sector organizations should therefore use every opportunity that arises in procurement, regulation and project funding to require the use of open standards when they are available, or to promote their development when they are not. Patrick added that it was worth checking out the Internet Society’s IoT White Paper that covered many of these challenges.

Ivan Pepelnjak (ipSpace) followed up with how to automate IPv6 deployments. IPv6 configuration is very similar to IPv4 and is inherently boring which leads to mistakes, but mistakes that can be expensive. Every well-defined repeatable task can be automated, and IPv4 to IPv6 migration is no exception. The suggestion is therefore to utilise scripts (e.g. based on Perl) that can automatically analyse router configurations, scrape subnet information from interfaces, and use algorithmic mapping to build IPv6 subnets and host addresses from existing IPv4 ones. It was important to identify major components such as access control lists and firewalls that can cause problems when migrating, and then test the tools with IPv4 before utilising them for IPv6.

Sander Steffann (SJM Steffann) and Ron Broersma (US Navy) continued the theme of autoconfiguration challenges by relating their experiences of DHCPv6 servers. The main issue was that existing implementations did not handle static prefixes per remote-id very well, and that it was difficult to develop stable workarounds. This led to the development of DHCPKit which was written in Python and could be modified with additional plug-ins as necessary.

Ron BroersmaRon Broersma then went on to discuss his experiences in deploying IPv6-only networks. The goal was to have products that supported IPv6 just as well as IPv4 (so-called ‘feature parity’), but many vendors were not testing their products in IPv6-only environments and then took a long time to address the problems when these emerged.

In the DREN III contract, it has been mandated that IPv6 had to work as well as IPv4, and that all network management had to operate IPv6-only. This took just a single paragraph and had been very successful at achieving the goal, even though it sometimes meant replacing non-compliant products. A basic filter applied was that a vendor’s website had to be available via IPv6, and they would be excluded from consideration if this were not the case.

Since 2007, they had also attempted to run the SPAWAR management network as IPv6-only as this was isolated and therefore an ideal testbed. They were now starting to remove IPv4 from various segments of the production network as dual stack adds complexity, increases the attack surface in terms of keeping all the security in sync, and the use of IPv4 inevitably means address translation at some stage which erodes performance and obfuscates the origin of attacks. This is therefore a strong argument to accelerate IPv6 deployment by removing as much IPv4 as possible, highlighting products that do not have IPv6 feature parity, and pushing the case that IPv6 can actually improve security.

Silvia HagenSilvia Hagen (Sunny Connection) followed-up with some IPv6 deployment figures. IPv6 traffic to Google was now doubling every 9-12 months and at the current rate would reach 60% by 2018. Belgium had reached over 40%, the US over 26%, and over 20% in a number of other European countries. Even countries with lower figures such as Canada, the UK, Finland and Estonia had seen significant increases in deployment in the past year as ISPs were actively rolling out IPv6 there.

Perhaps more significantly, there were signs that the growth in CGNs (Carrier-Grade NATs) due to shortages of IPv4 addresses was beginning to cause noticeable performance issues, applications to fail, and geolocation issues for content and marketing providers, along with security problems and expensive maintenance. Access to IPv6 sites was soon expected to outperform access to IPv4 sites, which is why content providers are now pushing IPv6 so heavily. According to Gartner, the cost of deploying IPv6 is around 6% of an annual IT budget if spread over a 3 or 4 year period, but waiting too long will suddenly require an urgent and therefore much more expensive transition.

Slovenian IPv6 SummitThese experiences were mirrored by Stephanie Schuller (LinkedIn) who discussed the roll-out of IPv6 at LinkedIn. Since their IPv6 launch in September 2014, they had seen IPv6 traffic rise to over 10% globally (hitting 13% in mid-2015) but had also seen a significant improvement in performance on mobile networks running IPv6 (up to 40% in France). This was explained as being due to higher network RTTs over IPv4 causing more TCP timeouts and therefore greater page download times.

The proliferation of CGNs had been one of the primary reasons for moving to IPv6 as these complicate security and are also responsible for performance issues on mobile networks. There are significant challenges for an enterprise the size of LinkedIn to implement IPv6 as they rely heavily on redundant and reliable web services which means that load balancers, firewalls and monitoring tools all need to work properly with IPv6. However, by implementing dual stack data centres, they have been able to work out the operational aspects, train staff, and gradually build or obtain compliant applications and tools with a view to eventually moving to IPv6 only.

13503108_10154873080671002_215988164006058764_oFernando Gont (SI6 Networks) changed tack slightly by discussing the security and privacy implications of IPv6 addressing. With IPv6, the Interface Identifiers (IIDs) do not change over time, this makes it possible to correlate network activity and undertake network reconnaissance, and as the IIDs are often linked to NIC vendor, could identify vulnerabilities. The aim of RFC 7217 is therefore to introduce stable privacy-enhanced IIDs that are generated using several different factors, and have been implemented in the Linux v4 kernel, NetworkManager v1.2.0 and dhcpcd 6.4.0.

The Internet Draft (I-D) draft-gont-6man-non-stable-iids also proposes an update to RFC 4941 to allow the exclusive use of temporary stable addresses which might typically be used for roaming. This requires IIDs to be different for each prefix, non-predictable, and without embedding Layer 2 addresses (e.g. MAC). Other approaches such as MAC address randomisation should be considered flawed, as discussed in the I-D draft-got-predictable-numeric-ids. In the meantime, it’s advisable to read I-D draft-gont-6man-address-usage-recommendations-00 that provides advice on which address types to configure and how
to employ them in different scenarios.

A very full day was rounded-off with a presentation from Luka Manojlovič on connecting health centres in Nova Gorica using IPv6. He discussed the problems that needed to be solved and what they learned.

Finally, the local network operators present at the event provided short updates on their IPv6 developments. IPv6 is already offered as an option by the two largest operators Telekom Slovenije and T-2, with Telekom Slovenije enabling it by default for new subscribers.

All the presentations from the meeting can be found on the Go6 website.

Deploy360 Events IPv6

Reflections from the 10th Slovenian IPv6 Summit…

Fernando Gont opening the meeting
Fernando Gont opening the meeting

The 10th Slovenian IPv6 summit, held on 9 June 2015, attracted around 120 attendees –  an impressive number for a country with only 2 million people. Of course, the agenda was packed with good content – Fernando Gont opened the meeting with a keynote on generic IPv6 security and some myths around that and he also closed the meeting with a presentation about his findings and tests around “IPv6 Extension Headers in the wild” and how they are filtered on the Internet.

You can see all the video recordings of the sessions and presentations from the meeting agenda.

10th Slo IPv6 summit
10th Slo IPv6 summit

After the keynote, our friend Timo Hilbrink from XS4ALL explained their path to IPv6 for all residential customers, lessons learned, issues encountered, and ways to fix them. His presentation set the mood that IPv6 for residential customers can be a reality today and just after his presentation we scheduled a short report from Primož Dražumerič from Telekom Slovenija about their IPv6 test pilot with “friendly residential customers”. To everyone’s enjoyment, the report showed that they had no issues – everything is fine and they already have 600 residential customers on dual-stack and they are completely ready to enable it by default – basically with just one press of the button (or one phone call).

That’s exactly what we wanted to hear after six years of awareness raising, bringing the IPv6 knowledge to the country and persuading operators to deploy IPv6 and enable it for all residential customers.

Well, we have some good operational practice from other countries – Comcast, TWC, Verizon and others in US, Telenet from Belgium, RCS-RDS from Romania, Swisscom in Switzerland and also other rising countries in IPv6 deployment show us that enabling IPv6 by default for all customers is not an issue anymore and there are no reported disasters when it’s happened – in contrary, nobody even noticed.

We were all waiting for the main moment of the day – the panel discussion between CxO level representatives from Slovenian ISPs on why IPv6 is not entirely enabled between content and residential end users. Slovenia is one of the champions of enabling content on IPv6, as we have enabled IPv6 on servers that generate approximately 60% of all http traffic in the country – see stats!

ISPs discussing IPv6
ISPs discussing IPv6

The panel was moderated by our dear friend Davor Šoštarič and consisted of representatives from all the biggest ISPs in the country. The discussion started in a very positive manner, until one of the panelists started explaining that there is no demand for IPv6 from users, there is no business case, and no incentive to switch on IPv6. This remark took all others into the same stream and one of them added that there is no content on IPv6 so there is no incentive to enable it for residential customers.

At that point in time I was completely disappointed with the discussion, my blood pressure reaching dangerous levels. When audience questions opened, I made my remark, noting that I disagree with their several points of view for the following reasons:

  • users will never demand IPv6; transition must be as seamless as possible
  • IPv6 is a technology update and not a business update
  • operators need to offer connectivity to the whole Internet, not just part of it
  • if there were 10 different Internet protocols, they would have to support all of them and reach the whole Internet
  • there is lots of content on IPv6 – Google, Facebook and other big global players enabled IPv6 and also Slovenian national search engine, TV and other big news portals are accessible on IPv6
  • measured fact is that users can reach Facebook 40% faster over IPv6
  • big players are removing IPv4 from their internal networks and datacenters (for example Facebook) and translating at the edge
  • as operators they should take good care of their customers and develop their networks as the Internet develops…

Why was I so sad?

Operators in my country came a long way in deploying IPv6, some of them spent a lot of money for equipment changes in order to be able to support IPv6 for residential customers, did all the testing of CPE and did the pilot testing with some limited number of their customers – and Telekom Slovenije even enabled the “IPv6” button in its self-service customer portal where people can automatically enable IPv6 on their connection.  So they walked many kilometers of this windy and rocky road, but they stopped 1 meter before the finish line – enabling IPv6 for all residential customers – saying that there is no business case, user demand, and content.

I’m completely puzzled. Parse error.

Just the last step, one small step for operators – but giant leap for mankind…

From the reactions from the operators after this panel session I can see that this was a very useful exercise to go through as it was thought provoking – now new ideas started to emerge among operators, going in direction “hey, why exactly shouldn’t we do it?”

Let’s see, time will tell. Hopefully a short one.

The event continued with other very good (as usual) presentations from Ivan Pepelnjak on IPv6 microsegmentation, Nathalie Kunneke Trenaman on her efforts to bring IPv6 into her home and home devices, a Brocade presentation on their switch fabrics and also a presentation from Simon Delakorda on the “impact of not deploying IPv6 on information and civil society”. Thoughtful exploration of this aspect, bridging the technical and civil society communities understanding of each other. The event was continued on the second day with our second SINOG meeting, a young Slovenian Network Operators Group with more broad topics of discussion that are also of interest to operators.

Conclusion: Maybe we managed to move the IPv6 situation in the country a bit further; it’s just one more step and you are done, dear ISPs. Do it, move forward and never again look back.

[NOTE: If you can read Slovenian, Jan has written a similar post (in Slovenian) on the Go6 web site.]