From AMS-IX to DE-CIX

blog Thursday, August 26, 2021

If the number of users of a product or service increases, then it has a positive effect on its added value. This is the definition of network effect. But if we try to describe an internet exchange (IX) the similarities between these definitions are especially striking. An IX is the physical location where different parties, such as ISPs, exchange traffic with each other. Because multiple parties are connected and thus share intermediate connections, the benefits are shared as well. The subsequent snowball effect that follows is enormous, as each additional member is another shared interconnection. It sounds like internet exchanges are a vital part of the internet, but are they? And what exactly is the added value of an internet exchange for your IT organization?

A direct path from the Netherlands (AMS-IX) to Germany (DE-CIX) equals low latency

The most cited advantage of connecting to an IX is low latency. As a member of AMS-IX, you can theoretically exchange traffic directly with 850+ affiliated parties. In practice, this translates to high quality traffic. This is because the routes have a shorter path. Also, the number of hops that this network route must go through is lower. It even has a positive effect on packet loss and packet delay variation. At shorter distances this saves milliseconds, at longer distances it means no unnecessary detours to foreign ISPs. This makes internet exchanges an indispensable link in improving internet traffic for you and your customers.

Sometimes parties see significant amounts of traffic being exchanged between each other, for example between Worldstream and Orange. Despite being connected to an IX, the quality of internet traffic can be increased by setting up a direct (e.g., physical connection) peering session. With a Private Network Interconnect (PNI), the parties gain control of the intervening traffic through a BGP session. This is a link between the two parties, without an intermediating IX. Such a session is a matter of setting up a virtual configuration on their own routers. Afterwards, these parties can be considered peering partners. So, both peering sessions and interconnections over internet exchanges yield better internet traffic.

IXes make the internet as accessible as possible

If we speak from the perspective of AMS-IX, the main focus for IXes is to make interconnections as accessible and secure as possible. This is in line with their goal to make the internet a better place and to keep the quality of internet traffic high for all connected parties. They keep this quality high because the connections at AMS-IX are stable and the path taken is so short that the traffic is delivered in a good manner. Another factor is the fact that AMS-IX is a non-profit organization and, from that philosophy, has no commercial characteristics.

In contrast to AMS-IX, there are IXes that have a commercial interest in using their ports. One example is DE-CIX, a carrier and data center neutral IX with a business focus. Most internet exchanges are based on reciprocal agreements in which traffic is exchanged without compensation. A small sample of IXes fall between commercial and non-profit. These IXes are used, for example, for research purposes like BNIX, by a government agency, (MyIX) or are a provider of colocation (Equinix).

Recently, we expanded to Germany with a PoP location in Frankfurt at maincubes' FRA01 data center. That DE-CIX is a stone's throw away from maincubes is only a small part of the story. Frankfurt is perhaps the largest interconnection hub in the world. You can easily score an interconnection from all kinds of German ISPs. In addition to several nearby data centers, it's easy for us to peer traffic to German ISPs like Vodafone Germany, Telefonica Germany, and Liberty. By the way, this can also be done through a PNI. With the end goal of giving our customers the optimal path to the destination.

What does my IT organization gain from connecting with an IX such as DE-CIX?

There are several industries that benefit from ultra-low latency. Think of content delivery networks, managed service providers, telecom companies and international broadcasters. Some use cases within these sectors are social networks, user generated platforms, news platforms, gaming, and e-trading such as cryptocurrency. In the latter example, we refer solely to the servers that act as full nodes to make transactions faster. It is a selective group of customers that benefit from extremely low latency. A gain based on latency when connecting directly to an IX is only of extra importance for enterprise customers.

In specific cases, opportunities arise. For example, if a customer is connected to Speed-IX and has a lot of traffic with another party and with Worldstream. Then a peering session can be requested to let traffic run directly through that IX. We are connected to Speed-IX. Moreover, if an IX itself sees that there is a lot of traffic running between Speed-IX and Worldstream.

Connecting to an IX has the main advantage of being able to create links easily. Interconnections with connected parties are made in no time. Especially if you as an IT organization do not have the resources to set up your own peering session. The effect in such a case is that high quality internet traffic with low latency can be offered at a relatively low price.

So IXes, how exactly do these interconnections work?

Earlier we mentioned that with an IX you can set up direct connections with members. Whether this is possible has everything to do with whether the party is connected to the route server. Previously, if you were connected to an internet exchange it was not possible to exchange traffic right away, so for that purpose a BGP session had to be set up. Due to the increasing popularity of AMS-IX it was no longer practical to set up such session for every connected member. To resolve that, the route server was introduced. AMS-IX has two route servers, to maintain redundancy. We are connected to that route server and then advertise all routes coming from our network.

Then that route server causes interconnection between all other parties, including their advertised networks, that are connected to the AMS-IX. So those route servers act as a link between all these connected networks. As a result, you only need to be connected to two peers (the two route servers) to reach the entire network. A direct BGP session with an internet exchange, such as AMS-IX, is then no longer necessary. Still, this means you won't reach every party, because some IX users choose not to connect through those route servers. We are connected to the route servers of IXes AMS-IX, DE-CIX, Speed-IX, Asteroid and NL-ix. This enables a direct route between Worldstream, these IXes and the parties connected to them.

The actual interconnection can be made through a 1 Gbit/s, 10 Gbit/s or a 100 Gbit/s Ethernet port or a combination. What you often see is the use of Extreme Networks routers, specifically the SLX 9850. Such a system offers an exceptional number of ports and then forms the core of an internet exchange. To monitor the resilience of certain higher connections (10Gbit/s and 100Gbit/s), a Glimmerglass system is often used in conjunction with photonic cross-connect (PXC). The PXC connects a router to one of the Provider Edge routers.

Very strictly speaking, an IX is just a giant switch. More specifically, IXes are large Layer 2 LANs that in turn have multiple switches connected to them. When you connect to these you live within the AMS-IX platform and that's it. After that it is up to you to make the actual connections between other parties. They themselves do nothing with the actual connections, they act as an intermediary that facilitate these connections. However, there are exceptional cases like internet exchange DjIX, which is geographically the hub for submarine cables, like the AAE-1. As the name Asia-Africa-Europe suggests, that submarine cable stretches 25,000 kilometers across several continents. Again, this is an exceptional situation; laying dark fibers or submarine cables to extend the reach of the internet is not the responsibility of an IX. They are just making sure that that connection stays within that platform.

The next expansion of the global network backbone

Recently AMS-IX announced that they want to expand. There is a huge demand for data center space in Amsterdam, possibly because that is where AMS-IX is located. Internet traffic has become quite centered there. You can't just put another data center in that location if you want to. The fact that AMS-IX is going to expand to several locations is a consequence of that. This includes our sister company Greenhouse Datacenters in Naaldwijk. A remarkable fact is that AMS-IX specifically chose Greenhouse to set up a PoP there. The demand for data centers will of course remain, but this change might result in a more geographically spread demand for data centers. You don't necessarily have to colocate in Amsterdam anymore if you want to connect to an IX directly. Our data centers are located close to The Hague, Rotterdam, Leiden, Delft and Zoetermeer. We can now offer colocation space with the same advantages, which is useful for organizations who do not necessarily have a wish to be based in Amsterdam.

The AMS-IX has numerous members and plenty of nearby physical data centers. internet exchange DE-CIX is similar, with over 1,000 members. This German IX is a giant global interconnection hub in Frankfurt, where you have everything at your fingertips to set up a PoP. For example, there are at least fifty other data centers in the area. As a result, Frankfurt is a strong choice to house your IT infrastructure. However, not every IX is worth connecting to. Recently, IX.br is listed as handling the highest amount of traffic throughput worldwide and has a lot of connected members. However, if you look at the geographical locations it is clear why this IX was created. Namely, to improve the quality of the internet in Brazil, and in Brazil only. For organizations operating outside Brazil there would be no strategic advantage in connecting to IX.br, at least for the time being. A lesson can be learned here, make sure that new interconnections make sense when reviewing the bigger picture. In our case, theoretically speaking, expanding further within Europe would be more in line with the current strategy. For now, we have a link to two of the largest IXes in the world: AMS-IX and DE-CIX.