DE-CIX NYC - 1 Tbps of Traffic. Read the full background story.

Let me take you back in time a few years. This is somewhat relevant in terms of understanding how we actually ended up taking a look at the New York Peering Ecosystem.

In 2009, the DE-CIX Team was pretty small. If I remember correctly there were maybe 10 or 12 of us. For most all of us that was a fairly busy year. During that time we onboarded many Eastern European networks, the result of a Sales effort which we started back in 2007.

My colleague Ivo Ivanov and I had begun travelling all over Eastern Europe to motivate Telecoms and ISP’s to join DE-CIX. I have to admit that was a very interesting experience. From large Telcos residing in their shiny new office buildings to companies that run their business from a shed in their backyard, we talked to all of them with significant success.

I do recall that Victor Belov from MTS/Comstar once invited us to an Irish Bar, where all of the Moscow CTO’s met once in a while. It was a freezing -20 degrees centigrade outdoors but after a few glasses of Vodka and Guinness, we started to conduct business. Lesson learned: selling in Eastern Europe is not a job for someone whose liver is not used to seeing a good amount of blood alcohol pass through.

However, from today’s perspective, something else was much more relevant. I remember I met some folks from the UAE regulator at the first ever MENOG conference in Bahrain in 2007. A year later, those guys would give me a call and ask for a Consulting offer to help them make the UAE an Internet hub.

We were all pretty surprised when we won this deal (and I would love to have seen the faces of the Detecon and ADL folks that lost). So we organized a team of DE-CIX consultants (to be quite honest, we were all experts in our field but none of us – maybe with the exception of our fellow colleague Gerd Simon – had any experience in Consulting). However, a little over one year later, working from both our field office at the Jumeirah Beach residences in Dubai and from Frankfurt, we presented the regulatory folks with a 369-page recommendation report. I joined the final presentation via a video conference, as my wife was expecting our baby imminently. That was my first video conference ever, and while our slides were not as brilliant as those they would have had from Detecon, the recipients were happy with the outcome.

(All flights cancelled, stuck because of a sandstorm in the middle east)

It was at that time that we realized we had just gone through the process of providing them with a blueprint for successful Internet exchange development. While I felt that my boss and DE-CIX Group CEO, Harald Summa, at times was not exactly happy about us being part of that project, he was the one who stated, “now we finally wrote down on a piece of paper what we actually can do”.

Moreover, while a great deal of effort was involved on the DE-CIX side, I believe that none of us were really surprised that we had planned and deployed the UAE-IX in Dubai. My friend, Ivo, did a great job in working with the Arabs to successfully negotiate that deal. UAE-IX eventually became the No.1 IX in the Gulf region and everything else that has been developed the the gulf region since is just a lame copy.

Marco Brandstaetter (Regional Director MEA) with Ivo Ivanov (CEO, DE-CIX Int).

So that small German group, which we were back then, had a blueprint to develop an IXP outside Germany and we actually operated an exchange outside of Germany. How cool was that?

So, we realized we can operate brand new greenfield exchanges in the desert. But where else could we add value? Maybe we could re-create that same success in a competitive market?

In 2013, I started to look at a number of markets. However, due to my time in the US as a peering guy for Deutsche Telekom, I was pretty familiar with the New York (NY) market. Equinix were doing their IBX Exchange, Telx (now part of DRT) had established their TIE exchange, but this was limited in terms of relevance and KDDI-owned NYIIX, run by some friendly Japanese guys in New York, was in the lead, not forgetting Shriharis “Big Ape” Exchange.

It was pretty obvious that the NY market was very fragmented and that the Data Center guys, who hated one other, would at no point manage to get together, merge those IX activities and get it right.

So, we carefully initiated certain conversations with “potential partners” for a DE-CIX project in the city. Obviously it wasn't our original goal to further fragment the NY exchange market - instead we wanted to lead the efforts to consolidate the market. The outcome was, in fact, rather dreadful. One large DC group would not even talk to us and another group in 60 Hudson did not know what to do with their exchange and seemed irritated by the fact that a group of Germans tried to initiate something. The Japanese made it very clear that they have a business to run and that they would rather consider some form of Harakiri than work with us. The owner of that smallish IXP told us that his wife did not like the idea of selling the exchange.

So far, this is not really a success story.

The only business case was one for Delta and Hampton Inn. I believe that in 2013, the biggest part of DE-CIX’ travel spending covered German-US travel.

On my way to the 165 Halsey Carrier Hotel in Newark

I think that at this point, Ivo Ivanov, Arnold Nipper, and myself were the only three guys in the company that thought it was probably still a good idea to move forward.

At that time, we were approached by a Canadian gentleman, who had just created a start-up in the US called *** (this is probably still under NDA ;-) . This group only had very little business, but they had very ambitious plans. So we tried to figure out what a US start-up and DE-CIX had in common and how we could jointly develop something in New York.

We learned that a VC-funded US start-up company and a German IXP, owned by a non-profit association, did not have anything in common. Zero. Nada.

Back to square one.

Now we had a really good idea of what was going on in New York, but sadly no partner. We were probably hoping for a little more air under our wings back then.

After completing the first drafts of what an exchange topology could look like on a piece of flipchart paper (engineers love it when non-engineers do that), Ivo and I started building a business plan.

With the help of Shay Flavin at Zayo, we managed to secure some good deals for most of the network and colocation we would need on Day 1. That included a good deal for “riser fibers” inside 60 Hudson and 111 8th Avenue, something that would turn out to be an important piece of our infrastructure.

The revenue element of the business plan was what we called a “guesstimate” We had only a rough idea how long it would take to make this the No. 1 Exchange in that metro. But I was fascinated by the idea that we could actually build this.

When we finally got internal approval to get “my exchange” deployed, things became interesting.

Topology of DE-CIX NY - first 10 sites - very neat, right ?

All of a sudden, our friends from the Netherlands made an announcement that they would deploy an exchange in the most non-dense carrier facilities in New York during an open-IX meeting. That was certainly interesting. I was irritated, and for a second, I asked myself if I had gotten something wrong. Then I asked why such a reputable player who knew that we intended to expand to New York would announce that move.

Nevertheless, the team around Daniel Melzer and Kay Schr?der started to deploy kit in our cages in 60 Hudson, 111 8th Avenue, 325 Hudson, 32 Ave of the Americas, 165 Halsey in Newark and a few other slightly less relevant sites. The best experience we had was with the guys in 165 Halsey. They had a great colocation product and we could even place the order for the inhouse cabling with the fiber group directly. No rip off, no MRC xconnects. That is how we like it.

(Myself with Rob Powell from TelecomRamblings in front of a DE-CIX' NY cage in 2014)

Sourcing all the bits and pieces and finally turning up the exchange was a major effort and paved the way to our launch event, scheduled to take place at the CNN Center in Atlanta during NANOG 60 in 2014.

To make things slightly more complicated, a major ice storm kicked in, and even on the day of the event, it was not guaranteed that the caterer would make it to the venue. Thanks to Carolin Geib for keeping everything under control.

This turned out to be a great event, hors d’oeuvres, champagne, a DE-CIX NYC trailer in the small CNN cinema, followed by my presentation and a tour of all of CNN. For me, this was my most memorable event in my DE-CIX Live. I do not have any words to describe how nervous I was. I really wanted everything to be perfect that night.

Needless to say, NANOG banned us from sponsoring or branding any of their conferences, since, apparently, our event was not approved by NANOG. Looking back I found that this was an interesting move. In the meantime, I believe we managed to spend some extra dollars with them.

The exchange went live in 2014, with only a few participants. DE-CIX Engineering did a great job and provided us with a rock-solid exchange. God knows some of the engineering was very basic on Day 1, lots of 1U switches and colored DWDM transceivers. All locations backhauled via VLANS to our 60 Hudson and 111 8th Avenue peering switches.

(DE-CIX Peering Nodes in cages @ Zayo 60 Hudson St & 111 8th Ave)

When I visited our POP in 325 Hudson Street, I remember that we had one cabinet and used 5 units, maybe 6 counting in the patch panel. A Production switch, a management switch, two passive DWDM multiplexers and an optical switch for backbone protection. Several feet down the hallway was one of the Dutch IXP nodes. A cage full of equipment in a site that hardly had any customers back in the day. It was obvious that having both of us operate in town was a rather bad development.

Andreas Sturm, who headed up Sales at DE-CIX, tried absolutely everything to get our NY business of the ground, but it was obvious that we needed to hire a local expert. I put a post on LinkedIn that we were looking for someone to run the show in the US. Ilissa Miller, whom I have known since the early 2000s when I worked for Deutsche and she worked for Telx, contacted me and recommended that I speak to a gentleman named Ed d’Agostino. It transpired that Ed was already one of my LinkedIn contacts; we believe we must have met at a PAIX peering event in the late 90's.

It was at the Metro Connect Event 2014 that Andreas and I sat down on the patio outside the event hotel and had a chat with Ed. It turned out that he liked the idea of helping us bring our ideas to fruition. He accepted the challenge and to this day, I am still delighted to have him on board.

Ever since, Ed has been fighting an uphill battle to make DE-CIX NYC the No.1 exchange in the NY/NJ metro, for much of the time single-handedly. He would tell everyone at HQ very clearly and very loudly about the level of support he needed and why those Germans are forever trying to get things done. This was brutal at times and I colleagues complaining frequently.

(Ed d'Agostino, Vice President & General Manager, DE-CIX North America Inc.)

In the end, both HQ and Ed must have done something right. Ed managed to onboard new participants every single month. He continuously related the DE-CIX story, regardless of whether suspects or prospects wanted to hear it.

In 2018, we passed 200 connected networks. We continued to add new sites and onboarded new partners. All the efforts paid off, and ultimately, DE-CIX became the No.1 exchange in the market. That was always the game plan. This is a “winner takes all business”.

Our Dutch friends decided to leave the NY market in 2019.

On a side note: Initially, I thought it was a joke that after we made our announcement to operate in Mumbai, the very same group also made their announcement to operate there. I have now come to the conclusion that they do not like us. As of January 2022 PeeringDB now shows 393 connected networks for DE-CIX Mumbai (aka Mumbai-IX) and 54 for the other group.

By adding more networks at DE-CIX NYC naturally, more traffic was handled. The fact that we have just hit the one terabit/sec peak traffic record is not that spectacular. It is the result of a plan, and I would have expected to see that traffic benchmark earlier. With over 260 networks, we are clearly bigger than our Japanese friends, serving 174 networks (as per the PCH IXP list 20 Jan 2022).

But as always, we were and are fighting very stiff competition. Moreover, while some of those DC representatives would probably still send me the same “no-bid” feedback when asking for a colocation offer, others have become loyal partners.

Many customers have become friends of DE-CIX. Some of them even act as ambassadors for our service, of which we are very proud.

We strongly believe in a carrier and data center neutral approach. We always strive to deliver the best service, superior to any other group in our niche.

I am proud to be part of a team of trusted colleagues, all of whom have been vital in making this project a success.

I would also like to say a big thank you to all those that have helped us become established in the Big Apple. Special kudos go out to Ed d'Agostino, Harald Summa, Ivo Ivanov, Felix H?ger, Daniel Melzer, Arnold Nipper, Klaus Müller, Carolin Geib, Andreas Sturm, Andrea Haberland, Rob Parker, Tobi Neumann, Ilissa Miller, Christine Keck, Jaymie Scotto, Shay Flavin, Phil Koblence, Steve Schecter, Brad Ickes, Hunter Newby, Eric Shepcaro ?,

We have emerged from a facilitator of Public Peering only to the operator of DE-CIX Apollon, the versatile swiss Army knife for Interconnection. Including a range of services tailored for the Enterprise. You want to learn more ? Reach out to me or any of my colleagues. We will be happy to explain our portfolio in detail.

You like this article ? Feel free to share it!

?

?