Why Triangles are bad for your Network

You’re planning to work on the IoT, or doing a project with networks. Networks are awesome because they have emergent behavior. Let me tell you a story to show what I mean. This involves network infrastructure at the lowest, simplest layer.

You’ll need to understand the situation. I was managing a team of firmware and software engineers at a company that made Voice over IP (VoIP) phones. The phones we made worked over the WiFi network.

My boss was a great boss, and a good person. He was a no nonsense manager and had a reputation among people who did not know him as being a hard ass.

You see he had a way, when he knew you were full of crap and trying to bluff through a meeting, of asking simple questions to get to the truth. A super smart person, with broad experience, and a fine tuned bull detector is hard to fool. So he would start asking simple yes/no questions. He would start simply, “we can all agree that if the power is off the board won’t work, right?”

Now, that sounds reasonable in print. When you were being asked it was someplace between the Socratic Method and a Guantanamo Bay interrogation. I have been through it a number of times. I have witnessed full grown, experienced senior people on the receiving end of this interview method, with red burning eyes and a trembling lower lip. Tears have been produced.

The worst possible thing you could do is continue to argue. The only answers allowed were “yes”, “no”, or “I don’t know, let me go find the right answer.”

Now VoIP phones use the network for all the audio data in the phone calls. This was a smaller company, but growing fast. Part of our job was managing the corporate network, which was also the phone system. Eventually we had to upgrade our network. The hodge-podge of switches and network gear assembled over years was not holding up. We worked with the fledgling IT group and bought brand new Cisco Ethernet switches. They were big, 48 ports, with gigabit fiber for the switch to switch interconnect. The company spent big $$ to run a fiber across the street to the new building.

For a small company, we spent a lot of money.

Soon we started seeing some weird behavior. You need to know a little about Ethernet networks to understand.

Switches and Hubs

You can’t even buy hubs anymore. Ethernet is the lowest layers of the OSI model. It covers putting the bits onto the wire (the physical layer) and the media access layer (MAC). The standard for reference is 802.3. I will simplify for this example, everything is accurate, but not all the details are included. I am talking about modern Ethernet, with Cat5 and Cat6 wires and snap in plugs.

Every device on the network has a unique Ethernet address. The rules say “there shall be only one.” It gets programmed into the hardware. No duplicates allowed. An address is 6 bytes long, and usually shown in hex 00:2A:3B:11:22:33 for example. The first byte (00) is for a unicast device. Your computer’s address can by found using the command ipconfig in Windows, or ifconfig in Linux.

Broadcasts on a network use the Ethernet address FF:FF:FF:FF:FF:FF if you know hexadecimal numbers, that is all 1’s. Broadcasts get sent to every connected node in the network.

Think of Ethernet as an envelope on the wire. On the front of the envelope is the Ethernet header. It has the sender address, the destination address, and the length of the Ethernet packet. Just like an envelope, you can put a letter (the data) into the Ethernet packet and send it to the destination. The envelope does not care what is in your letter. A love letter or coupons, the post office does not care. The packet can not be too big, ~1500 bytes, just like the post office, an over-sized letter needs more postage.

There are two main types of devices to transport the Ethernet packets around on your network. A hub or a switch.

A hub is simple. It has multiple Ethernet connections where you can plug in the wires from your PC, XBox or whatever. When a packet is sent from your PC, the hub sends it out to all the ports (plug ins) on the hub.

Switches are smarter. Same thing with multiple connection ports for your wires. A switch will receive a packet, then send it only on the wire where the receiver address lives. So another PC on the network will never see a packet not for it. Cool, right. There is no traffic on a wire unless the device that is interested in the data lives on that wire.

Of course, you can plug switches into other switches, so there could be 50% of your college, lab, business, office on one port of a switch.

Broadcasts, of course, get sent to everyone.

Now, what was wrong?

We had everything installed. We had phones roaming all over our 3 buildings. We had gateways to turn the phone line audio into Ethernet packets, and we had a special box that did network magic to make the phones share the wireless network smoothly. People in the industry call this Quality of Service (QoS).

Part of my job was monitoring the phone network. The problem was, the monitoring software kept overflowing with packets. All over the network, phones were broadcasting randomly. We would see traffic from one phone sent all over the place, then it stopped. Sometimes it would come and go. Some phones never, some all the damn time. This totally hosed the wireless network. It was like being in a Skype call when someone else in the house starts watching Netflix while doing a Windows update. Things went to shit.

We had the Cisco FAE come out. Once, twice, four times. He could not explain it. This was brand new equipment. He had never seen anything like it. What was going on here?

On the last visit, we went into the conference room and drew network diagrams, looked a network sniffer traces and pulled our hair out trying to understand. WTF?

Then, my boss started. “Now, you know how a hub works?”

FAE, “yeah??”

“A hub sends all packets all over the network, right?”

“Yes.”

“A switch, especially the very expensive switches we just purchased and have not yet paid for should not send traffic all over the place, … right?”

“Well yes, but… ”

“Just yes or no, I want to go through all this”

“So yes”

“The phone sends to the gateway, the gateway sends to the QoS box, and it sends to the phone, right?”

“Yes”

Then it hit me. Like a bolt of sunlight filtering through the blinds. The bulb went on over my head. I said, “wait, we made a triangle!”

My boss looked at me with a don’t interrupt, while I extract a pound of flesh before we throw this guy out into the street with his defective boxes.

The FAE looked at me with a pleading deer in the headlights look.

I pointed to the board and said, “it’s a triangle.”

Why are triangles bad, I’ll explain in part 2.

Leave a Reply

Your email address will not be published. Required fields are marked *