Networking: Data Center Physical layer

Single mode Vs multi-mode fiber:

Single-Mode Fiber (SMF):

Core Diameter: Small (around 9 micrometers).
Light Propagation: Single light mode, minimal dispersion.
Distance: Capable of long distances, typically over 10 km, up to 80 km or more depending on the transceiver used.
Transceiver for 25/100G: For 25G, use SFP28 LR (Long Reach) transceivers; for 100G, use QSFP28 LR4 or ER4 transceivers.

Multi-Mode Fiber (MMF):

Core Diameter: Larger (50 or 62.5 micrometers).
Light Propagation: Multiple light modes, more dispersion.
Distance: Shorter distances, up to 550 meters for 10G, reduced for higher speeds (100-150 meters for 25G/100G).
Transceiver for 25/100G: For 25G, use SFP28 SR (Short Reach); for 100G, use QSFP28 SR4 transceivers.

QSFP28 LR4	QSFP28 SR4
hot-pluggable	hot-pluggable
SM fiber(upto 10KM)	Multi-mode fiber(100 Meters)
WDM tech: 4 lanes. each lanes has different Wavelength	parrallel optics tech(4 lanes. each lanes transmit 25G)

Choose single-mode fiber for long-distance communication and multi-mode fiber for short-distance, high-bandwidth connections within data centers or buildings.

Transceivers: an optical transceiver is a module that converts electrical signals into optical signals and vice versa, allowing data to be transmitted over fiber optic cables.

Optical transceivers come in various form factors, such as SFP (Small Form-Factor Pluggable), SFP+ (enhanced SFP), QSFP (Quad Small Form-Factor Pluggable), QSFP28 (for 100Gbps), QSFP-DD (Double Density, for 400Gbps), and others. They plug into a port on a switch, router, or network interface card and are used in conjunction with fiber optic cables to enable high-speed data transmission.

Transceivers have an optical connector interface (like LC, SC, or MTP/MPO) for the fiber cable on one end and an electrical connector on the other end that connects to the networking equipment.

AOC vs MTP/MPO

An AOC (Active Optical Cable) is a cable that has fiber optic components and electronics embedded within the connectors at either end. These cables are designed to provide a high-speed connection between networking equipment, such as switches, servers, and routers. AOCs are typically used for short-range, multi-lane data communication and data center interconnects.

Key characteristics of AOCs include:

Integrated Transceivers: Each end of an AOC has an integrated transceiver that converts electrical signals to optical signals and vice versa.
High Bandwidth: AOCs support high data rates, commonly used for 10G, 25G, 40G, 100G, and higher speeds.
Plug and Play: AOCs are factory-terminated and tested, making them easy to install and use without the need for field termination or splicing.
Power Consumption: They typically consume less power than separate transceivers plus fiber cables due to the integrated design.
Distance: AOCs are suitable for short to medium distances, ranging from less than a meter up to 100 meters or more, depending on the quality and design of the cable.

On the other hand, MTP/MPO (Multi-fiber Termination Push-on/Multi-fiber Push-On) cables are a type of multi-fiber cable that terminates with an MTP or MPO connector. These connectors can house a specific number of fibers, typically 12, 24, 48, or more, in a single connector, allowing for the simultaneous connection of multiple fibers.

Key differences between AOC and MTP/MPO cables:

Active vs. Passive: AOCs are active cables with embedded electronics, while MTP/MPO cables are passive and only contain optical fibers without any active components.
Transceivers: AOCs have transceivers built into the cable ends, whereas MTP/MPO cables need to be plugged into separate transceivers at the equipment.
Use Case: AOCs are commonly used for direct-attach connections within racks or between adjacent racks, while MTP/MPO cables are often used for structured cabling solutions, backbone cabling, or to facilitate high-density fiber networks.
Flexibility: MTP/MPO cables can be part of a more extensive and flexible cabling system, including patch panels and break-out cables, allowing for various configurations and reconfigurations.

In summary, AOCs are active cables that come with built-in transceivers for direct connections between devices, while MTP/MPO cables are passive, high-density fiber cables that require separate transceivers and are used in structured cabling systems. The choice between the two depends on the specific requirements of the network, including distance, bandwidth, power consumption, and deployment flexibility.

Chipsets/ASICs:

The Tomahawk 4 and Jericho chipsets are both produced by Broadcom and are designed for different use cases within networking equipment such as switches and routers.

Tomahawk 4 Chipset:

Targeted Application: Data center networking, particularly for high-density, high-throughput environments.
Bandwidth: Designed to support high data rates, with versions capable of providing up to 25.6 Tbps switching capacity.
Port Speeds: Supports 1G, 10G, 25G, 40G, 50G, 100G, 200G, and 400G Ethernet ports.
Use Case: Ideal for cloud-scale data centers, high-performance computing, and AI applications where large amounts of data are moved within a data center.
Features: Focuses on high-speed data transfer with features like large on-chip buffers, low latency, and support for a large number of ports.
Port Density: Provides high port density, enabling more ports per switch.

Jericho Chipset:

Targeted Application: Service provider networking and carrier-grade routing.
Bandwidth: Offers lower overall switching capacity compared to Tomahawk 4 but includes features tailored for complex routing tasks.
Port Speeds: Supports a variety of port speeds, with a focus on 10G, 25G, 40G, 100G, and beyond.
Use Case: Suited for edge and core routers in service provider networks, where it’s necessary to manage traffic between different networks with sophisticated routing capabilities.
Features: Includes advanced features for carrier networks, such as deep packet buffers, hierarchical Quality of Service (QoS), large routing tables, and support for MPLS (Multiprotocol Label Switching) and segment routing.
Routing Capabilities: Designed to handle complex routing and traffic management tasks required in service provider networks.

Data-Center Physical:

Let’s say we have one building which has 6 Data Halls. Each hall is 5MW of the power available.

Terminology:

S0: Top of the rack switch. Once S0 can connect to upto 32 downstream servers and have 8 uplinks. We can utilise one S0 for upto 4 racks which are adjacent. If we have Dual NIC servers, we will place 2 S0 for 4 racks.
S1: the 1st layer of spine which connects to S0. S1 will have 40 downstream links and 8 uplinks. a total of 8 S1 will be present. A set of 40 S0 and 8 S1 will be called POD.
A single POD can have 32(one S0 will have 32 server) x 40(we will have 40 S1) = 1280 server in a POD
Considering S0 downlinks as 100G and uplinks as 400G. we will have 100 x 32/8 x 400 = 3200/

Rack placement:

In each DataHall, we can place 3 PODs(POD1 will have 40 S0, POD2 will have 40 S0 and POD3 will have 40 S0). Since each POD gives around 1280 servers, we will have 1280 x 3 = 3840 servers in one DataHall.

Let’s say we have A to T rows in the DataHall and each row can have 24 racks. In total, we can have 24 x 20 = 480 racks in the a datahall.

We can start placing 40 POD1 servers starting from row A. After POD1 servers are placed, we can have S1 devices in one of the ROW. We will have S1-0 rack which will connect to POD1 S0s, S1-1 rack which will connect to POD2 S0 and S1-2 rack which will connect to POD3 S0.

Rack power and width:

Rack Power	Rack Width
S2/S1: 16KW	S2/S1: 800mm
ODF: 0KW	ODF: 800mm

Spare: 16KW	Spare: 800mm
BMC: 16KW	BMC: 800mm
MGMT: 10.46KW	MGMT: 600mm
DCIM: 10.46KW	DCIM: 600mm
Security: 10.46KW	Security: 600mm
Server: 10.46KW	Server: 600mm