Defining the Boundaries of 'Storage' and 'Physical Network' in vSphere Infrastructure: From LUN Concepts to Physical Cabling
VMware has been a hot topic lately, and with commercial deployment becoming difficult for many Japanese companies, an increasing number of firms are migrating their virtual infrastructure.
That said, many companies still rely on VMware infrastructure, so it is likely that the probability of it being used, both as an entry point and for veterans, remains high.
I would like to discuss such best practices again this time.
Now, in the design and construction of server infrastructure, what engineers find most confusing is the 'storage connection form' and 'physical network accommodation design'.
Particularly in virtualization infrastructure (vSphere), the distinction between the storage that ESXi sees (datastore) and the storage that the guest OS sees (file server, etc.) tends to become ambiguous. Also, the 'physical level coordination' performed with the network team after the server chassis selection is finished is a phase that easily becomes a factor in project delays.
In this article, I will explain the key points of design that infrastructure engineers should keep in mind, focusing on the following two points.
The clear difference between datastores and external storage (NFS/SMB) and the concept of LUNs
Physical network design after server procurement (port accommodation, media type, MTU)
1. The decisive difference between 'Datastore' and 'External Storage'
The term 'storage' is broad, and the layer it refers to depends on the context. In a vSphere environment, it is necessary to clearly distinguish and design for the following two.
① Datastore: The 'land' for VMs
Connection entity: ESXi (Hypervisor)
Usage: System area for running virtual machines. This is where virtual disk files (.vmdk) and configuration files (.vmx) are stored.
Protocol: FC, iSCSI, NFS (for ESXi)
Image: The 'land' for building a house (VM). Without land, a house cannot be built. The C drive or root partition for the OS to boot always exists inside a VMDK file placed on this datastore.
② External Storage (Guest Mount): The 'warehouse' for VMs
Connection entity: Guest OS (Windows / Linux)
Usage: Data sharing area mounted after the OS boots. Business data, backup files, user documents, etc., are placed here.
Protocol: SMB (CIFS), NFS (general purpose)
Image: A 'rental warehouse' that you visit from your house (VM). The OS itself is on the datastore (land), but the application uses the warehouse (NAS) over the network as a destination to write data.
[Design Point]
If there is a requirement to 'build a file server', the architecture differs completely depending on whether the actual file is 'placed inside a VMDK (on a datastore)' or 'written directly to a NAS (Isilon/NetApp, etc.)'. In large-scale environments, the latter (②) tends to be chosen.
2. Intuitively understanding 'LUN (Logical Unit Number)'
When using block storage (FC/iSCSI), 'LUN' always appears. I will break down this concept, which beginners often stumble over, into a physical image.
What is a LUN? It is 'a single slice of a cut cake'
Enterprise storage (the chassis) contains dozens to hundreds of physical disks (HDD/SSD). These are treated as one massive pool (RAID group).
However, it is rare to hand over the entire pool (e.g., 100TB) to a server. Therefore, we logically carve out only the required size from the pool.
RAID Group / Pool: The entire whole cake
LUN: A shortcake cut into a single slice
A LUN is this 'carved-out logical disk' assigned a number. From the perspective of the server (ESXi), a LUN appears as 'a single physical HDD.' We format this (as VMFS) and use it as a datastore.
[Why split LUNs?]
If you cover everything with one massive LUN, it can cause performance degradation when I/O contention occurs. Also, to localize the impact of failures, it is a design theory to split LUNs (datastores) by purpose or cluster.
3. 'Physical Network Design' after determining server requirements
After the server specs (CPU/Memory) are decided, we enter the coordination phase with the network team (NW staff). What is important here is to solidify the discussion about 'physical (ports and cables)' before talking about 'logical (VLANs)'.'physical (ports and cables)'.
Port accommodation design (role division of physical interfaces)
Physical servers have multiple LAN ports. We define 'which purpose they serve and which switch they connect to'.
-
Management Port (Mgmt/OOB):
Purpose: ESXi management console (communication with vCenter), for SSH.
Bandwidth: 1Gbps is sufficient.
Importance: Even if the service is completely stopped, recovery work is possible if this is alive.
-
IPMI / iDRAC / iLO Port:
Purpose: Hardware management (BMC). Used for operations when power is off and for hardware monitoring.
Features: Operates independently of the OS. Usually connected to a dedicated management switch (ToR).
-
Service / Data Port:
Purpose: Business communication (Web/DB), storage communication (iSCSI/NFS), and vMotion.
Bandwidth: Uses high bandwidth such as 10Gbps / 25Gbps.
Configuration: For redundancy (teaming/bonding), always use 2 or more ports and connect them to separate switches.
P2P (Point to Point) and Cabling Tables
P2P here refers to 'the one-to-one connection relationship between a server's physical port and a switch's physical port.' While looking at the rack diagram, create a specific 'cabling table (Port Map)' such as 'Server A Port 1 to Switch X Port 5.' If this is incorrect, rework will occur during on-site work. Well, rework happens quite often, though.
Well, rework happens quite often, though.
4. Explanation of 'Physical/Link Layer' Terms Used on Site
Organizing basic physical layer terms that should not be left ambiguous in coordination meetings.
① Metal (Copper / RJ45)
Overview: General LAN cable (UTP cable).
Usage: Primarily used for 1Gbps management ports and IPMI. It is also used for 10Gbps (10Gbase-T), but because of its high power consumption and heat generation, it is becoming less common as a primary data center line.
Pros: Inexpensive and easy to handle.
② SFP / SFP+ / SFP28 (Transceivers)
-
Overview: 'Conversion modules' inserted into switch or NIC ports.
SFP: Supports 1Gbps
SFP+: Supports 10Gbps
SFP28: Supports 25Gbps
Optical Fiber: Enables long-distance transmission. Optical cables (such as LC connectors) are plugged into SFP modules.
DAC (Direct Attach Cable): Copper cables with SFP modules fixed to both ends from the start. They are standard for server-to-ToR switch connections for in-rack wiring within 3 to 5 meters due to being 'inexpensive, low-latency, and low-heat'.
③ MTU (Maximum Transmission Unit)
Overview: The maximum size of a packet that can be sent in a single communication.
Standard (Default): 1500 bytes. This is used for the Internet and general LANs.
Jumbo Frame: 9000 bytes.
Why it is important: When transferring large amounts of data, such as storage traffic (iSCSI/NFS) or vMotion, sending it in small 1500-byte chunks increases CPU load. Expanding to 9000 bytes reduces the number of header processing operations and improves throughput.
Points to note: Communication will not be possible unless the settings match across all server NICs, physical switches, and storage on the communication path.
5. Summary: Bridging from Requirements Definition to Physical Design
The value of an infrastructure engineer lies not in deciding catalog specifications, but in the ability to translate them into a 'reliably functioning physical implementation'.
Storage requirements: Distinguish whether it is a VM boot area (datastore/LUN) or a data storage area (external storage), and select the protocol accordingly.
Network requirements: Look at the rear ports of the server and create a diagram showing which physical cables (copper/fiber/DAC) will be used for IPMI, management, and service traffic, and which switches they will be connected to.
By reaching an early consensus on these two points with the network and storage teams, project rework will be dramatically reduced. The physical layer may seem mundane, but please remember that it is the foundation that supports the stability of the entire system.