#VLSM (Variable Length Subnet Mask) SUBNETTING:
This is basically taking a large subnet and dividing it out into smaller subnets. This is important because you want to segment or VLAN off your traffic. So you’ll subnet into smaller subnets for data, wireless LAN, VoIP, and other subnets to be used in site VLANs.
Example for three remote sites:
10.1.0.0/16 VLAN11 - 10.1.11.0 = Accounting VLAN21 - 10.1.21.0 = HR VLAN31 - 10.1.31.0 = IT VLAN41 - 10.1.41.0 = Printers VLAN51 - 10.1.51.0 = Security VLAN61 - 10.1.61.0 = VOIP 10.2.0.0/16 VLAN11 - 10.2.11.0 = Accounting VLAN21 - 10.2.21.0 = HR VLAN31 - 10.2.31.0 = IT VLAN41 - 10.2.41.0 = Printers VLAN51 - 10.2.51.0 = Security VLAN61 - 10.2.61.0 = VOIP 10.3.0.0/16 VLAN11 - 10.3.11.0 = Accounting VLAN21 - 10.3.21.0 = HR VLAN31 - 10.3.31.0 = IT VLAN41 - 10.3.41.0 = Printers VLAN51 - 10.3.51.0 = Security VLAN61 - 10.3.61.0 = VOIP
I always like to make all my sites cookie cutter. You can see above we too a large /16 and sliced it into many /24’s.
Above is an example for several remote sites, not the Primary/Secondary Data Center. In the your Data Center you might have VLANs for Servers, Security, 3rd Party, etc… So your Data Center VLANs won’t be exactly like your remote site VLANs. With that in mind and like I stated above, I like to cookie cut the design. This includes the Data Centers. So your Remote Sites will be cookie cutter to each other and Data Centers will be cookie cutter to each other. This is assuming you have at your remote sites an MDF/IDF setup and then your Primary and Secondary Data Centers.
There might be situations where you are just over 254 IPs but under 510. A /24 won’t work. You could then use a /23 which has 510 IPs. This all comes down to planning your design. If you see a department or your ILO/MGMT close to 200 IP’s from the start, you might not want to start with a /24 and just start with a /23. So it all depends. You might just say, it’s OK if we run out of the /24, we can just create another VLAN because we designed it to grow that way. For example, let say you have another VLAN, 71 for ILO/MGMT which is 10.3.71.0/24. You are about to run out but it’s perfectly fine to create VLAN 72 10.3.72.0/24 and so on (VLAN 73 10.3.73.0/24). Like most things in IT, it all depends but take a step back and look at what you have and where you are going.
Subnetting is not rocket science, it just takes planning. You obviously don’t want any overlapping. If you have overlapping, you pretty much have two choices:
a. change the subnet to one that doesn’t overlap
b. NAT/PAT
Speaking of MDF/IDF and Data Centers. This is so flexible these days and used differently so use it however you like :). You’ll see that “most” companies will go with:
Remote Sites:
MDF – Larger than the IDF but not exactly a fully built Data Center. These will house the Demarcs, Routers, Switches, Firewalls, etc…
IDF – Closets where the Switches go. They fiber back to the MDF. So end user cable runs will run back to the IDF to a bunch of patch panels. From there they get patched into the Access switch. The Access switches are uplinked to the MDF distribution switch.
Data Centers:
These will house corporate Servers, SANs, VOIP System, Demarcs, sometimes Internet, etc… So basically these Data Center will host internal applications/services, Voice, VM environment, etc… This is changing where cloud is moving more and more applications/services out of your data center and into the cloud.
You won’t hear a company say, we have 200 Data Centers unless they are Facebook or Google because they actually do have a lot of fully built Data Centers. It’s usually we have our 2 – 4 Data Centers with a fully redundant MPLS network along with several large point to point circuits for replication purposes. The remote sites sit off the MPLS network. Again, it all depends :).
#IPv4 SUBNET ALLOCATION:
Going off of what’s above where we are allocating /24’s for departments, below are some good best practices for other allocations.
• Use private addresses for internal networks. This one I’ve been through where the company was using Public for internal and it was such a nightmare.
• Allocate /24 subnets like we did above for your users by department if possible. At least create a user VLAN. I like to keep my subnets as small as possible. The largest subnet I would use on a network is a /21. The reason is because switches FLOOD the network. It floods/broadcasts to find other hosts on the network. This gets worst with the more you have on the network.
• Allocate /24 subnets for VoIP devices (IP phones). You’ll also want to VLAN off things like Security devices, Video Conferencing devices and Printers.
• Use /30’s for point-to-point links.
• Use /32’s for loopback addresses.
• Allocate subnets for network management and server ILO/MGMT.
• Allocate subnets for remote access and VPN Tunnels.
• For the public facing networks use public IP Addresses.
• Make sure you reserve subnets for growth/scalability.
#VPN:
There are typically two types of VPN’s:
Remote Access – Users VPN into the network using a client
Site-to-Site or Business-to-Business – Company X creates secure tunnel to Company Y
• When going between your internal IP Addresses to a public external IP Address, use NAT and PAT as needed
• When servers on the DMZ need to be visible or accessible from the public, use Static 1-to-1 NATs
• When your internal users need access to the public internet, use PAT. This is many private addresses to one public address translation.
#SUMMARIZATION (DISTRO to CORE):
Now that you have built your networks with VLANs, you are going backwards here and doing the opposite from network routing point of view. What I mean by that is, lets say you are building out your VLANs like the example above
10.1.0.0/16 VLAN11 - 10.1.11.0 = Accounting VLAN21 - 10.1.21.0 = HR VLAN31 - 10.1.31.0 = IT VLAN41 - 10.1.41.0 = Printers VLAN51 - 10.1.51.0 = Security VLAN61 - 10.1.61.0 = VOIP 10.2.0.0/16 VLAN11 - 10.2.11.0 = Accounting VLAN21 - 10.2.21.0 = HR VLAN31 - 10.2.31.0 = IT VLAN41 - 10.2.41.0 = Printers VLAN51 - 10.2.51.0 = Security VLAN61 - 10.2.61.0 = VOIP 10.3.0.0/16 VLAN11 - 10.3.11.0 = Accounting VLAN21 - 10.3.21.0 = HR VLAN31 - 10.3.31.0 = IT VLAN41 - 10.3.41.0 = Printers VLAN51 - 10.3.51.0 = Security VLAN61 - 10.3.61.0 = VOIP
Beautiful…. but lets say you have 100 remote sites. Take 100 and times it by the number of VLANs you have. That number could get very high. In our example it would be 100 x 6 = 600. That’s 600 subnets that would get routed on your network. Instead of routing all 600, we could just “summarize” the routes to the /16 and only advertise 6 instead of 600.
Summarization helps reduce number of routes in the routing table, thereby reducing your routing update traffic and router overhead (received routes).
#THREE TIER DATA CENTER NETWORKS:
This design is going away thanks to ACI. ACI is a TWO TIER design. I’ll briefly go over both below.
Three Tier Design:
CORE>AGGREGATION/DISTRIBUTION>ACCESS
CORE>SPINE>LEAF
Two Tier Design:
SPINE>LEAF
Two Tier is also called “collapsed core” so might see that pop up. You’re “collapsing” the core and distribution layers into one.
CORE – Should be Layer 3:
• Maximize redundancy and route flow efficiency by using redundant triangle connections between switches.
• Use routing protocols that provides a loop free topology. I like using either OSPF or EIGRP at this layer but you can use BGP. BGP is not the best when it comes to convergence.
• Use Layer 3 and NOT Layer 2 hence using the protocols above at this layer.
• Use two equal-cost paths to every destination network.
DISTRIBUTION LAYER – Where you Layer 2 SVI’s sit:
• If layer 2 is used between DISTRIBUTION AND ACCESS layers, use first-hop redundancy protocols (FHRPs):
Hot Standby Router Protocol (HSRP) – Cisco Proprietary
Virtual Router Redundancy Protocol (VRRP)
Gateway Load Balancing Protocol (GLBP)
• Use Layer 3 routing protocols between the DISTRIBUTION AND CORE switches to allow for fast convergence. There’s also load balancing but be careful with this because I’ve had to fix an environment where they had EIGRP equal cost load balancing which was causing asymmetric routing due to mis-configuration which was causing problems.
• Peer with links you want to use as a transit or send your packets. Don’t create unnecessary links. Keep it simple with triangles.
• Use the distribution switches to connect Layer 2 VLANs that span multiple access layer switches.
• Summarize routes from the distribution to the core of the network to reduce routing overhead.
• Use Virtual Switching System (VSS) to eliminate the use of Spanning Tree Protocol and the need for an FHRP.
• Build Layer 3 triangles and not squares
• Use technologies VSS and/or VPC
ACCESS LAYER – Where your devices plug into (Mostly Servers in Data Centers and Mostly End Users, Printers, Phones in IDFs):
• Limit VLANs to a single closet when possible to provide the most deterministic and highly available topology. if possible, try to dedicate your VLANs to a IDF/Closet
• Use Rapid Per-VLAN Spanning Tree Plus (RPVST+) if Spanning Tree Protocol is required. It provides for faster convergence than traditional 802.1 default timers.
• Set trunks to ON and ON with no-negotiate.
• Prune unused VLANs to avoid broadcast propagation; this is commonly done on the distribution switch. VLAN Trunking Protocol (VTP) version 2 and version 3 automatically prune unused VLANs.
• Use VTP Transparent mode because there is little need for a common VLAN database in hierarchical networks.
• Disable trunking on host ports because it is not necessary. Doing so provides more security and speeds up PortFast.
• Consider implementing routing in the access layer to provide fast convergence and Layer 3 load balancing.
• Use the switchport host command on server and end-user ports to enable PortFast and disable channeling on these ports. Alternatively, you can use the spanning-tree portfast default global command.
• IMPORTANT!!! Design out your STP environment. I can write a whole blog post this. It’s important to know where your ROOT is going to be and the backup ROOT. You also want to use the Cisco STP built-in features:
PortFast: Bypasses the listening/learning phase for access ports. “spanning-tree port type edge trunk” is a command you might see a lot in a Nexus environment on Trunk ports.
Loop Guard: Prevents an alternate or root port from becoming designated in the absence of bridge protocol data units (BPDUs).
Root Guard: Prevents external switches from becoming root.
BPDU Guard: Disables a PortFast-enabled port if a BPDU is received.
• I would add here to try and avoid using Spanning Tree by using Fabric Path. Fabric Path uses ISIS and it doesn’t rely on STP.
#VTP:
First why use VTP in the first place. VTP just a way to setup a VLAN database and share it with other switches in the same domain. You only need to make changes to one switch for VLANs, and that gets propagated to all other switches.
Always use VTP v3 if possible. Below are some of the benefits for v3. If you need to run v2, configure all switches in a VTPv2 domain in Transparent mode. ALL vlan changes in this mode are local. For v3 turn off VTP since that’s now an option.
What’s new in v3:
• IMPORTANT!!! Fixes the issue with VTP v1/2 where the higher revision number wipes out the database and brings down the network. This is a biggie because I’ve seen this happen. Basically, VTP will use the VTP database with the highest revision number. Well, if someone plugs an old switch into the network with a higher revision number, that switch will be the master. Down goes the network. The revision number goes up after each config change. So older switches usually have higher preference numbers. The best thing to do is to put ALL new switches into Transparent mode this way the Revision number goes back to 0. Then configure the VTP settings.
• Supports extended VLANs (1006 – 4094)
• Private VLAN propagation
• MST is now advertised
• Support for flagging VLANs as RSPAN (disables MAC learning on the VLAN)
• VTP can be turned off as opposed to just transparent mode
• VTP password can be hidden or secret