Ever wondered who’s on the other end of your Cisco Service Request? well wonder no more as I put on my CiscoChampion hat and play journalist for the week at Cisco Live Milan
When you next look up at the night sky, you may see a bright spec in the distance, and that bright spec is set to get a lot brighter.
The spec of which I speak is Software Defined Networking (SDN) and is set to change the network as we know it forever and perhaps a lot sooner than first thought.
With the “commoditisation” of Pure SDN solutions and hybrid SDN solutions which also harness custom ASICS, things will change! Maybe not today, not tomorrow but they will change.
We have plenty of warning about this meteor strike, not to try and divert it, as impact is inevitable, but we have fair warning to prepare for it and to evolve our traditional networking skill set in time.
I do not see the results of this strike, being an immediate extinction level event for traditional networkers but more like a huge lake gradually drying up.
At the moment the lake is huge and teaming with life but gradually as businesses move towards SDN solutions, the traditional networking lake will slowly start to dry up until a few who are unwilling to adapt are flapping in a pool of mud awaiting their imminent fate.
This is not by any means meant to be a doom and gloom “End of the traditional networking world is nigh” type post, but a positive post that the networking world is about to get real interesting and bought kicking and screaming into the modern world of flexibility, agility and fast provisioning. And I for one am not close enough to retirement age to ignore it, and am actually quite looking forward to the new Challenge.
Having attended Cisco Live Europe and VMware PEX this month, I’ve spoken at length to the relevant business units, and I am very much encouraged by the commitments and training road maps being put in place to bring us “Traditional Networkers” on this new and exciting journey ahead.
Your indispensible guides to making your IT life simpler.
So what is VXLAN and why do we need it?
Well put simply it’s VLAN with an X in the middle :-) the X standing for eXtensible. VXLAN was a joint project between Cisco, VMware, Redhat and Citrix which is why it has been so widely adopted, and underpins the majority of SDN offerings.
And as to why we need it, well that’s mainly to address two limitations of using regular VLANs. Scale and Flexibility.
As we all know standard 802.1Q VLANs scale to just over 4000 VLAN Ids, and while that number sounds a lot and is fine in most cases, large Service Providers, Enterprises and Multi Tenant environments ,would certainly need more.
VXLAN encapsulates the standard Ethernet frame and adds a header to it including a 24bit VXLAN ID field which increases the number of VLANs from 4096 to 16million logical segments, while only adding approx 50 Bytes of overhead to the frame (udp header)
In this world of ever increasing workload flexibility and agility we need a way of quickly and safely providing connectivity between Virtual Machines anywhere in the network where we have capacity.
Historically this was done by extending VLANs everywhere that a Virtual Machine may be required. This as we all know comes with a raft of potential issues around Scale, Complexity and resiliency
As the Layer 2 Frame is encapsulated into an IP Packet it can now cross Layer 3 boundaries! This opens up a whole raft of use cases.
These use cases include, but are certainly not limited to:
• Running layer 3 all the way to the edge of your network then mapping your VXLANs over the top (overlay) getting the best of both worlds of a L3 transport but Layer 2 adjacency / reach ability wherever you need it.
• Extend your Layer 2 into any Public/Hosted Cloud allowing you to move VMs in and out of a hosted service as and when you need to. (Cloud Burst)
• Extending a VLAN over a Layer 3 Data Centre Interconnect (DCI) for Disaster Recovery (DR) to allow VM mobility between Data Centres.
Also IP packets make much better use of Port-Channelled links unlike other encapsulation technologies like MAC in MAC.
So how does VXLAN work?
The VXLAN enabled switch (The Nexus 1000v VEM in my example below) learns the VM’s MAC Address, and the assigned VXLAN ID; it then encapsulates the frame according to the port profile the VM is assigned to.
When the VM first comes online the VEM assigns it to a defined Multicast Group, which carries all, Broadcast, Unknown Unicast and multicast traffic (B/U/M). Known Unicasts are sent directly to the correct destination VEM/port.
Although all VMs/Tenants are assigned to the same Multicast group the VXLAN segment IDs are used to only deliver traffic to the same VXLAN thus maintaining and ensuring tenant separation.
The resulting VXLAN “tunnels” terminate at either end on the VXLAN enabled Switches the VM’s/Servers are connected to. These Switches are referred to as Virtual Tunnel End Points (VTEPs)
Figure 1 below shows the VXLAN encapsulation (Wrapper) put around the original Ethernet frame.
Figure 1 VXLAN Encapsulation
The Outer IP’s added by the VEM are for the VTEPs, VTEPs can be a virtual switch residing in a hypervisor like the Nexus 1000v or a logical switch residing in a physical switch.
If you want to “break out” of the VXLAN and have your VM talk to a Bare Metal device or a gateway for routing then a VTEP Gateway is required. This VXLAN gateway has an interface in the VXLAN and an interface in the classical Ethernet VLAN then bridges between the two.
Examples of VXLAN gateways are the Cisco ASR1000v/CSR1000v or the VXLAN Gateway Services Module for the Nexus 1110/1010 Virtual Services Appliance. Some VXLAN enabled physical switches are also capable of providing VXLAN gateway functionality.
As mentioned above VXLAN relies on having an IP Multicast Enabled network between VTEPs.
There are 2 Cisco (non IETF) enhancements which negate the need for an IP Multicast enabled network.
1) Head-end software replication.
The VTEP (Nexus 1000v in my example) sends a copy of the B/U/M Traffic via unicast to all possible VTEPs on which the destination MAC could be located. (works well for smaller deployments)
2) The second solution relies on the control plane of the Nexus 1000V virtual switch, the Virtual Supervisor Module (VSM), to distribute the MAC locations of the VMs to the Nexus 1000V Virtual Ethernet Module (VEM, or the data plane), so that all packets can be sent in unicast mode. While this solution seemingly conflicts with the VXLAN design objective of not relying on a control plane, it provides an optimal solution within Nexus 1000V-based virtual network environments. Compatibility with other VXLAN implementations is maintained through IP Multicast, where required.
VXLAN Configuration example:
First Ensure IP multicast is enabled on the switch and SVI interfaces.
Ip pim sparse-dense-mode (on the L3 interfaces)
Ip pim birdir-enable (recommended as any endpoint could be a sender or receiver)
Ip send-rp-announce Loopback0 scope 16 birdir (sets switch up as an RP)
Ip pim send-rp-discovery Loopback0 scope 16
Verify with “sh ip pim interface” and “sh ip pim rp map”
On Cisco Nexus 1000v VSM
Feature Segmentation (enable VXLAN Feature, requires advance license)
Segment id 5000
Create the Layer 3 control interface uplink port-profiles for the VEMs
Port-Profile type vethernet Control_Uplink_1001
switchport mode access
switchport access vlan 1001
system vlan 1001
Port-Profile type vethernet Control_Uplink_1002
switchport mode access
switchport access vlan 1002
system vlan 1002
Create the Port-Profile the VMs will connect to:
Port-Profile type vethernet VXLAN_5000_Tenant1
switchport mode access
switchport access bridge-domain 5000
Verify on VSM with
Show bridge domain
Verify on Switch with
Sh ip mroute 184.108.40.206
First test with both VM’s on the same host/port-group then vMotion VM2 to ESX2
VXLAN Packet Walk
Let’s take the above example and do a PING from VM1 (MAC1) on ESX01 to VM2 (MAC2) on ESX02
1. Virtual machine VM1 on ESX01 sends an ARP packet with Destination MAC as “FFFFFFFFFFF”
2. VTEP (VEM) on ESX01 encapsulates the Ethernet broadcast packet into a UDP header with Multicast address “220.127.116.11” as the destination IP address and VTEP address “10.200.1.150” as the Source IP address.
3. The physical network delivers the multicast packet to the hosts that joined the multicast group address “18.104.22.168”.
4. The VTEP on ESX02 receives the encapsulated packet. Based on the outer and inner header, it makes an entry in the forwarding table that shows the mapping of the virtual machine MAC address and the VTEP. In this example, the virtual machine MAC1 running on ESX01 is associated with VTEP IP “10.200.1.50”.
5. The VTEP also checks the segment ID or VXLAN logical network ID (5000) in the external header to decide if the packet has to be delivered on the host or not.
6. The packet is de-encapsulated and delivered to the virtual machines connected on that logical network VXLAN 5000.
7. Virtual Machine MAC2 on ESX02 responds to the ARP request by sending a unicast packet with Destination Ethernet MAC address as MAC1.
8. After receiving the unicast packet, the VTEP on Host 2 performs a lookup in the forwarding table and gets a match for the destination MAC address “MAC1”.
9. The VTEP now knows that to deliver the packet to virtual machine MAC1 it has to send it to VTEP with IP address “10.200.1.50”.
10. The VTEP creates unicast packet with destination IP address as “10.200.1.50” and sends it out.
11. The packet is delivered to ESX01
12. The VTEP on Host 1 receives the encapsulated packet. Based on the outer and inner header, it makes an entry in the forwarding table that shows the mapping of the virtual machine MAC address and the VTEP. In this example, the virtual machine MAC2 running on ESX02 is associated with VTEP IP “10.200.2.50”.
13. The VTEP also checks segment ID or VXLAN logical network ID (5000) in the external header to decide if the packet has to be delivered on the host or not.
14. The packet is de-encapsulated and delivered to the virtual machine connected on that logical network VXLAN 5000.
I will do a Video walkthrough on how to set VXLAN up using my Cisco UCS and Nexus 1000v and Nexus 5000 Lab and post here when done.
Thanks for stopping by and look after that Datacenter of yours :-)
Anyone with more than a passing familiarity with Twitter will no doubt have seen a hashtag whizz passed entitled #CiscoChampion or even #CiscoChampion(s) (more on the latter later)
So what does this mean? well you may well be familiar with other vendors Advocacy programs like “EMC Elect” or VMware “vExpert”, well “Cisco Champion” is Cisco’s.
There are several “Flavors” of Cisco Champion I for example am humbled and proud to be a Cisco Champion for Data Center.
How did I become a Cisco Champion? well you have to be active in the Social Community and be willing to “Give Back” to the community and give those in the community the benefit of your knowledge and experience. What form this takes is not fixed but it could be a blog or via Twitter or the Cisco Community sites, or a combination of all 3.
So what’s changed for me since becoming a Cisco Champion? well quite a lot really, not only do I feel more empowered, but also that I now really have a voice (or at least one that people listen to) as well as getting a lot more “Cisco Love”; not to say that I didn’t get any before, as working for a Gold Partner I certainly get my share.
But since becoming a Cisco Champion this “Love” has increased to a whole new level!
By “Cisco Love” I mean access to Betas, inside scoops, early blogger briefings, guest blog spots, participation in great events and promotions etc.. etc..
So does this mean Cisco have now bought my Soul, and that I am no longer able to blog objectively?
Is it like Cisco have driven my punk ass to the Vets and laid me on that table and made the unkindest cut of all? Removing any last shred of independent thought or dissidence.
Hell no!, as a Cisco Champion and advocate our objectivity is what Cisco require, they want it, they need it, they crave it.
After all a constructive criticism from an advocate in any walk of life is an opinion really worth listening too.
All Cisco Champions are told to continue to be themselves in their online social activities, and make it clear they are NOT Cisco representatives.
And as for the #CiscoChampion(s) hashtag, well it’s because we are plural i.e. there’s more than one of us but the logo is singular, and we have to match the logo. Nothing more interesting than that I’m afraid.
Nominations for the latest flavor of Cisco Champion close on January 24th so if you know anyone who could be a Cisco Champion for Enterprise Networks then recommend them at the below link.
You can find more information on the Cisco Champions program here: http://www.cisco.com/web/about/facts_info/champions.html
I’ve been meaning to do this video for ages and finally had some time to do it.
The most common questions I tend to get are generally around booting a Cisco UCS Server from the SAN. Now in order to take full advantage of the statelessness of Cisco UCS servers we certainly want to avoid any dependency on a blade and SAN Boot is a great way to do it. And in Cisco UCS it’s an absolute dream to set up.
But I’ve decided not to just stop at the Cisco UCS config but also include the SAN switch config and the Array config. Why? you may ask. Well in this day of ever increasing convergence roles are merging and Silos crashing so it makes sense to have a good overview of the entire process. And even if all these elements are still conducted by separate admins in your environment, well it’s still great to have an appreciation of the information they need in order to work closer and more efficiently with them. I have seen too many cases when trying to troubleshoot a Boot From SAN issue (or any issue for that matter) when different admins did not communicate with each other and used different (not wrong) naming conventions etc.. and it just made end to end troubleshooting that bit harder. The more consistent we can make something by working together and sharing information certainly makes everyone’s job allot easier.
Anyway grab yourself a Scotch and sit back and let the next 60mins wash over you, it always goes down smooth.
Last week saw the latest major update to UCS Manager in the form of version 2.2 codenamed “El Capitan”
It certainly doesn’t seem a year since I wrote the summary for the then eagily awaited 2.1 “Delmar” release” but I guess time really does fly when your having fun!
UCSM 2.2 will be the last Major version to include support for Generation 1 hardware. 6100 FI’s, 2104 IOM, M1 Servers and M1 Only Adapters. As such it is expected to be a long-lived release, so expect patches and major bug fixes for approximatley 12 months longer than normal major releases (Circa 4 years).
Remember that Cisco offer the “UCS Advantage Trade in program” which provides an easy path in which to upgrade Generation 1 hardware to the latest versions.
USCM 2.2 Features Overview
If you have ever been a bit frustrated that loading a huge bare metal ISO to a CIMC took a while as you had to go via the 1Gbs FI MGMT port then this should make you happier. With UCSM 2.2 it is now possible to optionally access the CIMC of M3 blades over the same in band network as the data path giving access to all those those lovley 10Gb uplinks. You may also have a requirement to seperate UCSM Management traffic from CIMC Management traffic well now you can. CIMC Out of band is the same as it was you just have the option of connecting to either the In Band or Out of Band CIMC Address. CIMC In-band access supports KVM console, vMedia & Serial over LAN (SoL)
Trusted Platform Module (TPM) Inventory:
Allow access to the inventory and state of the TPM module from UCSM (without having to access the BIOS via KVM).
Well thats about it, hope there is somthing in this update for you, there sure is for me :-)
If you have ever wanted a sneaky peak under the UCS Kimono (GUI) then this posts for you.
The goal of this post is to clarify the end-to-end path from a Cisco UCS vNIC through the UCS Infrastructure to the point we egress from the Cisco UCS Fabric Interconnects.
Having this information and being able to check utilization and statistics of all virtual and physical interfaces within the Cisco UCS environment will save you allot of time and give you a much better understanding of how all the elements tie together.
This post builds on from a previous post “Understanding UCS VIF Paths” where we used a combination of the GUI and CLI to establish the end-to-end traffic path used by a vNIC/vHBA. In this post we exclusively use the CLI, so if you haven’t done so already perhaps worth checking the previous post out first.
Anyway I was troubleshooting an intermittent performance issue the other day from a Cisco UCS Blade all the way back to the Storage Array. And thought it would make a useful post to document this part of the process.
And certainly if you ever get as far as needing to open a Service Request (SR) with Cisco, being able to provide the below information will save you and TAC allot of time.
During this process I will be attaching directly to the ASICs within the IO Modules and these ASICs differ depending on whether you are using Generation 1 or Generation 2 Hardware.
As a nice “Cheat Sheet” I have provided the below table and graphic to show the relevant Cisco UCS ASIC and code names some of which we will need for this process.
In this example we will confirm the end-to-end path of a vNIC named vNIC_FabB1 of Service Profile DCN4PBKW001 which is in Chassis 3 Slot 2
First determine the Virtual Interface (VIF) and Which Fabric the vNIC/vHBA is currently using (If Fabric Failover is enabled)
Command “show service-profile circuit server <Chassis #>/ <Slot #>”
As we can see vNIC_FabB1 is Active and Primary on Fabric B and Passive and Standby on Fabric A. Therefore we can determine that the Active Fabric for this vNIC is Fabric B.
We can also see that the VIF associated with this vNIC is VIF 2024.
SideNote) This vNIC will connect via a Virtual Network Link (VN-LINK) to a vEth port of the same name vEth2024 on Fabric Interconnect B which can be viewed and statistics collected via the Connect NXOS command at the UCSM CLI.
The next thing to determine is which internal IOM (FEX) Port VIF 2014 is using.
From the UCSM CLI
Connect iom <Chassis #>
Show platform software woodside sts (use “redwood” for IOM 2104)
I love the above command because it shows a representation of exactly how the FEX is being used. You can see all the Internal Blade Facing “Satelitte” ports or Host Interfaces (HIFs) and all the External FEX Network Ports or (NIFs).
As can be seen Blade 2 has access to internal FEX ports 3&4 but has only one active connection to the FEX on FEX Port 3, which maps to HIF 27 (Outlined in Red)
NB) The reason FEX Port 4 is disabled, is that the ports of a 220x FEX alternate between the mLOMs and the Mezzanine slots of the Blades, the Mezzanine slots in the above example being empty (hence all alternate (even) ports display as “–“ for Disabled)
Now we know which Host Interface (HIF) we are using, we next need to determine which FEX Network Interface (NIF) is being used.
If you are using a Port-Channel between the FEX and the FI all servers will be mapped to the port-channel and distributed over the members by the LACP algorithm.
In this case the FEX Links have been left at the default setting which is “Discrete Pinning” mode and as such then the relationship between server slot and FEX Network Interface is as follows:
So as can be seen above FEX Port 3 maps to Network Interface 2.
The HIF to NIF mapping differs depending on the IOM used and how many FEX cables are actually connected, the above shows all four links of a 2204XP connected, the below example shows how the HIF to NIF mapping occurs if 2 FEX Cables are used:
So Blade 2 (FEX Ports 3 &4) maps to Network interface (NIF 2)
OK so next we establish which Server Interface (SIF) on the Fabric Interconnect we are using , which we do with the below command:
Show fex <Chassis #> detail
So as you can see FEX Port E3/1/3 is using FI Fabric Port Eth1/10
The last port we need to know is which FI Uplink port we are pinned to
Show pinning server-interfaces | inc Veth2024
NB) show pinning border-interfaces active can also be used to see the information from another perspective.
As you can see Veth 2024 is pinned to FI Uplink Port-Channel 11
So armed with all the above information you can draw out all the ports in the Cisco UCS traffic path. This in itself will save a lot of time if you need to engage TAC.
NB) For further information on advanced Cisco UCS Troubleshooting at the CLI I would strongly suggest checking out the recorded session by Robert Burns (First CCIE DC, TAC and Cisco Community Legend) available for free at https://www.ciscolive.com just set up an account.
BRKCOM-3002 – UCS Performance Troubleshooting
While not perhaps the most interesting topic for some, this is a post I have been meaning to do for some time, and the recent Intel E5-2600v2 CPU Additions into the Cisco UCS lineup have kicked my butt into writing this post.
Like most blogs, this site started off purley as an online respositpory for my own reference, and if the infomation helped someone else, then hey happy days.
One of the most enjoyable aspects of my job is training internal staff and external customers, and as such not only am I required to have good practical skills but also good classroom theory.
In every Cisco UCS Course I deliver, I always give a session on Intel processor architecture and how the Intel CPU’s have evolved and how that evolution matches into the Cisco UCS product line.
In the “old days” this was easy; an M1 Blade = Intel XEON 5500 (Nehalem) and an M2 Blade = Intel XEON 5600 (Westmere), then came the Nehalem EX (6500/7500) the Westmere EX (E7-2800 and E7-4800) , the Sandy Bridge E5′s and now the Ivy Bridge E5′s. And with all these numbers and codenames flying around it is no surprise that people can get a bit confused.
This prompted me to knock up a nice little “Crib Sheet” on what processors are used in what models along with their codename and official launch name designators.
Intels Processor evolution happens in two steps a “Tock” which is a microarchitecture change and then a “Tick” which is the same microarchitecture only made smaller. For example the Cisco UCS journey began using the Nehalem Microarchitecture “Tock” on a 45nm High-K Process, then came the Westmere “Tick” where the process was shrunk to give us the same Nehalem Microarchitecture but this time on a 32nm High-K process. This reduction in process size usually is coupled with an increase in core count due to the fact that as the technology is made smaller, Intel can fit more cores onto the die.
Intel also have certain “Segments” or types of CPU’s which are EN, EP and EX
EN = Entry Level (Used in B22M3)
EP = Efficient Performance (2 Socket)
EX = Expanded (up to 4 Socket with Expanded memory architecture)
So all the above leads nicely into the below crib sheet. which details the Microarchitecture, Process Size, The Cisco UCS Server it is used in, and the Maximum Core/Memory it can support.