Figure 1

I will also detail the relevant CLI commands to confirm, and troubleshoot the complete VIF (Virtual Interface) Path.

If you have ever seen and understood the film “Inception” you should have no problem understanding Cisco UCS, where virtual machines are run on virtual hosts which run on virtual infrastructure and abstracted hardware but in all seriousness it’s really not that complicated.
The diagram below shows a Half width blade with a vNIC called eth0 created on a Cisco VIC (M81KR) with its primary path mapped to Fabric A. For simplicity only one IO Module to Fabric Interconnect link is shown in the diagram, as well as only one of the Host Interfaces (HIFs / Server facing ports) on the IO module. In this post I will focus in on eth0 which is assigned virtual circuit 749.

Figure 2

Virtual Circuit
First column in figure 1, Virtual Circuit number, this is a unique value assigned to the virtual circuit which comprises the virtual NIC, the virtual cable (red dotted line in figure 2) and the virtual switch port. The virtual switch port and virtual circuit have the same identifier in this case 749.

If you do not know which virtual circuit will be used for the particular MAC address you are interested in, or which Chassis and Server that virtual circuit resides on, you can use the below commands to find out.

Figure 3

The above output shows that the MAC address is behind Veth749, now in order to find out which Chassis and Server is using Veth749 issue the below command.

Figure 4

The Interface to which Veth749 is bound to is Ethernet 1/1/2 which equates to Chassis1 Server 2 (you can ignore the middle value) the description field also confirms the location and virtual interface name on the server (eth0)
As you know, (having read my blog post on “Adapter FEX”:-) ) the M81KR “PALO” adapter is actually a mezzanine fabric extender just like the IO Module (FEX) in the Chassis. What this means is when I create a virtual interface on the adapter, that interface is actually created and appears as a local interface on the Fabric Interconnect (FI), whether it’s a vNIC which appears as a Veth port on the FI or a vHBA which appears as a Vfc interface.
This means we will have many virtual circuits or “virtual cables” going down the physical cable, Cisco UCS obviously needs to be able to differentiate between all these “virtual cables”, and it does so by attaching a Virtual Network TAG (VN-TAG) to each virtual circuit. This way the Cisco UCS can track and switch packets between virtual circuits, even if both of those virtual circuits are using the same physical cable, which the laws of Ethernet would not normally allow.

Adapter Port
The Cisco VIC (M81KR) has two physical 10Gbs traces (paths / ports) one trace to Fabric A and one trace to Fabric B. This is how the VIC can provide hardware fabric failover and fabric load balancing to its virtual interfaces.These adapter ports are listed as 1/1 to Fabric A and 2/2 to Fabric B.

In the case of a Full Width blade, which can take 2 Mezzanine adapters this obviously doubles the number of paths to four.
In the case of the VIC 1240 and VIC 1280 which have 20Gbs and 40Gbs to each fabric respectively there is still only a single logical path to each fabric as the links are hardware port channels 2x10Gbs per fabric for the VIC 1240 and 4x10Gbs per fabric in the case of the VIC 1280.

The new M3 servers which have LAN on board (mLOM) provide additional on board paths.

Fex Host port
In the lab setup I am using, the FEX modules are 2104XP’s which have 8 internal Server facing ports (Sometimes referred to as Host Interfaces (HIFs)),which connect to the Blade slots, port 1 to blade slot 1, port 2 to blade slot 2 and so on

Fex Network Port
The 2104XP IO Modules also have 4 Network Interfaces (NIFs / Fex Uplinks) which connect to its upstream Fabric Interconnect.

Figure 5

Fabric Interconnect
The 2 FI interfaces listed in Figure 1 are FI Server Port and FI Uplink

The server facing ports on the Fabric Interconnect are called FI Server Ports and can be confirmed in the output of the “Show Interface fex-fabric” command. The FI server ports are listed in the second column “Fabric Port”. In figure 5

The FI Uplink interface can be found by checking the pinning of the Veth interface.

Figure 6

As can be seen from the above figure, Veth749 is pinned to FI Uplink (Border Interface) Port-Channel 1

Armed with all the above you should now have the information necessary to understand the packet flow within the UCS and be able to troubleshoot as well as monitor and understand failover.

Hope this helps.

But as the Banner of this site says “Independent site for Cisco UCS”, I have listed my top 5 Things / feature requests I would like to see addressed in future updates. Now I’m not talking major features or funcionality enhancements which Cisco either already has in its roadmap or will no doubt be addressing anyway, but more about the little things that would just be nice to have.

Now not being a programmer some of my “wish list” may not be as simple as it sounds, but hey if you don’t ask etc..

1) Rename service profiles
While I love the power of templates, it is an annoynace that I don’t have full control of the service profile names generated therefrom, or be able to rename them afterwards. Instead I find myself cloning service profiles (Clones can be freely named) and then binding them to an updating template. Not a huge effot I know but still. This has been addressed in part,by being able to attach a user defined label to service profiles, but these labels do not show up on a KVM session or KVM Launch Manager. (One of the few downsides of an XML based system perhaps.)

2) Negate View privilage
Cisco UCS is very well suited to host Multi-Tenant environments, and RBAC and locales are great features for controlling access but currently “Tenant A” admins, still have read only visability to “Tenant B’s” policies and resources. A “no access” privilage or filter would be nice to negate even view rights to particular locales. This should also include KVM visability, i.e. only service profiles within a locale should be visable via the KVM console to users assigned to that locale. Now I appreciate that in a lot of cases even multi-tenant environments the UCS admins are common, but there many times they are not, even different business units within an organisation may have different UCS admins, and especially KVM users.

3) vHBA passthrough to a VM in ESXi.
I have had occasions when I have wanted to pass through a vHBA created on a Cisco VIC to a VM, Tape Library control being the last requirement. This however is not supported (yet) but I guess this may well just be a limitation of I/O virtualisation at present not just Cisco UCS. For this use case using VMware VMDirectPath I/O on a vHBA would be required rather than a vHBA equivilent of VM-FEX.
Only UCS options were to use a small bare metal blade, which for a very low workload seemed a bit of a waste, or use a small C series rack mount.

4) Windows Teaming driver for Palo vNICs
A Windows 2008 R2 Baremetal teaming driver for the Cisco VIC would be nice, so I can have simple teamed Load balancing from a Bare metal windows host.

5) One click asset export from UCSM
In most UCS installs there is a need to capture all part and serial numbers, it would be nice to have a 1 click asset export option in UCS Manager to export all parts and serial numbers to a spreadsheet or CSV file. Rather than having to do several CLI show commands, or use scripts.

If anyone has found good interim solutions to any the above, it would be great to hear from you, equally what “gripes” would you like to see addressed in future updates?

Reference version 2.0(1w)

B200 M3

These new 2 socket (Sandy Bridge-EP) servers are built on Intel’s much anticipated E5-2600 processors and Patsburg chipsets, as well as lots of enhancements throughout the server architecture.

This new range will also have LAN on board (mLOM) a VIC 1240 (PALO 2) it provides 2 x 20Gb traces per server with an upgrade module which will take it upto 2 x 40Gb, but still be able to take a standard MEZ card (Should finally silence those who complained that a single MEZ card in a half width blade is a single point of failure despite the immense MTBF)

More Grunt, now with 8 cores per EP CPU, which will compliment the additional bandwidth features of the UCS Gen 2 Hardware nicely.

Maximum memory capacity is 384 GB (with 16 GB DIMMs) capable or running at upto 1600MHz

Also announced was Multi-UCS Manager which alows the management of multiple UCS domains, even spread across geographical areas.

The full spec sheet can be found at
B200 M3
http://www.cisco.com/en/US/prod/collateral/ps10265/ps10280/B200M3_SpecSheet.pdf
C220 M3
http://www.cisco.com/en/US/products/ps12369/index.html
C240 M3
http://www.cisco.com/en/US/products/ps12370/index.html

2204 IO Module
The new UCS Generation 2 2204 IO Module is (as its name would indicate) basically just a 2208 cut in half, i.e. 4 Network ports and 16 blade facing ports (2 x 10Gbs traces per server, per Fabric = 40Gbps per half width blade) this will be a good fit in situations which want a good performance/cost balance option, ustilising double the server traces of a 2104XP when used in conjuction with the VIC 1280.

All products should hopfully be shipping by March 2012