There comes a time, when if I have to answer the same question a certain number of times, I think “this obviously requires a blog post”, so I can just tell the next person who asks to go and read it.
This is such a question.
“Ok so I have a VIC 1240 mLOM on my M3 Blade which gives me 20Gb of Bandwidth per Fabric, Correct?
Correct!
Cool, I also have a 2204XP IO module that gives me 20Gb of Bandwidth per Fabric to each of my blade slots, Correct?
Correct!
Fantastic, so if I use one with the other I get 20Gb of I/O per Fabric per Blade, Correct?
Wrong!
Huh?
Ok lets grab a white board marker and lets go!
I can really understand the confusion around this, because at first the above logic makes perfect sense, it’s only when you open the UCS Kimono that you see the reason for this behaviour.
So as we all know the M3 blades give us a nice Modular LAN on Motherboard (mLOM) which is a VIC 1240, this gives us 2 x 10Gb Traces (KR Ports) to each IO Module.
We also have a spare Mezzanine adapter slot which can be used for a Port-Expander (to effectively turn the VIC1240 into a VIC1280, or can be used for any other compatible UCS I/O Mezzanine card or an I/O Flash module like the IO Drive 2 from Fusion I/O.
This Mezzanine slot also provides 2 x 10Gb Traces (KR Ports) to each IO Module.
Ok, now the “issue” is that the ports of the I/O Module alternate between the on board VIC1240 and the Mezzanine Slot, So to use a Blade in Slot 1 as an example with a 2204XP I/O module. I/O module backplane port 1 goes to the VIC1240, and Port 2 on the I/O module goes to the Mez slot. This is why you only get 10Gb of usable I/O with this combination.
Not sure why Cisco did not trace I/O Module ports 1 and 2 to the mLOM and 3 and 4 to the Mez, I guess the way they have done it allows you to always have access to the Mez slot even if using a 2204XP I/O module. (as mentioned above the Mez slot can be used for other cards not just CNA’s)
So as you can see, when using a 2204XP and the VIC1240 with no Mez adapter, only one of the two 10Gb traces actually matches up. (See Below)
B200M3 VIC 1240, No MEZ, 2202XP
OK, so how do you get your extra bandwidth well one of two ways, either add a mezzanine adapter, or use the 2208XP IO Module or Both.
If you were using a 2208XP I/O module with your VIC 1240. Backplane port 1 on the I/O Module goes to the VIC1240, port 2 on the I/O module goes to the Mez slot, Port 3 on the I/O Module goes to the VIC1240 and port 4 on the I/O Module goes to the Mez. So as you can see, this comdination does give you the two 10Gb traces to your VIC1240.
The other combinations of modules and resulting bandwidth are explained below.
For clarity only the backplane ports of the I/O Module that map to Blade slot 1 are shown.
2204XP Combinations
2208XP Combinations
- 2208XP Combinations
Note that while resulting bandwidth may be the same with certain combinations, the hardware based port-channels are different. Obviously the more ports in the same port-channel will make traffic distribution more efficient.
Also bear in mind that when using the port-expander UCS Manager sees the VIC1240 and the port-expander as a single VIC.
If a VIC1280 is used in conjunction with the VIC1240 they are completely independant adapters from each other, and are treated as such by UCS Manager.
As ever comments most welcome.
this worries me as i have today B200M2 and B230M2 with 2204XP with a 2-link chassy discovery…(FI had limited license ports so i couldn’t off the bat take advantage of the full 4-link). My setup is 2 chassis, 2 FI 6120s, 2x2204XP on each chassis, 4 B200M2 and 4 B230M2, but soon i will need to add more blades and i doubt i can get them with the Cisco Palo M81KR. Based on your post i will not be taking advantage then if i bump up the licenses of the FI and turn into a 4-link ?
Hi Renato
This post certainly shouldn’t worry you, just make you aware of exactly how much potential bandwidth you will have given a certain combination of VIC’s, Mez’s and I/O Modules.
Also bear in mind with this I am talking about the usable bandwidth to the Blade NOT between the FI and the I/O Module which is a completely different conversation.
The main consideration with your FI to IOM links would be that you do not have an immediate bottleneck from the server to the FI.
When using one of the higher bandwidth VIC adapters you should be port-channeling your FI to IOM links (Not the default setting) to at least the usable bandwidth of your VIC. So in your case as you have the 2204XP I/O modules the most you can get out of any of your servers will be 20Gb per fabric (which would require the VIC1280 in your M2 Blades) with your M81KRs you obvisously only still get 10Gbs per fabric. (I say only this is still way more than enough for most use cases)
You are right in saying that your new blades likley will not come with M81KRs ( The VIC1280 is the same cost so why would you want them anyway), you will still have a choice of certain M2 Servers and M3 Servers and this is where you have to choose the model that most suits your needs.
If you go M3 then this post will help you understand that if you want access to the full 20Gb of I/O per server, per fabric possible with your 2204XP I/O Modules then you should also order the port-expander Mez card.
From experience your 2 links between your FEX and FI will likley be sufficient, if you channel them you will have a single 20Gb pipe per fabric that all the blades in the chassis will map to. And when your new blades come with the VIC1280 or VIC1240 with the Port-Expander your 20Gbs per fabric to each blade will have the potential (al-be-it shared) use of the 20Gb of I/O between the IOM and the FI.
Hope this all makes sense, if not fire back.
Colin
Would I be correct in saying that if you where going for ASIC redundancy, aside from bandwidth, you would need the 1240 and 1280? Or are the port groups each individual adapter resilient?
Hi Richard
Great question, and one I did consider covering in the main post, but thought it would detract from the main point, so really glad you asked it here.
You are right in the fact that if you want full VIC redundantcy yes you will need two VICs which are capable of operating without the other. A port-expander for example will not function if its VIC1240 fails.
With regards to port group reduntantcy, I would be very very surprised if the loss of that port groups ASICs had any effect on the other port group. (Perhaps one of my Cisco employee readers can confirm)
My View of this is this, I have now installed what must be (quick calculation) over 1000 VICs over the last 4 years and have never had one fail,(maybe I’m just lucky) Also Redundantcy is generally inforce at a highler layer anyway (VMware HA/FT) or Clustering for Bare Metal workloads etc..
But if having Mezzanine reduntantcy within a Blade is a requirement in your environment then go for it, Thats one of the reasons the option is there. You just have to bare in mind there would be a bit of additional planing to do. i.e. create two vNICs use a placement policy to put one on the VIC 1240 and one on the VIC1280 (different vCons) and then Team them at the operating system level.
Thanks again for the question.
Colin
Hi
I have my FIs connected directly to the core nexus 7k. In addition, we have an iscsi appliance that is alo connected to the 7k. In order for ucs servers to use jumbo frames to the appliance, we have enabled jumbo frame mtu 9216 end to end
on the ucs, we have defined the mtu on the iscsi vnics to 9216
we have a qos policy for iscsi that is set at platinum with a cos of 5
on the 7k, we have mtu 9216 on the iscsi appliance ports as well as the FI ports.
Do I need to setup qos policies on the 7k for return traffic?
thanks
Hi Tony As I’m sure you know setiting the MTU size on your cos 5 traffic on the UCS only effects egress traffic and not return traffic. So you would need to ensure that traffic from your iSCSI Applicance has the mtu set and supported all the way to the UCS and marked with cos 5 )alternativley you could set the mtu on the UCS best effort class to 9216 which would then match return all return traffic regardless of cos value.
I always use the old ping -l (mtu size) -f (Don’t Fragment) to confirm my mtu is fully supported end to end. i.e from the UCS Blade ping (address of iSCSI Appliance) -l 9000 -f
If you get a reply your good, if you get a message saying somthing like “Packet Needs to be fragmented but DF set” then you know somthing in the path is not configured for jumbo frames correctly.
Regards
Colin
Hi,
anyone had oversubscription issues so far with 2204XP or 2208XP? In the end there is ja 4:1 oversubscription with backplane to fabric ports (16:4 or 32:8). Most of my customers don’t have such immense throughput requirements – but im curious of someone had seen such scenarios.
Hi Dan
I Certainly haven’t, most of my clients don’t touch the sides of a 2204XP yet let alone a 2208XP, even in a fully populated Chassis.
Colin
Great detail on this, thanks, but it doesn’t explain why port 1 can’t talk to port 32 on the internal connections of the FEX. By design, it seems like they should. Also, how is the traffic handled within the FEX. If 32 internal ports are connected and only port 1 and 4 externally are connected, is there any control, or is is like pumping 32 gallons of water through 2 spigots?
Hi Kevin
By design the FEX cannot switch traffic within it. The fraffic has to go up to the Fabric Interconnect (Controlling Bridge) to be switched at layer 2.
(This is the way FEX Technology aka 802.1Qbh now called 802.1BR)
So If a Blade in Slot 1 (using HIF Port 1 on the FEX) needs to have an East/West conversation with a Blade in Slot 8 (using HIF Port 32 on the FEX) then that is fine but would be switched at L2 within the FI (If same VLAN and Same Fabric) or be sent to the upstream LAN switch to be routed (if this is a L3 conversation) or L2 Switched across fabrics withion the same VLAN.
Hope that makes sense.
Re the second part of your question it is a simple case of contention ratios, divide the number of Host Interfaces (HIFs) you are using by the number of Network Interface (NIFs) you are using and you have your Maximum contention ratio, so if you have a Chassis full of 8 Blades all with VIC1280′s or VIC1240′s with Port-Expanders and you are using the 2208XP FEX and are using all 8 Network Interfaces then, you are correct you would have 32 x 10Gb ports all using the 8 x 10Gb uplinks (i.e. a 4:1 ratio)
The reality though is that not all 32Gb ports will be running full line rate at the same time or even close to it, so a 4:1 or even an 8:1 will generally not be a cause of contention.
Regards
Colin