There comes a time, when if I have to answer the same question a certain number of times, I think “this obviously requires a blog post”, so I can just tell the next person who asks to go and read it.
This is such a question.
“Ok so I have a VIC 1240 mLOM on my M3 Blade which gives me 20Gb of Bandwidth per Fabric, Correct?
Cool, I also have a 2204XP IO module that gives me 20Gb of Bandwidth per Fabric to each of my blade slots, Correct?
Fantastic, so if I use one with the other I get 20Gb of I/O per Fabric per Blade, Correct?
Ok lets grab a white board marker and lets go!
I can really understand the confusion around this, because at first the above logic makes perfect sense, it’s only when you open the UCS Kimono that you see the reason for this behaviour.
So as we all know the M3 blades give us a nice Modular LAN on Motherboard (mLOM) which is a VIC 1240, this gives us 2 x 10Gb Traces (KR Ports) to each IO Module.
We also have a spare Mezzanine adapter slot which can be used for a Port-Expander (to effectively turn the VIC1240 into a VIC1280, or can be used for any other compatible UCS I/O Mezzanine card or an I/O Flash module like the IO Drive 2 from Fusion I/O.
This Mezzanine slot also provides 2 x 10Gb Traces (KR Ports) to each IO Module.
Ok, now the “issue” is that the ports of the I/O Module alternate between the on board VIC1240 and the Mezzanine Slot, So to use a Blade in Slot 1 as an example with a 2204XP I/O module. I/O module backplane port 1 goes to the VIC1240, and Port 2 on the I/O module goes to the Mez slot. This is why you only get 10Gb of usable I/O with this combination.
Not sure why Cisco did not trace I/O Module ports 1 and 2 to the mLOM and 3 and 4 to the Mez, I guess the way they have done it allows you to always have access to the Mez slot even if using a 2204XP I/O module. (as mentioned above the Mez slot can be used for other cards not just CNA’s)
So as you can see, when using a 2204XP and the VIC1240 with no Mez adapter, only one of the two 10Gb traces actually matches up. (See Below)
OK, so how do you get your extra bandwidth well one of two ways, either add a mezzanine adapter, or use the 2208XP IO Module or Both.
If you were using a 2208XP I/O module with your VIC 1240. Backplane port 1 on the I/O Module goes to the VIC1240, port 2 on the I/O module goes to the Mez slot, Port 3 on the I/O Module goes to the VIC1240 and port 4 on the I/O Module goes to the Mez. So as you can see, this comdination does give you the two 10Gb traces to your VIC1240.
The other combinations of modules and resulting bandwidth are explained below.
For clarity only the backplane ports of the I/O Module that map to Blade slot 1 are shown.
Note that while resulting bandwidth may be the same with certain combinations, the hardware based port-channels are different. Obviously the more ports in the same port-channel will make traffic distribution more efficient.
Also bear in mind that when using the port-expander UCS Manager sees the VIC1240 and the port-expander as a single VIC.
If a VIC1280 is used in conjunction with the VIC1240 they are completely independant adapters from each other, and are treated as such by UCS Manager.
As ever comments most welcome.