This site is maintained and updated by Colin Lynch a data center subject matter expert.
I have no vendor affiliation, I just call it as I see it.
E-mail: colin@ucsguru.com
Twitter: @ucsguru
Cisco Champion for Datacenter.
Search UCSguru.com
-
Recent Posts
Categories
Helpful UCS Links
- UCS Gui APP for Android Phone
- Manage your UCS from your iPhone
- Cisco UCS Advantage Videos
- B Series Model Comparison
- C Series Model Comparison
- Configuration Examples and TechNotes.
- UCS Firmware Downloads
- UCS Visio Stencils
- Cisco HyperFlex LinkedIn Group
- Cisco UCS LinkedIn Group
- Cisco UCS Community
- Cisco Partner Datacenter Community
- UCS Power Calculator
- Cisco UCS Layout Tool
Blogroll
UCSguru.com 
Hi Colin, Thanks for the beautiful posts that you have which create a lot of knowledge on UCS all around. I am a regular follower on your posts & appreciate them very much. I had some questions though & had them for quite some time which I did not know who to ask, so may be you can help me with them. So a couple of questions to start with, but, I have a lot of them (some stupid) also š
1) UCS allows you to abstract identities in the form of the different types of pools that we can create like UUID, WWNN, WWPN & others. How do we make sure that these entities still remain unique in the universe for eg. UUID, anyone can now assign any UUID to their blade but how do we make sure that the initial concept that a UUID is universally unique still remains?
2) As I have studied there are 2 types of split brain scenarios with UCS. Partition in time & partition in space. Partition in space I understand happens when the cluster links are not working. What would be an example case for Partition in time?
3) How many WWNN do I assign to a full width server. Since I don’t have access to one I was not able to ascertain. From what I know a WWNN is assigned to 1HBA & WWPN to different ports on that HBA. So if I have 2 menlo cards on my full width server I should have 2 WWNN & 4 WWPNs. Is that right?
4) I see in some of the Cisco presentations the approach for VN-Link that is most suggested is the one which uses VIC with N1KV in UCS. Why is that the most recommended & what would be an example use case?
Thanks again for the nice posts & hoping for an answer to the above questions.
Regards,
-Tarun
Hi Tarun
Thanks for following and the great questions!
1) You are correct that we have to ensure each UCS defined address is unique but the boundary of that uniqueness differs i.e. MAC addresses only have to be unique within a subnet, UUID’s only have to be unique per UCS Domain and WWNN & WWPN’s only have to be unique within the Switched Fabric.
In all my Cisco UCS designs I create a standard for that Client as to how their UCS addresses are defined. This not only ensures uniqueness, (within that client at least) but allows us to define some useful information into WWPN and MAC addresses a luxury not available when using Burned In Addresses (BIA’s) I’m sure you will agree being able to look at a MAC addresses from a debug or Wireshark capture and immediately know what that device is, where it is and what OS it is running reduces troubleshooting times by orders of magnitude. Or If I see WWPNs with BB in them on my Fabric A then I immediately know someone has made a config / cabling error in the SAN somewhere.
I have shown some address naming conventions I use below.
2) Again you are correct the “Partition in Space” split brain scenario is when a node is no longer receiving heart beats across the L1 / L2 interconnect from the other node member. There is however a Serial Electrically Erasable Programmable Read Only Memory (SEEPROM) on the chassis accessed via the Fabric Extender which shows whether the upstream FI is still reachable, so if in the very unlikely event you loose both your copper cluster links, The fabric interconnects will still know whether or not its peer is alive and forwarding by checking the SEEPROM on the chassis. And NO failover will occur. You will however get alerts in UCSM and no new state information will be synchronized between the FI’s.
The other type of split brain is as you mention referred to as a partition in time. this occurs when a FI attempts to start the UCSM on an outdated configuration. Again, Cisco UCS Manager detects and resolves this type of split brain using the SEEPROM.
3) Again you are right if your are using the Menlo (M71KR / M72KR) then you have 2 vHBA’s per adapter, and if you have 2 of them in your full width blade that blade will then require 2 x WWNN and 4 x WWPN. Obviously if you use the Palo adapters (M81KR) or VIC 1280/1240 the obviously you can have allot more vHBA’s and will obviously require more WWPN for them.
I generally assign 128 WWPN per Pool, that way it scales well, doesn’t take up excessive entries in the XML database, and all hosts get the same last 2 digits and the only difference between then is the Fabric designation, which makes troubleshooting, aliasing and zoning a dream š e.g. Host 1 fc0 (Fabric A) = WWPN 20:00:00:25:B5:AA:00:01, Host 1 fc1 (Fabric B) = WWPN 20:00:00:25:B5:BB:00:01,
4) You have 2 choices when it comes to using VN-LINK either Software (Feature rich but software switched in the Hypervisor) or Hardware (VM-FEX previously called VN-LINK in hardware) which gives hardware switching performance and hypervisor by pass (if used in VMDirect Path mode)
Nexus 1000v is generally my preferred option as it is very flexible and feature rich and gives a nice NXOS command line so is very familiar to Nexus Admins. VM-FEX on the other hand uses UCS Manager for network administration so keeps the GUI admins happy. Bottom line is if low latency is a big requirement go VM-FEX, if features is the main requirement go N1KV.
The figure below shows the feature comparrison between N1KV and VM-FEX. Also see my VM-FEX videos at here
Hope this clears things up for you, if you need any more info you know where I am š
Regards
Colin
Hi Colin,
Thanks a lot on this long & detailed reply & I am sorry to keep asking you all these question, but, honestly I really appreciate your answers because I’ve been pondering on them for so long & haven’t got an answer from anyone on them.
1) It is clear now. I understand now that every implementation of UCS has to take care to implement a unique format of identifier. It just came as a little odd to me that by default UCSM does not enforces anything for implementing uniqueness for identities that are supposed to be universally unique like UUID’s.
2) Thanks again for the detailed explanation on “Partition in Space”. I also understand a little bit on how can “Partition in Time” happen, but, I am not able to imagine what sequence of events can lead to a Partition in Time kind of a situation? I mean what kind of situation would we need this safeguard?
3) Thanks for the detailed explanation on this. It is crystal clear now.
4) Your videos on VM-FEX are just out of this world. I have gone through them & I love them. I learned a few things after watching those videos. What I am still not sure about though is what will be the use case for using a VIC along with a N1KV in a UCS solution? I have seen that as a recommended implementation from Cisco. Why?
Once again, even before you attempt to answer those question I want to thank you again for helping me out here. I can’t emphasize that more.
Regards,
-Tarun
1) Cisco have to leave somthing for us to make a living š
2) You don’t really have to safeguard against a partition in time, UCS does this for you via the SEEPROM, i.e. if a Fabric Interconnect is “out of state” config is behind and for what ever reason UCSM tries to go active on the Out of State FI, then that is where the potential for a Partition in Time could occur, but again UCS mitigates this via the SEEPROM in the chassis. There is a bit about this in the following doc. http://www.cisco.com/en/US/prod/collateral/ps10265/ps10281/white_paper_c11-525344.html
3) Good
4) Using a VIC inconjucion with a N1KV does provide some benefits but is not mandatory by any means. The VIC in this case only provides the UPLINKs for the Nexus 1000V. The benefits being that as these uplinks interfaces are virtual and you can configure / reconfigure them as you wish. As you know when ever you create a vNIC on the Cisco VIC, this creates a corresponding Veth on the fabric interconnect and connects them together with a virtual cable, this is what is the Virtual Network Link. VN-LINK. The VIC really comes into its own when using VM-FEX as you know.
Regards
Colin
Hi Colin,
Another question just came in my mind.
For a 4-link topology, if 1 link fails & we re-ack the chassis. Documentation says re-pinning would happen & it would pin odd slots to odd uplink & even slots to even uplink. What if uplinks 1 & 3 on the IOM go down then what would be the pinning like because both the odd uplinks from the IOM are now down?
Hi Tarun
If you have a 4 link topology and 1 link fails then the adapters pinned to that link will failover (If configured to do so) the remaining 3 links will continue to be used (3 Links in discrete pinning mode is not a valid startup topology but is a valid running topology, as in the above scenario)
OK if you then Re-Ack the chassis UCS will say 3 links is not valid and reduce the links to two and re-pin all adapters acordingly i.e. Odd blade slots to first active link and evens to the second.
If links 1 and 3 went down leaving only links 2 and 4, I would expect Odd blades to PIN to Link 2 and Even blades to PIN to Link 4 when the chassis is Re-ACK’d.
Regards
Colin
Hi Colin,
I am loving it! So nice, to have a reply from a UCS expert. Thanks again for your time. That’s what I was also thinking (& I am proud I think on the lines of a UCS expert), but, then if odd servers are pinned to Link2 would it not negate the rule for 2 link topology which is odd slot servers to odd numbered uplink & even slot servers to even numbered uplink?
I am sure there is going to be an exception for this kind of situation, but, I could not find any documentation for the same.
Regatds,
-Tarun
Pingback: Have a Question about Cisco UCS? Ask it Here! | UCSguru.com
I Colin can you shed some light on the below issue ?
https://supportforums.cisco.com/thread/2175621
Hi Zaheer,
Most performance issues I have seen are down to misconfigured system, QoS, divers or MTU, but obviously could be anything, you have gone the right way with opening up a TAC case, and couldn’t have a better person looking into it for you. Roberts an awesome UCS guy!
Please report the solution back to this thread once found.
Good luck
Colin
Hi Colin,
Thanks for your wonderful blog which serves as a central repository for UCS. I’m from India, Currently working as Lead Engineer @ HCL technologies Chennai, India, in Data Center Solution Testing.
I have a quit a good knowledge on Cisco UCS since i work for HCL -Cisco having 4+ Years of Experience , I’m also VMware VCP5 certified going to write NCDA next month, Currently seeking for job opportunities overseas. Could please suggest me the best way to apply for a Job in Data Center relevant role.
It would be really helpful if you point me into the right direction.
Regards
Saravanan.M
Hi
Glad you find the site useful.
Your certianly doing the right things, having Good Compute, Networking, Storage and Virtualisation skills along with the required certs is certainly the way the Data Center Engineer/Consultant should be going.
Also worth perhaps persuing the CCNA DC and CCNP DC.
The best way of getting a DC job these days would be via http://www.linkedin.com/, if you create a nice profile and corresponding C.V mentioning the above technologies, you will likly start getting the recruitment agencies circling and contacting you.
Also do a “linked in” search on recruitment and connect with those headhunters that seem best suited to your requirements.
Good Luck
Colin
Thanks Colin for your kind reply..,
Hi Colin,
My hearty congratulations on a wonderful blog. I have a query regarding Cisco UCS boot from SAN for VIC adapter implementations. If I configure the Boot Target WWPN in Boot policy/Boot table with a wrong/no WWPN, the vHBAs login to FC switches without any issues, but not to the storage array (in my case Cisco B/C series server, MDS, EMC VNX/VMAX storage). Is there any particular reason for this behavior? This situation is prior to any OS installation while I am doing the initial setup for Boot from SAN.
Hi Sushant
Thanks for the question.
In short this is expected behaviour, you can configure any target WWPN in your boot policy and the vHBAs will always login to the Fabric. If fact I sometimes do this if I do not know the correct WWPN’s but want my initiator WWPN’s to appear in the fabric so the SAN admin or myself can just cut and paste them into the correct Zones and create the alias etc..
Of course using dummy target WWPN’s won’t actually login to the Storage Array as 1) The SAN will not be zoned with them, and 2) even if it was the WWPN do not exist on the array so the Port-Login (PLOGI) from the initiator (Blade vHBA) will not complete.
Regards
Colin
Hello Colin,
I have a couple of questions. Btw, thanks for the great bolg. We have a DMZ zone that is to be implemented and the some of the VM’s on the UCS infrastructure have to be sitting on the DMZ.
We have 5 servers running on the UCS environment and 3 other to be deployed in about two weeks from now.
FI: 6248, Chassis: 5108, Storage: VNX 5300 are the components in our solution.
1. How do we segregate the Lan Traffic from DMZ traffic?
2. If I want one ESX host to lie on the DMZ then how do I do it?
3. Or if I want to have a couple of VM’s on the DMZ how do I do it?
The network guyz are planning to connect FI to DMZ switches as well as the Network UPlink switches. So the traffic between LAN and DMZ follows the path: LAN-> Uplink Switch -> Layer3 Switch -> Firewall -> DMZ switch -> DMZ servers and vice versa.
Please let me know the solution.
Regards
Dhruva S kolli
Hi Dhruva
Yes certainly “doable”, since UCSM 2.0 it is fine connecting an FI to two seperate upstream Networks and just mapping the correct VLANs to the relevant Network uplinks.
Playing devils advocate with my Security hat on, the above is vulnerable to a Cisco UCS misconfiguration, i.e if the UCS admin accidently trunked a DMZ VLAN down to a Production ESXi Server. Of course further misconfigurations would need to occur to be a problem i.e create a DMZ port group on the Prod vSwitch and apply the wrong IP (if not using DHCP) not likley I’ll grant you. But if you are subject to a strict security policy or regulatory body like PCI, having this potential Layer 2 adjantcy at the FI level may not be enough to satisfy them.
In such a case I would also look at a policy based Gateway, like Cisco VSG or Juniper Networks’ vGW where you can put all your Prod VM’s into one “Policy Pot” and all your DMZ VM’s into another “Policy Pot” then have a simple rule Prod cannot talk directly to DMZ and vice versa.
Regards
Colin
Hi,
We are exploring information about UCS 1500 blade system for our telco solution.
One thing which is not so clear on how can we use a 3rd party PCIe card (like for T1/E1 and LAN connections) in this system? Is their any standard PCIe expansion module available? How about AMC expansion?
Any information on this will help.
Regards.
Hi Schanda
To use 3rd Party PCIe adapters you would need to use Cisco UCS C-Series Rackmounts.
If you have a look at the below link you will see all supported Adapters.
C-Series Rackmounts can be seemlessly integrated into a Cisco UCS Manager Domain and with UCSM 2.2 supports single wire direct connect to the Fabric Interconnects.
Regards
Colin
Colin,
Thanks for such a great site first off. I’ve been reading here for some time now, and have learned more real-world info here than probably anywhere else in my studies thus far.
I have a question concerning the correct VEM for VM-FEX. I have a UCS system running 2.2(1c) and a vSphere 5.5 environment. I’m playing with VM-FEX; however, have blown up my lab numerous times trying with these current versions. Cisco seems to have removed the VM-FEX VEM download link from the UCSM front page, and maybe I’ve been looking so long for the information on Cisco’s website that it’s right in front of me and I can see it, but I can find an updated matrix of which VEM to use for ESXi 5.5. The last version I tried was the VEM550-201401164104-BG-release bundle included with the N1Kv, but couldn’t get it to work right and I suspect the VEM version was the issue.
Any words of wisdom to help me sir? Thanks in advance for any help you might be able to give, and especially thanks again for the excellent site.
Best regards,
Mike
Colin,
Just a follow-up to my original question. I’ve figured it out. It appears the VM-FEX VEM is now different and distinct from the N1Kv VEM. Once I figured out where Cisco stashed the files on CCO, I was able to get everything working properly.
Take care!
Mike
I am doing implementation of CISCO UCS at Customer site. We are facing a challenge here regarding the Management IPs.
The Issue is the Management IP of the existing VLAN is different from the new created VLAN. Because of this We are no able to take KVM console for the newly installed servers. We have even done the network changes in management ports, but still the we are not getting connected through KVM.
Can i trunk FI management port? or how to resolve this issue.
Please help me .
Hi in UCSM 2.2 you now have the option of using Inband Management IP addresses.
http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2/b_UCSM_GUI_Configuration_Guide_2_2_chapter_011101.html
Regards
Colin
Hi Colin,
I was wondering what happens if an FI is connected to 2 different SANs (similar to Disjoint L2 in LAN)?
Regards,
Amit.
Thanks for great information on UCS, I have started working on my first UCS env, trying to boot ESXI5.1 on B200 M3 blades from SAN. Installer can see one LUN from EMC SAN & Installs OS But on consequent reboot during POST (quiet boot disabled) VIC 1240 is not able to see boot lun & OS does not load, not sure if issue is with end to end SAN connectivity? also there are 2 vhba’s assigned to each blade then why only one boot lun is seen by installer?
Have you added the vHBA’s into the Boot Policy? and if so ensured the names of the vHBAs in the boot policy are exactly what you have called them in the Service Profile (Case sensitive)
If the ESXi installer can see the Boot LUN and you have installed the OS on it, then it sounds like the SAN and Array elements are OK, and it is likely a UCS Boot Policy issue.
Have a look through my “Boot From SAN Video walk through” as well as the Boot from SAN trouble shooting section” I wrote in the “Ask a question” section and see if that solves it for you.
Good Luck
Colin
Hi, I have B200 M3 blade, after taking KVM console while trying to activate vertical device, I am getting error as āVirtual device redirection is already in use by another userā. but I am the first user to access KVM and activate virtual device.
I am very much interested to read the articles in ucsguru.com. Expecting the new articles.
Hello Collin. First of all thanks for this blog and share your experience.!
I would like to ask you about something about SAN connectivity of UCS.
Is it possible to connect UCS to a SAN FC switch that does not support NIPV? I’m thinking in switching UCS to SAN Switching mode but I do not know if that will work.
I’ve read this article:
http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-infrastructure-ucs-manager-software/116082-config-ucs-das-00.html
And maybe setting UCS SAN switching mode and configure a different Storage connection policy for each B-Blade mapped to the Controller WWpN will work. Then it is up to the controller to add security allowing WWpN to LUN’s access.
thanks for your help!
Hi Martin
I doubt that would work, the point of NPIV is to allow multiple Flogi’s (WWPN’s) on the same F-Port, without it the port would just keep flapping WWPN’s as it saw the different requests. As you may have noticed in EHM the FC ports on the FI will not even come up if the upstream switch does not have NPIV enabled. I’d be surprised if you can find a SAN switch these days that does not support NPIV as it is a pretty standard feature.
Regards
Colin
Hi,
Using UCS Director/Central/Manager, we can create various pools/policies using GUI etc and push it to the server. Can you please help me understand how does this happen? Communication from GUI to the actual hardware, how does it happen?
Thanks!
Dear collin,
Lovely site on ucs and great info..
I have some doubts regarding hyperflex with hyperv. In a 3node cluster after a reboot one od a node is not able to access the datastore. The storage cluster is showing healthy and failover cluster is also good. But on datastore it shows node cannot access the path, the share does not exist. This is a MS hyperv based installation with constrained delegation (as instructed by cisco).
Is there anyway to add the share onto the node or make it accessible. It would be of great help.
Thank youu