Technical Note
Round-Robin Load Balancing
VMware ESX Server 3.5, VMware ESX Server 3i version 3.5, VMware VirtualCenter 2.5
VMwareESXServer3.5andESXServer3iversion3.5enhanceESXServernativemultipathingbyproviding experimentalsupportforroundrobinloadbalancing.Thistechnicalnoteexplainshowroundrobinload balancingworksandhowtosetit. NOTERoundrobinloadbalancingisexperimentalandnotsupportedforproductionuse. RoundrobinloadbalancingisavailablefromtheVIClientbutismarkedasexperimental.
Understanding Round-Robin Load Balancing
ESXServerhostscanusemultipathingforfailover.WhenonepathfromtheESXServerhosttotheSAN becomesunavailable,thehostswitchestoanotherpath.ESXServerhostscanalsousemultipathingforload balancing.Toachievebetterloadbalancingacrosspaths,administratorscanspecifythattheESXServerhost shouldswitchpathsundercertaincircumstances.DifferentsettableoptionsdeterminewhentheESXServer hostswitchespathsandwhatpathsarechosen. WhentoswitchSpecifythattheESXServerhostshouldattemptapathswitchafteraspecifiednumber ofI/Oblockshavebeenissuedonapathorafteraspecifiednumberofreadorwritecommandshavebeen issuedonapath.Ifanotherpathexiststhatmeetsthespecifiedpathpolicyforthetarget,theactivepath tothetargetisswitchedtothenewpath.The--custom-max-commandsand--custom-max-blocks optionsspecifywhentoswitch. WhichtargettouseSpecifythatthenextpathshouldbeonthepreferredtarget,themostrecentlyused target,oranytarget.The--custom-target-policyoptionspecifieswhichtargettouse. WhichHBAtouseSpecifythatthenextpathshouldbeonthepreferredHBA,themostrecentlyused HBA,theHBAwiththeminimumoutstandingI/Orequests,oranyHBA.The--custom-HBA-policy optionspecifieswhichHBAtouse.
Setting the Path Switching Policy
Youcansetthepathswitchingpolicyforfailoverandforloadbalancingbyusingtheesxcfg-mpath command. NOTEYoucansetpathswitchingforfailoverintheVIClient.SeetheSANConfigurationGuide.Youcanset loadbalancingonlyfromthecommandline. YoucansetthepathswitchingpolicyonaperLUNbasisbyusingtheesxcfg-mpathcommands --policy customoption.Ifyouspecify--policy custom,youmustalsospecifyoneofthecustompolicyoptions. BecausethepathswitchingpolicyissetonaperLUNbasis,youmustalwaysspecifytheLUNusingthe--lun option.
Copyright 2007 VMware, Inc. All rights reserved.
Round-Robin Load Balancing
ThefollowingoptionsaresupportedanddiscussedinmoredetailinTable 1.
esxcfg-mpath --lun <lun> [--policy [mru|rr|fixed|custom]] [--custom-hba-policy|-H [mru|preferred|any|minq]] | [--custom-target-policy|-T [mru|preferred|any]] | [--custom-max-commands|-C <max_commands>] | [--custom-max-blocks|-B <max-blocks>]
Table 1. Policy Options for esxcfg-mpath
Option --policy -p [mru|rr|fixed|custom] Description SetthepolicyforagivenLUNtomru,rr,fixed,orcustom.Thisoptionrequiresthat the--lunoptionisalsopassedtoindicatetheLUNtosetthepolicyfor. Mostrecentlyused(mru)selectsthepathmostrecentlyusedtosendI/Otoadevice. Roundrobin(rr)usesthemrutargetselectionpolicyandtheanyHBAselection policytoselectpaths. Fixed(fixed)usesonlytheactivepath. Custom(custom)setstheLUNtoexpectacustompolicy. NOTEAfteryouvesetthepolicytocustomforaLUN,youcanspecifyoneormoreof theotheroptions.ForanyLUN,yousetthepolicytocustomonlyonce. --custom-hba-policy -H SelectionpolicyfortheHBA.Useoneofthefollowing: preferredUsetheHBAthatspartofthepreferredpath. mru UsethemostrecentlyusedHBA. minqHBAthathasthesmallestnumberofoutstandingI/Ocommandswhenthe ESXServersystemdoesthepathselection. anyUseanyavailableHBA. --custom-target-policy -T Selectionpolicyforthetarget.Useoneofthefollowing: preferredUsethetargetthatispartofthepreferredpath. mruUsethemostrecentlyusedtarget. anyUseanyavailabletarget. --custom-max-commands -C MaximumnumberofI/Orequeststobeissuedonagivenpathbeforethesystemtries toselectadifferentpath.Asettingofzero(0)disablesswitchingbasedoncommands. Specifythenumberofcommandsasaninteger.Defaultis50. --custom-max-blocks -B MaximumnumberofI/Oblockstobeissuedonagivenpathbeforethesystemtriesto selectadifferentpath.Asettingofzero(0)disablesswitchingbasedonblocks. Specifythenumberofblocksasaninteger.Defaultis2048.
Notes
Ifyousetthecustom-max-blocksandcustom-max-commands,options,thesystemattemptstoswitchpaths assoonasoneofthelimitsisreached. IfyousetthetargetortheHBApolicytopreferred,thesystemchoosesthetargetortheHBAofthepreferred pathwhenpossible.Ifapreferredpolicyissetonanactive/passiveSANarray,andthepreferredtargetisnot ontheactiveSP(StorageProcessor),thesystemdoesnotselectthepreferredtargetbutatargetontheactiveSP. PathswitchingisnotperformedifanoutstandingSCSIreservationisonthetarget,orifapathfailoveris underway.PathswitchingisdelayeduntilanI/Orequestisperformedwhennoreservationsorpathfailovers arepending.
Copyright 2007 VMware, Inc. All rights reserved.
Round-Robin Load Balancing
Examples
esxcfg-mpath --lun=vmhba0:0:0 -p custom
SetsthisLUNtouseacustompolicy.Afterthepolicyhasbeensetto customforaLUN,youdonothavetoincludethe-p customoption whenyoufurtherspecifythecustompolicy. ChangestoadifferentHBAafter4000blockshavebeenissuedonthe currentpath.UsesanyavailableHBA. Displaysthefollowing.
Disk vmhba0:0:0 /dev/sda (34732MB) has 1 paths and policy of Custom: maxCmds=50 maxBlks=2048 hbaPolicy=any targetPolicy=mru Local 1:4.0 vmhba0:0:0 On active preferred
esxcfg-mpath --lun=vmhba0:0:0 -H any -B 4000 esxcfg-mpath -q --lun=vmhba0:0:0
# esxcfg-mpath --lun=vmhba0:0:0 --custom-max-commands=100 # esxcfg-mpath -q --lun=vmhba0:0:0
Setsto100themaximumnumberofcommandsthatcanbeissued beforethesystemswitchestoadifferentpath. Displaysthefollowing(asaresultoftheprecedingcommand).
Disk vmhba0:0:0 /dev/sda (34732MB) has 1 paths and policy of Custom: maxCmds=100 maxBlks=2048 hbaPolicy=any targetPolicy=mru Local 1:4.0 vmhba0:0:0 On active preferred
Global Disk Configuration Options
Twoglobaldiskconfigurationoptionsdeterminethedefaultsettingsfor--custom-max-blocksand --custom-max-commandswhentheroundrobinpolicyisset: SPBlksToSwitchNumberofblockssentoveragivenpathbeforeapathswitch. SPCmdsToSwitchNumberofI/Ocommandssentoveragivenpathbeforeapathswitch. Youcanuseesxcfg-advcfgtoviewandsettheseoptions,forexample:
# esxcfg-advcfg -g /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 50 # esxcfg-advcfg -s 200 /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 200 # esxcfg-advcfg -g /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 200
VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2007 VMware, Inc. All rights reserved. Protected by one or more of U.S. Patent Nos. 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149, 843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030, and 7,281,102; patents pending. VMware, the VMware boxes logo and design, Virtual SMP and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation. Linux is a registered trademark of Linus Torvalds. All other marks and names mentioned herein may be trademarks of their respective companies. Revision 20071119
