{"id":442,"date":"2011-06-13T00:41:00","date_gmt":"2011-06-13T00:41:00","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/2012\/10\/apache-cxf-load-balancing-and-failover.html"},"modified":"2012-10-21T19:49:52","modified_gmt":"2012-10-21T19:49:52","slug":"apache-cxf-load-balancing-failover","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2011\/06\/apache-cxf-load-balancing-failover.html","title":{"rendered":"Apache CXF Load Balancing And Failover"},"content":{"rendered":"
\n
A while ago we’ve faced the requirement of load-balancing web services clients based on Apache CXF<\/a>. Also the clients should automatically fail-over when some of the servers are down. To make it even worse, the list of servers target addresses was to be obtained from external service and updated at runtime.<\/p>\n

Eventually we ended up with home-grown load-balancing micro-library (ESB\/UDDI\/WS-Addressing seemed like an interesting alternatives, but they were an overkill in our situation). If we only knew Apache CXF already supports all these features (almost) out of the box?<\/p>\n<\/div>\n

\n<\/div>\n
Don’t blame us though, only reference<\/a> to this feature points to a very poor documentation<\/a> page (if you call 404 \u201cpoor\u201d). If it’s not in official documentation, I would expect to find it in Apache CXF Web Service Development<\/a> book \u2013 unfortunately, bad luck there as well. But hey, isn’t exploring such features yourself even greater fun? This is the client configuration we are starting with:<\/p>\n<\/div>\n
<beans xmlns=\"http:\/\/www.springframework.org\/schema\/beans\"\r\n       xmlns:jaxws=\"http:\/\/cxf.apache.org\/jaxws\"\r\n       xmlns:clustering=\"http:\/\/cxf.apache.org\/clustering\"\r\n       xmlns:util=\"http:\/\/www.springframework.org\/schema\/util\">\r\n\r\n    <jaxws:client id=\"testServiceClient\"\r\n                  serviceClass=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\"\r\n                  address=\"http:\/\/serverA\/simple\">\r\n    <\/jaxws:client>\r\n\r\n<\/beans>\r\n<\/pre>\n
Endpoint interface is not important here, enough to know the testServiceClient<\/span>  is being injected to some other services and load balancing and  failover features shouldn’t affect existing code. Note the service  address is fixed and hard-coded (of course it can be externalized and  read upon startup).<\/p>\n
\n<\/div>\n
Surprisingly enabling failover alone is pretty simple, straightforward and self-explanatory (despite being XML):<\/div>\n
<jaxws:client id=\"testServiceClient\"\r\n              serviceClass=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\"\r\n              address=\"http:\/\/serverA\/simple\">\r\n\r\n    <jaxws:features>\r\n        <clustering:failover>\r\n            <clustering:strategy>\r\n                <bean class=\"org.apache.cxf.clustering.RandomStrategy\">\r\n                    <property name=\"alternateAddresses\">\r\n                        <util:list>\r\n                            <value>http:\/\/serverB\/simple<\/value>\r\n                            <value>http:\/\/serverC\/simple<\/value>\r\n                            <value>http:\/\/serverD\/simple<\/value>\r\n                        <\/util:list>\r\n                    <\/property>\r\n                <\/bean>\r\n            <\/clustering:strategy>\r\n        <\/clustering:failover>\r\n    <\/jaxws:features>\r\n\r\n<\/jaxws:client>\r\n<\/pre>\n
The serverA<\/span> address is used as a primary endpoint, but when it fails all failover endpoints (serverB<\/span>, serverC<\/span> and serverD<\/span>) are examined in random order. To play a bit with this configuration I advice you to turn on Apache CXF logging<\/a> of requests and responses:<\/div>\n
<cxf:bus>\r\n    <cxf:features>\r\n        <cxf:logging\/>\r\n    <\/cxf:features>\r\n<\/cxf:bus> \r\n<\/pre>\n
Once again (!) official documentation does not mention about very convenient configuration parameter prettyLogging<\/span>  that can be applied to logging feature in order to make XML requests  and responses being properly formatted (new lines and indentation)  before being logged. I wouldn’t recommend it for production setup, but  during development and testing having SOAP messages formatted is  invaluable:<\/div>\n
<bean id=\"abstractLoggingInterceptor\" abstract=\"true\">\r\n    <property name=\"prettyLogging\" value=\"true\"\/>\r\n<\/bean>\r\n<bean id=\"loggingInInterceptor\" class=\"org.apache.cxf.interceptor.LoggingInInterceptor\" parent=\"abstractLoggingInterceptor\"\/>\r\n<bean id=\"loggingOutInterceptor\" class=\"org.apache.cxf.interceptor.LoggingOutInterceptor\" parent=\"abstractLoggingInterceptor\"\/>\r\n\r\n<cxf:bus>\r\n    <cxf:inInterceptors>\r\n        <ref bean=\"loggingInInterceptor\"\/>\r\n    <\/cxf:inInterceptors>\r\n    <cxf:outInterceptors>\r\n        <ref bean=\"loggingOutInterceptor\"\/>\r\n    <\/cxf:outInterceptors>\r\n    <cxf:outFaultInterceptors>\r\n        <ref bean=\"loggingOutInterceptor\"\/>\r\n    <\/cxf:outFaultInterceptors>\r\n    <cxf:inFaultInterceptors>\r\n        <ref bean=\"loggingInInterceptor\"\/>\r\n    <\/cxf:inFaultInterceptors>\r\n<\/cxf:bus>\r\n<\/pre>\n
So our service nicely fails over to backup endpoints if primary one is  not available. But we have four equivalent servers and we want our  client to treat them equally hitting each one with similar probability  (round robin? random?). Here is when load-balancing is entering the  stage:<\/div>\n
<jaxws:client id=\"testServiceClient\" serviceClass=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\">\r\n\r\n    <jaxws:features>\r\n        <clustering:loadDistributor>\r\n            <clustering:strategy>\r\n                <bean class=\"org.apache.cxf.clustering.SequentialStrategy\">\r\n                    <property name=\"alternateAddresses\">\r\n                        <util:list>\r\n                            <value>http:\/\/serverA\/simple<\/value>\r\n                            <value>http:\/\/serverB\/simple<\/value>\r\n                            <value>http:\/\/serverC\/simple<\/value>\r\n                            <value>http:\/\/serverD\/simple<\/value>\r\n                        <\/util:list>\r\n                    <\/property>\r\n                <\/bean>\r\n            <\/clustering:strategy>\r\n        <\/clustering:loadDistributor>\r\n    <\/jaxws:features>\r\n\r\n<\/jaxws:client>\r\n<\/pre>\n
\n
Please note that the client itself does no longer define the address attribute. This suggests that alternateAddresses<\/span> list is used exclusively throughout all the invocations and no primary address exists \u2013 which is actually the case. SequentialStrategy<\/span> will use one endpoint after another providing nice round robin implementation (RandomStrategy<\/span>  is available as well). Also in this configuration you will get failover  for free \u2013 if any endpoint fails, all endpoints starting from the first  one will be examined (obviously except the one that has just failed).<\/div>\n
\n<\/div>\n
Great! Now are CXF clients are much more rigid and  fault-tolerant. But in our journey for higher availability and  minimizing downtimes having alternate nodes being loaded only at  application startup (in other words \u2013 adding a new server requires all  clients restart) is too limiting. Fortunately we can make our  load-balancing a bit more dynamic in two simple steps.<\/div>\n
<jaxws:client id=\"testServiceClient\" serviceClass=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\">\r\n\r\n    <jaxws:features>\r\n        <clustering:loadDistributor>\r\n            <clustering:strategy>\r\n                <bean class=\"org.apache.cxf.clustering.SequentialStrategy\">\r\n                    <property name=\"alternateAddresses\" ref=\"alternateAddresses\"\/>\r\n                <\/bean>\r\n            <\/clustering:strategy>\r\n        <\/clustering:loadDistributor>\r\n    <\/jaxws:features>\r\n\r\n<\/jaxws:client>\r\n\r\n<util:list id=\"alternateAddresses\" list-class=\"java.util.concurrent.CopyOnWriteArrayList\">\r\n    <value>http:\/\/serverA\/simple<\/value>\r\n    <value>http:\/\/serverB\/simple<\/value>\r\n    <value>http:\/\/serverC\/simple<\/value>\r\n    <value>http:\/\/serverD\/simple<\/value>\r\n<\/util:list>\r\n<\/pre>\n
\n
Nothing fancy, extracting nested anonymous bean. But having access to this list (please note I used java.util.concurrent.CopyOnWriteArrayList<\/span>)  enables us to inject it to any other service, possibly changing its  state. How do I know this will work? Well, I spent few afternoons  debugging Apache CXF to finally discover load-balancing algorithm: at  first invocation CXF asks strategy for a list of possible nodes. Then it  passes this list back to the strategy asking to pick one (small wtf<\/i> here…) The strategy decides which address to use and removes picked address from the list (another small one<\/i>  here…) When CXF discovers the list is empty, story repeats itself. So  if we replace the list of alternate addresses at runtime, after one  round new list will be returned to the core CXF infrastructure.<\/div>\n
\n<\/div>\n
Because I’m a huge JMX advocate, here is how we are going to  modify the addresses list (you can use whatever mechanism you like):<\/div>\n
@Service\r\n@ManagedResource\r\npublic class AlternateAddressesManager {\r\n\r\n    @Resource\r\n    private List alternateAddresses;\r\n\r\n    @ManagedOperation\r\n    public void addAlternateAddress(String address) {\r\n        alternateAddresses.add(address);\r\n    }\r\n\r\n    @ManagedOperation\r\n    public boolean removeAlternateAddress(String address) {\r\n        return alternateAddresses.remove(address);\r\n    }\r\n\r\n    @ManagedAttribute\r\n    public List getAlternateAddresses() {\r\n        return Collections.unmodifiableList(alternateAddresses);\r\n    }\r\n\r\n}\r\n<\/pre>\n
<\/div>\n
\n
Yep, it’s the very same alternateAddresses<\/span> list used by SequentialStrategy<\/span>, so by simply modifying it we are altering CXF addressing behaviour. Arguably we could extend CopyOnWriteArrayList<\/span> adding few extra JMX-enabled methods (or, exploting Springs’ flexibility, expose List<\/span> methods directly via JMX!), but this would decrease maintainability and I would consider this as poor design.<\/div>\n
\n<\/div>\n
Finally, we can launch jconsole<\/span> or JVisualVM as on the screenshots below and enjoy our load-balancing infrastructure:
 
<\/div> <\/div><\/p>\n
<\/a><\/div>\n
<\/a><\/div>\n<\/div>\n
<\/div>\n
\n
Happy? Not really. While studying CXF source code I came across this dreadful JavaDoc comment on LoadDistributorFeature<\/span> and FailoverTargetSelector<\/span> classes which take significant part in load-balancing process:<\/div>\n
\n<\/div>\n
\/**<\/div>\n
 * […]<\/div>\n
 * Note that this feature changes the conduit on the fly and thus makes<\/strong><\/div>\n
 * the Client not thread safe.<\/strong><\/div>\n
 *\/<\/div>\n
\n<\/div>\n
Focus on the text in bold (OK, honestly, I don’t understand the rest). If you’ve worked with Spring for some time you know that testServiceClient<\/span>  bean is a shared singleton used by multiple threads concurrently (no,  making it prototype scope won’t help; why?), in contrary to default EJB  stateless session beans, which are pooled. Fortunately Spring has a  built-in solution for that as well. But before I finally came up with a  right solution, several obstacles arose.<\/div>\n
First, jaxws:client<\/span>  tag from CXF namespace does not allow to define bean scope, defaulting  to singleton, while we want to pool our clients. So I had to fall back  to good old bean definition with org.apache.cxf.jaxws.JaxWsProxyFactoryBean<\/span>.  No problem, slightly more verbose, although if you can’t\/don’t want to  use custom Spring namespaces, you might have used it from the very  beginning. Now the best part: I can simply wrap any bean with prototype  scope in a special proxy and Spring will automagically<\/i> create an object pool (based on commons-pool<\/a> library) and create as many bean instances as necessary to keep each bean used by only one thread. Here is the configuration:<\/div>\n
<bean id=\"testServiceClientFactoryBean\" class=\"org.apache.cxf.jaxws.JaxWsProxyFactoryBean\">\r\n    <property name=\"serviceClass\" value=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\"\/>\r\n    <property name=\"features\">\r\n        <util:list>\r\n            <bean class=\"org.apache.cxf.clustering.LoadDistributorFeature\">\r\n                <property name=\"strategy\">\r\n                    <bean class=\"org.apache.cxf.clustering.SequentialStrategy\">\r\n                        <property name=\"alternateAddresses\" ref=\"alternateAddresses\"\/>\r\n                    <\/bean>\r\n                <\/property>\r\n            <\/bean>\r\n        <\/util:list>\r\n    <\/property>\r\n<\/bean>\r\n\r\n<bean id=\"testServiceClientTarget\" factory-bean=\"testServiceClientFactoryBean\" factory-method=\"create\" scope=\"prototype\" lazy-init=\"true\"\/>\r\n\r\n<bean id=\"testServiceClient\" class=\"org.springframework.aop.framework.ProxyFactoryBean\">\r\n    <property name=\"targetSource\">\r\n        <bean class=\"org.springframework.aop.target.CommonsPoolTargetSource\">\r\n            <property name=\"targetClass\" value=\"com.blogspot.nurkiewicz.cxfcluster.SimpleService\"\/>\r\n            <property name=\"targetBeanName\" value=\"testServiceClientTarget\"\/>\r\n            <property name=\"maxSize\" value=\"10\"\/>\r\n            <property name=\"maxWait\" value=\"5000\"\/>\r\n        <\/bean>\r\n    <\/property>\r\n<\/bean>\r\n<\/pre>\n
\n
Have you noticed maxSize<\/span> and maxWait<\/span> pool attributes? They are insanely cool<\/strong>!  You can tell Spring not to create more than 10 clients in the pool and  if the pool is empty (all the beans are currently in use), we should  wait no more than 5000ms (and what happens afterwards is configurable!)  This is actually a very simple yet powerful throttling mechanism, much  simpler than JMS or explicit thread pools, we get absolutely for free!  E.g. don’t want to serve more than 20 concurrent web service clients?  Make your server endpoint access service bean being pooled with size  limited to 20. Client above this limit will be rejected as no service  bean is available.<\/div>\n
\n<\/div>\n
Of course in adults world nothing works as expected. I quickly discovered that JaxWsProxyFactoryBean.create is not thread-safe<\/i> and reported CXF-3558<\/a>. As a workaround I had to synchronize the client factory used by <\/span>CommonsPoolTargetSource<\/span> simply by subclassing it:<\/span><\/div>\n
import org.apache.commons.pool.ObjectPool;\r\nimport org.apache.commons.pool.PoolUtils;\r\nimport org.springframework.aop.target.CommonsPoolTargetSource;\r\n\r\npublic class SynchCommonsPoolTargetSource extends CommonsPoolTargetSource {\r\n\r\n    @Override\r\n    protected ObjectPool createObjectPool() {\r\n        return PoolUtils.synchronizedPool(super.createObjectPool());\r\n    }\r\n\r\n}\r\n<\/pre>\n
\n
Synchronizing the factory seems like a common need so I created SPR-8382<\/a> \u2013 maybe it will find its way to official release. BTW while working on this article I also reported IDEA-70365<\/a> \u2013 <\/span>Spurious “Could not autowire” error reported for beans of List type<\/i>.<\/span><\/div>\n
\n<\/div>\n
Finally! Our load-balancing  and failover works like a charm. Next step would be to temporarily  discard nodes that are down for couple of seconds and increase this time  if the endpoint is still down afterwards. But Apache CXF has so  terrible API in this area that I had to leave this topic for a while.  Maybe YOU can help?<\/span><\/div>\n<\/div>\n
Reference: <\/i><\/strong>Enabling load balancing and failover in Apache CXF<\/a> from our JCG partner<\/a> Tomasz at the NoBlogDefFound blog<\/a>.<\/p>\n
Related Articles :<\/i><\/strong><\/div>\n