{"id":288,"date":"2010-06-27T15:20:00","date_gmt":"2010-06-27T15:20:00","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/2012\/10\/debugging-a-production-server-eclipse-and-jboss-showcase.html"},"modified":"2012-10-21T19:14:38","modified_gmt":"2012-10-21T19:14:38","slug":"debug-production-server-eclipse","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2010\/06\/debug-production-server-eclipse.html","title":{"rendered":"Debugging a Production Server – Eclipse and JBoss showcase"},"content":{"rendered":"

Do you write code that has bugs? No, of course not. For the rest of us, mere mortals, who do write code with bugs, I would like to address a very sensitive issue: debugging an application that runs on a production server.<\/p>\n

So, your application is ready for deployment. Unit tests<\/a> were all successful, testers<\/a> found some minor bugs that were immediately fixed, the integration tests<\/a> with the other department’s modules went pretty smoothly, the QA department<\/a> made no complaints and the UATs<\/a> were passed with flying colors. Thus, your awesome code is now up and running on the production server.<\/p>\n

When the unthinkable happens. Your client notices some buggy behavior on the application and some of his customers have already started complaining. It seems that some nasty little bug has managed to get through all the testing procedures and make it to the live system. The client’s users pressure the client, the client managers pressure your managers and, guess what, your managers start pressuring you.<\/p>\n

You start up the test server and try to reproduce the bug<\/a>. Alas, everything runs correctly on the test server, so it might be a strange configuration issue or an edge case that causes the problematic behavior. The bottom line is that you cannot track the bug down using your testbed.<\/p>\n

What a poor developer should do? Debug the application that runs on the production server. Note that this should be considered as a last resort and when all other attempts to spot the bug have failed. Be sure that the slightest wrong move while on production server (which serves a large number of users) could massively affect the application and cause even bigger problems or a complete service outage.<\/p>\n

So, if you decide to take the risky road, read along on how to do it. Some basic guidelines before you get started. First of all, let your client know that you will connect to the production system and \u201cperform some checks\u201d. You don’t have to be specific on what will be done, but certainly don’t do anything without informing the client. Second, pick the time where real traffic is as low as possible. This is a no-brainer, you want as little as possible users to be affected, plus you don’t want to have the server running on heavy load. Third, be careful and try not to be hasty. There might be pressure, but take your time, it will be easier to track down the problem.<\/p>\n

I will use JBoss AS<\/a> and Eclipse in order to provide a hands-on example on how to perform the debugging. We will simulate a running application by deploying a simple piece of code on JBoss and executing a specific method. In most Java based application servers it is just a matter of configuration to start up the JVM with remote debugging enabled. Then, you use your favorite IDE, in my case Eclipse, to attach a debugger on the server’s port and start debugging. Note that enabling the remote debugging brings a small performance penalty, but I usually prefer to have the debugging option enabled so that I can connect to the server at will. In a different case, a JVM restart, thus a server restart, would be required in order to apply the new settings.<\/p>\n

First, let’s create the code that will perform the debugging on. We will use a Java MBean<\/a> which gets deployed on JBoss and has a predefined lifecycle. MBeans are managed beans, Java objects that represent resources to be managed. JBoss actually provides an implementation of an MBean Server<\/a>, thus MBeans can be deployed on it.<\/p>\n

The simplest way is to extend the ServiceMBeanSupport<\/a> abstract class and implement a service that conforms to the ServiceMBean<\/a> interface. First we create an Eclipse project named \u201cSimpleMBeanProject\u201d. Then we create an interface that our service will have to implement. The source code is:<\/p>\n

package com.javacodegeeks.jboss;\r\n\r\nimport org.jboss.system.ServiceMBean;\r\n\r\npublic interface SimpleServiceMBean extends ServiceMBean {\r\n\r\n    void start() throws Exception;\r\n\r\n    void stop();\r\n    \r\n    String getName();\r\n    \r\n    void execute(String input);\r\n\r\n}\r\n<\/pre>\n
Then we create the appropriate implementation class:<\/p>\n
package com.javacodegeeks.jboss;\r\n\r\nimport org.jboss.system.ServiceMBeanSupport;\r\n\r\npublic class SimpleService extends ServiceMBeanSupport implements SimpleServiceMBean {    \r\n    \r\n    @Override\r\n    public void start() throws Exception {\r\n        System.out.println(\"Starting SimpleService MBean\");\r\n    }\r\n    \r\n    @Override\r\n    public void stop() {\r\n        System.out.println(\"Stopping SimpleService MBean\");\r\n    }\r\n    \r\n    @Override\r\n    public String getName() {\r\n        return SimpleService.class.getCanonicalName();\r\n    }\r\n    \r\n    public void execute(String input) {\r\n        System.out.println(\"Executing with input \" + input);\r\n    }\r\n\r\n}\r\n<\/pre>\n
The code is really simplistic but with enough functionality for our demonstration purposes. The \u201cexecute\u201d method is the one that will be invoked in order to emulate a running application.
 
<\/div> <\/div><\/p>\n
A way to deploy the MBean is via bundling the two classes into a Service Archive (SAR)<\/a> file. This file is a zipped file that includes the MBean classes and the corresponding deployment descriptor, which in this case is a file named \u201cjboss-service.xml\u201d with the following contents:<\/p>\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n\r\n<service>\r\n\r\n  <mbean code=\"com.javacodegeeks.jboss.SimpleService\"\r\n     name=\"javacodegeeks:name=SimpleService\">\r\n  <\/mbean>\r\n\r\n<\/service>\r\n<\/pre>\n
The \u201cjboss-service.xml\u201d file must reside inside a folder named \u201cMETA-INF\u201d inside the SAR bundle. Then the archive has to be placed inside the <jboss-base-dir>\/server\/default\/deploy directory in order to deploy the MBean. The archive can be created by hand, it is just a zipped file after all, but a more elegant way is to create an ANT script<\/a> that will automate the procedure.<\/p>\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n\r\n<project name=\"SimpleService Project Build\" default=\"build-sar\">\r\n\r\n    <target name=\"init\">\r\n        <property name=\"base.dir\" value=\".\"\/>\r\n        <property name=\"lib.dir\" value=\"${base.dir}\/lib\"\/>\r\n        <property name=\"bin.dir\" value=\"${base.dir}\/bin\"\/>\r\n        <property name=\"src.dir\" value=\"${base.dir}\/src\" \/>\r\n        <property name=\"dist.dir\" value=\"${base.dir}\/dist\" \/>\r\n        <delete dir=\"${dist.dir}\"\/>\r\n        <mkdir dir=\"${dist.dir}\"\/>\r\n    <\/target>\r\n\r\n    <target name=\"compile\" depends=\"init\">\r\n        <echo message=\"Compiling source files...\" \/>\r\n        <javac destdir=\"${bin.dir}\" debug=\"on\">\r\n            <src path=\"${src.dir}\" \/>\r\n            <classpath>\r\n                <fileset dir=\"${lib.dir}\">\r\n                    <include name=\"**\/*.jar\" \/>\r\n                <\/fileset>\r\n            <\/classpath>\r\n            <include name=\"**\/*.java\" \/>\r\n        <\/javac>\r\n    <\/target>\r\n\r\n    <target name=\"build-sar\" depends=\"compile\">\r\n        <jar destfile=\"dist\/SimpleService.sar\">\r\n            <zipfileset dir=\"bin\">\r\n                <include name=\"com\/javacodegeeks\/**\/*.class\" \/>\r\n            <\/zipfileset>\r\n            <zipfileset dir=\"resources\" prefix=\"META-INF\">\r\n                <include name=\"jboss-service.xml\" \/>\r\n            <\/zipfileset>\r\n        <\/jar>\r\n    <\/target>\r\n\r\n<\/project>\r\n<\/pre>\n
When the SAR gets deployed, the \u201cSimpleService\u201d MBean will appear at the server’s JMX Console. This is a web interface that can be accessed at the following URL (replace the host accordingly):<\/p>\n
http:\/\/host:8080\/jmx-console<\/p>\n
Scroll down until you find the \u201cname=SimpleService\u201d entry and follow the link. The Mbean’s attributes, along with a list of operations, will appear there.<\/p>\n
\"\"<\/a><\/p>\n
You can manually invoke the \u201cexecute\u201d method with a String argument and the corresponding input will be written to standard output.<\/p>\n
Ok, after deploying the SAR, it is time to start the debugging. The first step is to have JBoss’s JVM start with remote socket debugging enabled. This is done via JVM’s arguments of course and in order to configure it, you have to do the following:<\/p>\n
Linux platform:<\/span> Open file \/bin\/run.conf and uncomment the line (remove the \u201c#\u201d) that reads
\nJAVA_OPTS=”$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n”
\n
\nWindows platform:<\/span> Open file \/bin\/run.bat and uncomment the line (remove the \u201crem\u201d keyword)
\nset JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n %JAVA_OPTS%<\/p>\n
The port that will be used is 8787. Make sure that \u201csuspend\u201d parameter is set to \u201cn\u201d (disabled) or, in a different case, when the server starts for the first time it will halt and wait for a remote debugger to be attached before proceeding.<\/p>\n
After that, start the server normally. It is now time to connect to the server via Eclipse. Go to \u201cRun ? Debug Configurations…\u201d and then double-click on the \u201cRemote Java Application\u201d option. At the \u201cConnect\u201d tab make sure that the \u201cSimpleMBeanProject\u201d is selected, provide the remote \u201cHost\u201d IP address or hostname (\u201clocalhost\u201d in my case) and the \u201cPort\u201d that the server listens at for incoming debugging sessions (8787 as configured previously). Finally, make sure that the \u201cAllow termination of remote VM\u201d is NOT selected, because if it is, the server’s JVM will shutdown the moment you disconnect the debugging. Not really a nice thing to happen to a production server. Ok, hit the \u201cDebug\u201d button to proceed.<\/p>\n
\"\"<\/a><\/p>\n
If the remote debugging is not enabled or if there is a connection problem (perhaps a firewall issue) you will see the following image:<\/p>\n
\"\"<\/a><\/p>\n
But if everything works correctly, the Eclipse debugger will attach itself to the server and you should be able to see something like this:<\/p>\n
\"\"<\/a><\/p>\n
As you can see, the monitored threads appear in the \u201cDebug\u201d view. If that view does not appear, go to \u201cWindow ? Show View ? Other…\u201d and search it under the \u201cDebug\u201d category.<\/p>\n
Now let’s assume that the \u201cexecute\u201d method of the \u201cSimpleService\u201d class simulates the code that gets executed on the production server with every incoming request. If you were performing debugging on a test server all you had to do is add a breakpoint inside the method, trigger a request and proceed with the debugging. However something like that will definitely not work on a production server. The moment you toggle a breakpoint, all requests will suspend and wait for your action (if the execution path passes from that method of course). That will stop the requests execution and most probably will get noticed by the users. Furthermore, you will be overwhelmed by the magnitude of the requests that you will have to monitor at the same time.<\/p>\n
What you have to do is add a conditional breakpoint that will only halt when specific input is provided, i.e. the one that you provide. So, disconnect from the remote server and then add a breakpoint inside the \u201cexecute\u201d method (at line 23). Then, right click on the breakpoint and from the menu that appears, choose \u201cBreakpoint Properties\u201d (the last one).<\/p>\n
\"\"<\/a><\/p>\n
The properties menu will come up. Check the \u201cEnable Condition\u201d checkbox and inside the textarea, write a condition. The breakpoint will be valid and suspend the execution only when that condition is true. Note that you actually write Java code inside the textarea and that you can use the familiar code assistance for that (using Ctrl+Space). Isn’t Eclipse an incredible tool? We want the breakpoint to kick in only when the method’s argument is \u201cmyinput\u201d.<\/p>\n
\"\"<\/a><\/p>\n
Start the remote debugging again and now you are sure that the execution will be suspended when your very own input is provided. To demonstrate this return to the JMX console and the \u201cSimpleService\u201d MBean view. At the \u201cexecute\u201d method, use a random argument:<\/p>\n
\"\"<\/a><\/p>\n
Hit the \u201cInvoke\u201d button and notice that the execution is not suspended by Eclipse. Now, use \u201cmyinput\u201d as the input value, hit \u201cInvoke\u201d and notice that Eclipse captures the execution.<\/p>\n
\"\"<\/a><\/p>\n
Now you are ready to proceed with the well known debugging options (step into methods, watch variable values etc.) without worrying that the system’s users will be affected.<\/p>\n
You can download the Eclipse project here<\/a>.<\/p>\n
Happy bug hunting!<\/p>\n
Related Articles :<\/i><\/strong><\/div>\n