Safe refactoring using Scientist

Refactoring is a critical yet often overlooked activity in the product development lifecycle. Despite its importance, teams tend to neglect it until they encounter significant showstoppers during the development process. Several factors contribute to teams neglecting refactoring, including.

• Pressure to meet product release dates
• Concerns about production stability
• Difficulty in writing high-quality unit and functional tests

When teams encounter these roadblocks, they often make a business case for refactoring, reengineering, or implementing new technology. Although there may be resistance, the product team typically agrees to it, and it is integrated into the development sprints. If you see a specific tech debt item in the sprint backlog, it is a clear indication that the code's health was not maintained due to business priorities, and now it is time to address it.

To put it in financial terms, failing to maintain code health is like missing an installment payment and incurring additional interest from your banker.

Integrated Development Environments (IDEs) have evolved to provide safe and efficient refactoring options such as renaming variables, extracting methods, simplifying branch conditions, inlining methods, and moving code. While these options are generally beneficial, some refactoring tasks are more complex and riskier. These tasks may involve changing the implementation of core components, such as modifying persistence, altering core algorithms, or adjusting underlying data structures to improve performance.

Some of the way to test such refactoring is by using feature flags or A/B testing.

While browsing through GitHub, I came across Scientist, a library that provides a way to verify critical refactoring. It offers an intuitive approach to code verification. It is based on experiment , observation & verification.

Let's take a look at some code snippets.

Experiment<Integer, Integer> experiment = new Experiment("Next Experiment");

experiment
        .withControl("BitCount Using binary string", x ->
                (int) Integer.toBinaryString(x)
                        .chars()
                        .filter(y -> y == '1')
                        .count()
        );

experiment
        .withCandidate("BitCount using native", x -> Integer.bitCount(x));

experiment
        .withParamGenerator(() -> 100)
        .compareResult("bit length", (control, candidate) -> control == candidate);

experiment
        .run()
        .publish();

This library has several components, including:

• Control function
• Candidate function
• Experiment parameters
• Result comparator function

Once you specify these parameters, you can run experiments. As you begin to use the library for more complex problems, additional considerations may arise, such as the number of times the experiment should be run, whether to run them in parallel, and setting timeouts

experiment
        .withControl("BitCount Using binary string", x ->
                (int) Integer.toBinaryString(x)
                        .chars()
                        .filter(y -> y == '1')
                        .count()
        );

experiment
        .withCandidate("BitCount using native", x -> Integer.bitCount(x));

experiment
        .withParamGenerator(() -> 100)
        .compareResult("bit length", (control, candidate) -> control == candidate);

experiment
        .times(100)
        .parallel()
        .run()
        .publish();

Other uses of Scientist

Thus far, we have discussed this library's potential for safe refactoring, but it can also be utilized for running experiments alongside real production code. This allows for the experiment to be run under the same constraints as the current code and produce useful feedback.

Since we are discussing experiments, it would be wise to store the results in a database or another system that can keep a log of the experiments.

In many cases, running these experiments can be costly, so previous results can be utilized to verify new code.

Furthermore, this library can be used to test multiple variations of new logic and select the most optimal one.

Code used in this blog is available @ github

Testing using mocks

Mock objects are very useful if used right way. I shared some of the experience of using Mock Objects in need-driven-software-development-using post.



In this post i share 2 things
- Contract based testing using mocks.
- Patterns to organized mock code.


Contract based testing 
Lets take scenario where you are building Money remittance service. Key component in such type of service is Currency Converter , Bank Service & FX Service.

50000 feet design of fictitious forex service will look something like below.

We have to write FX Service that needs Currency converter & Bank Transfer service. This is perfect scenario for contact based testing. 

Code snippet for FXService

Our new FX service has to follow below contract

• Interact with currency converter & Bank Transfer based on input/output contract.
• Makes 1 call to each of service.

One way to test FX service is to call the real service but that means slow running test and dependency on service that it has to up whenever our test is executing. Sometime calling real service is not an option because it is not developed yet. 

Smart way is to mock these collaborator( Currency Converter & Bank Transfer) and verify interaction using mocking framework.

Another advantage of testing with mocks that it enables to verify that both currency & bank transfer service are used by fxservice in expected way.

Lets look at mock based test.

This test is written using EasyMock framework and is mocking reply from collaborators.   

Write the test that you want to read

One of the important property of good test is that it is enjoyable to read. 

Mocks can make this goal more difficult to achieve as setup code for unit test will have very complex assembling logic that will be mix of some normal object set and some mocking expectation. I am sure you have seen before function in test that is used as dumping ground for setup required for all the tests in class. 

Lets look at some mock code we used earlier and try to improve it

Another way

Both of the above code is doing same thing but later one which is written with jmock has nice sugar method to express same thing.
This helps in keeping expectation clean and in context with code that is being tested. Collaborator object in the context are mocked out.
Simple pattern but very effective in making test readable.

Code used in this post is available on github

Need driven software development using Mocks

Excellent paper on mocking framework by jmock author. This paper was written in 2004 that is 18 years ago but has many tips of building maintainable software system.



In this post i will highlight key ideas from this paper but suggest you to read the paper to get big ideas behind mocking and programming practice.

 Mock objects are extension of test driven development.

Mock objects can be useful when we start thinking about writing test first as this allows to mock parts that is still not developed. Think like better way of building prototype system.

Mock object are less interesting as a technique for isolating tests from third-party libraries.

This is common misconception about mock and i have seen/written many codes using mock like this. This was really eye opening fact that comes from author of mocking framework.

Writing test is design activity

This is so much true but as engineer we take shortcut many time to throw away best part of writing test. Design that is driven from test also gives insights about real problem and it lead to invention because developer has to think hard about problem  and avoid over engineering

Coupling and cohesion 

As we start wiring test it gives good idea on coupling & cohesion decision we make. Good software will have low coupling and high cohesion. This also lead to functional decomposition of task.
Another benefit of well design system is that it does not have Law_of_Demeter, this is one of the common problem that gets introduced in system unknowingly. Lots of micro services suffer from this anti pattern.

Need driven development
As mocking requires explicit code/setup, so it comes from need/demand of test case. You don't code based on forecast that some feature will required after 6 months, so this allows to focus on need of customer. All the interfaces that is produce as result of test is narrow and fit for purpose. This type of development is also called top down development.

Quote from paper
"""
We find that Need-Driven Development helps us stay focused on the requirements in hand and to develop coherent objects.
"""


Programming by composition

Test first approach allows you to think about Composability of components, every thing is passed as constructor arguments or as method parameter.
Once system is build using such design principal it is very easy to test/replace part of system.
Mock objects allows to think about Composability so that some parts of system are mocked.

Mock test becomes too complicated
One observation in paper talks about complexity of Mock Test.
If system design is weak then mocking will be hard and complicated. It does amplification of problems like coupling, separation of concern.  I think this is best use of mock objects to get feedback on design and use it like motivator to make system better.

Don't add behavior to mock
As per paper we should never add behavior to stub and in case if you get the temptation to do that then it is sign of misplaced responsibility.

If you like the post then you can follow me on twitter to be notified about random stuff that i write.

Preparation of Coding Interview

I want to share my coding interview experience, so i have decided to write blog series about questions that are asked and this is first one in that series.

Coding interview are very common these days and if you find company that does not have it as part of selection process then i think you should stay away from them.

Lets start with why coding interview is required and reason is very simple especially in times when every company is adopting agile and they don't want to wait for months after hiring to know about coding skills of developer.

So that is the reason why it is the first gate that you have to pass as developer. 

Jeff Atwood share interesting experience about why-cant-programmers-program and type of question that is asked to filter out candidate. FizzBuzz is very good question to start with. 

I would advise to try FizzBuzz to make sure you are not surprised and think about testing also because we are in world of TDD. 


What types of question are asked

Different types of question are asked in coding interview and it depends on type of company you apply

 - Algorithms & Data structures

This is very common for Silicon valley company or product development company and some serious preparations is required for these types of interview if your daily work involves only playing with frameworks(i.e hibernate/spring etc). 

Things become more difficult for such interview because may developers(like me) are focused so much on learning emerging framework to build bleeding edge Resume that we tend to forget basic algorithm and start tagging ourself as some XYZ framework developer and company also help to the cause by putting such job requirement.

- Simple problems using TDD

This type of questions is also very common and it is make sure that you don't leave testing to QA team and write a code that is defect free , easy to understand and maintainable.

- Refactoring problems

You don't work on green field project every day and most of the time is spent in reading others code and adding features to it, so employer want to know "how do you quickly understand code written by somebody else and add new feature to it".

TDD becomes more important in such questions because it can be used to build all the safety net around existing code before you start changing it.

Above question can be asked as pair programming question or take a home type of question, so you have to build solution based on time given to you. 

Some of the things employer wants to check are

 - Simplicity/Readability of code.
 - Maintainability of code. 
 - Design principal usage.

Bottom line is you have to prepare for coding interview to survive in fast changing market.

Stay tuned up for questions.

Working Effectively with Legacy Code

Recently i attended Working Effectively with Legacy Code course by Michael Feathers

It was very good learning experience , he talks about how to work with code that does not have unit test or less unit test. He shared techniques can be used to improve legacy code and get better understanding of application.

This post is to share some of them before i forget:-)
  

Sprout Method/Class

This is pretty common technique but did't know that it has name. Adding new method/class sounds much easier than changing some existing code for new feature, so we should use this approach for any new feature that we want to introduce.

This approach can be used on existing method also to make it testable.

Work of caution that you don't want over do it !

Poor Man dependency injection

Everybody knows what dependency injection is, apart from some of the benefit it can also be used to make code testable, so for unit test you can have sample/dummy implementation that can be injected to your application code to make is testable. 

One of the problem with this technique is that it will result in method signature changes and that might mean that all the code in that call stack might need to change.

Just remember that you don't have to use spring to do this!

Extract and override

This is interesting one looks like silver bullet or Brahmastra for many problem.This pattern is used to have control on dependency that are hard to fake.

Assume there is function that makes some database/socket call and then does some calculation and you want to write unit test for calculation logic.

To make function testable you can do below changes

• Extract database part of logic and put that in new function
• Make that function protected. Thanks to OOPs , finally some good usecase for protected.
• Write new class that extends original class and only override database interaction function(i.e was protected)
• Use new class for your testing and you are done!

This is very powerful technique because you don't have to go through pain of changing  constructor/method parameter, so call stack remains intact.

Since it is based on method overriding, so you need to have discipline in team to make sure that fake class is only used for testing. 

Instance delegate

Used to test static function class. Create normal function that will delegate call to static function and during test create another class that will override new function that was created to add fake call .

Singleton 

Singleton make life very difficult from testing perspective, way to make it testable is allow injection of new singleton implementation and use that implementation for testing. 

Create interface to break big class

Although adding new method/class is much simpler but still lot of code is added to existing class/method and over period of time it becomes big i mean really big.

Approach to simplify class

• Creation of interface without any method 
• Class that you want to simplify should implements new interface
• All the function that was using class should now use new interface and compiler will gives you errors about missing method and you can start moving methods to interface to fix the issue.

Main advantage for this approach is that you don't have prerequisite of unit test , you can take benefit of compiler/IDE feature. 

Method & variable dependency graph

 When class grows overtime and it does not adhere to Single responsibility principle, working out what functions goes where could be tedious.

You can draw dependency graph of variable & method to workout how things are related. This can help to come up with new classes will will adhere to single responsibility principle.

Seam

Identifying seam plays important role in working out hidden dependency. 

Definition of seam from dictionary is 

A line of junction formed by sewing together two pieces of material along their margins.

Definition in context of re-factoring is - part of code will enable testing.

for e.g functions performs some I/O operation and then some calculation, so if you want to test calculation without doing any major changes to core logic then you can "Extract & Override" approach to solve this.

Scratch Refactoring
I am sure you might have seen one big class/methods with 100s or 1000s of lines and you have to scratch your head to understand what this does.
To make things more interesting that part of code is very critical to company and you have to do some changes.
Scratch Refactoring is very useful for such type of situation

• Take monster code that you want to simplify/understand
• This type refactoring is only focused on understanding code, so it should be done in palin text editor so that you are not worried about compilation error that IDE generates.
• Break big method in small and single concern method, simplify condition , delete some unused code

Main benefit of this approach is that you don't have to commit this in Svn/Git only purpose of this work is understand as much as possible by creating small code blocks.

While you are doing this you will get better understanding and code is simplified to extent that doing real re-factoring will be not be that difficult.
 
 Conclusion

I must say it was very useful session, lot of learning and fresh perspective.
I have got copy of Working Effectively Legacy book and will read & practice it to get more better understanding.

TDD - Roman Numerals kata -

Interesting experiment on TDD, i knew it is better but did't expect that proving it could be this simple .

Reference Blog : http://www.codemanship.co.uk/parlezuml/blog/?postid=1021