Monday, April 23, 2012

Distributed Transactions Failure Simulation

If you have many resources which are needed to be updated as one atomic operation, probably you are using XA transactions. For example, you need to change data in database and send notification about this change to external system. Important point to note - db change and notification doesn't make sense in case one of them fails for some reason. Everything could fail, we leaving in not ideal world. The question is how to test the worst case - resource failure and be sure that system is in predictable state before and after transaction. There are could be different approaches for this, but in this blog entry I'm going to describe approach I found really interesting. As for me it's one of the best approaches to ensure that transactions in application are really bullet proof. I'm talking here about "failure injection". "Injection" means that you can declaratively enforce some method to throw exception, without changing any code from service layer. It's really close to simulating real failure. All this possible with JBoss Bytemann (http://www.jboss.org/byteman). This Java agent can do much more, but it's out of scope of this post. They have support for jUnit, a special annotation which will enforce method you'll name to throw an exception. In my case I've played with db and jms XA transaction with Atomikos as transaction manager. As for me its good playground - H2 in-memory database and Apache ActiveMQ embedded in-memory broker. Friendly speaking I'm not fan of jms mocks, why do we need them if there is fully functional real JMS broker, just in memory. You even don't need to start and stop it manually. Let's take mockrunner (http://mockrunner.sourceforge.net/examplesjms.html)-

A few notes on the JMS implementation: The JMS test module is not a full blown message server, but it supports all necessary functionality to test JMS code. The implementation breaks the JMS specification in some cases. Messages are immediately forwarded to a proper receiver, if possible. Otherwise they're stored for later examination. There's no real transaction support, i.e. the framework keeps track if a transaction is committed or rolled back, but does not guarantee transaction atomicity. If you send a message in a transaction, it will be forwarded to the receiver, even if you rollback the transaction.
As you can see some special tricks are required in case of mockrunner, since it haven't atomicity guarantees. I should say that there are some tricky parts in Bytemann and Spring integration. Both of them are using @RunWith annotation for their internal infrastructure setup. Pity thing is that you can't use multiply @RunWith declarations in jUnit. So, we need to have both - @RunWith(BMUnitRunner.class) and @RunWith(SpringJUnit4ClassRunner.class)... oops. Not all so bad. It's possible to eliminate SpringJUnit4ClassRunner usage. Yep, it's not universal solution for multiply jUnit runners, but it's working in this case. To do the same as Spring class runner you can say this:
@Before
    public void setUpContext() throws Exception {
        this.testContextManager = new TestContextManager(getClass());
        this.testContextManager.prepareTestInstance(this);
}
From docs -
TestContextManager is the main entry point into the Spring TestContext Framework, which provides support for loading and accessing ApplicationContext application contexts, dependency, injection of test instances, transactional execution of test methods, etc.
So, final test looks like this:
@Test(expected = Exception.class)
@Transactional
@BMRule(name="Database goes down",
isInterface = true,
targetClass = "com.blogspot.ostas.lora.database.IUserDao",
targetMethod = "save",
action = "org.apache.log4j.Logger.getLogger(getClass())
.debug(\"DB failure simulation\");
throw new java.lang.RuntimeException(\"Simulated Database Failure\")")
public void save(){
   userService.saveAndNotify(user);
    try {
            LOGGER.info("Transaction status : "
            +jtaTransactionManager.getTransactionManager().getStatus());
        } catch (SystemException e) {
            LOGGER.error(e);
        }
    }
Here I'm enforcing DAO component of my service layer to throw runtime exception - it's simulated resource failure. Also, after this I'm checking db state using JDBC to be 100% sure that noting went wrong in case of JMS failure. As you can see, there is no changes in service layer code, no mocks, etc. Thanks for reading. Complete source code of my pet project you can find here: https://github.com/daoway/AtomikosSandbox

No comments: