Monday, April 23, 2012

Distributed Transactions Failure Simulation

If you have many resources which are needed to be updated as one atomic operation, probably you are using XA transactions. For example, you need to change data in database and send notification about this change to external system. Important point to note - db change and notification doesn't make sense in case one of them fails for some reason. Everything could fail, we leaving in not ideal world. The question is how to test the worst case - resource failure and be sure that system is in predictable state before and after transaction. There are could be different approaches for this, but in this blog entry I'm going to describe approach I found really interesting. As for me it's one of the best approaches to ensure that transactions in application are really bullet proof. I'm talking here about "failure injection". "Injection" means that you can declaratively enforce some method to throw exception, without changing any code from service layer. It's really close to simulating real failure. All this possible with JBoss Bytemann (http://www.jboss.org/byteman). This Java agent can do much more, but it's out of scope of this post. They have support for jUnit, a special annotation which will enforce method you'll name to throw an exception. In my case I've played with db and jms XA transaction with Atomikos as transaction manager. As for me its good playground - H2 in-memory database and Apache ActiveMQ embedded in-memory broker. Friendly speaking I'm not fan of jms mocks, why do we need them if there is fully functional real JMS broker, just in memory. You even don't need to start and stop it manually. Let's take mockrunner (http://mockrunner.sourceforge.net/examplesjms.html)-

A few notes on the JMS implementation: The JMS test module is not a full blown message server, but it supports all necessary functionality to test JMS code. The implementation breaks the JMS specification in some cases. Messages are immediately forwarded to a proper receiver, if possible. Otherwise they're stored for later examination. There's no real transaction support, i.e. the framework keeps track if a transaction is committed or rolled back, but does not guarantee transaction atomicity. If you send a message in a transaction, it will be forwarded to the receiver, even if you rollback the transaction.
As you can see some special tricks are required in case of mockrunner, since it haven't atomicity guarantees. I should say that there are some tricky parts in Bytemann and Spring integration. Both of them are using @RunWith annotation for their internal infrastructure setup. Pity thing is that you can't use multiply @RunWith declarations in jUnit. So, we need to have both - @RunWith(BMUnitRunner.class) and @RunWith(SpringJUnit4ClassRunner.class)... oops. Not all so bad. It's possible to eliminate SpringJUnit4ClassRunner usage. Yep, it's not universal solution for multiply jUnit runners, but it's working in this case. To do the same as Spring class runner you can say this:
@Before
    public void setUpContext() throws Exception {
        this.testContextManager = new TestContextManager(getClass());
        this.testContextManager.prepareTestInstance(this);
}
From docs -
TestContextManager is the main entry point into the Spring TestContext Framework, which provides support for loading and accessing ApplicationContext application contexts, dependency, injection of test instances, transactional execution of test methods, etc.
So, final test looks like this:
@Test(expected = Exception.class)
@Transactional
@BMRule(name="Database goes down",
isInterface = true,
targetClass = "com.blogspot.ostas.lora.database.IUserDao",
targetMethod = "save",
action = "org.apache.log4j.Logger.getLogger(getClass())
.debug(\"DB failure simulation\");
throw new java.lang.RuntimeException(\"Simulated Database Failure\")")
public void save(){
   userService.saveAndNotify(user);
    try {
            LOGGER.info("Transaction status : "
            +jtaTransactionManager.getTransactionManager().getStatus());
        } catch (SystemException e) {
            LOGGER.error(e);
        }
    }
Here I'm enforcing DAO component of my service layer to throw runtime exception - it's simulated resource failure. Also, after this I'm checking db state using JDBC to be 100% sure that noting went wrong in case of JMS failure. As you can see, there is no changes in service layer code, no mocks, etc. Thanks for reading. Complete source code of my pet project you can find here: https://github.com/daoway/AtomikosSandbox

Thursday, February 02, 2012

Atomikos distributed transactions scalability

At this moment I'm working on the project where we using distributed transactions. The number of XA Resources in our case is not big at all, but for me is very interesting to know how Atomikos can scale when the number of XA Resources is large. That is why you are reading this blog post. I've desided to push this pet project on GitHub for you to be able to test/play/join the effort, etc. - https://github.com/daoway/AtomikosDistributedScalability

What do I have... I have Atomikos 3.7.1, Spring 3.1 (if you are still using 3.0 consider migration to 3.1 - it's better) and Hibernate. In this particular case I've decided to use embedded in memory H2 database to eliminate possible network overhead and to simplify setup of underlying farm of XA Resources. It's possible because H2 has XA compatible data source implementation. BTW, it can be used also for unit/integration testing instead of Oracle XA Datasource to test transactional behavior for example.

So, I'm playing with H2 XA Army. I have a lot of in-memory H2 database instances, one simple domain object and Hibernate. I want to persist this object across all nodes in XA transaction and calculate time for this effort. Then increase the number of XA resources for some Delta and repeat this procedure until I'll get OutOfMemoryException. Just kidding :) All this stuff is in-memory and RAM amount is limited, for sure, the maximum of XA resource I've played with was 2000.

Here is the result of my experiment (click on link to see Adobe Flex chart with test results) :


As you can see, replication time grows linear, together with number of XA resources participating in "distributed" transaction. If so, we can conclude that Atomikos has linear scalability.

Thanks for reading. Any comments/suggestions are welcomed.

Tuesday, August 02, 2011

Low memory in Cloud Bees

I've noticed that memory goes low at Cloud Bees RUN@CLOUD service. As I wrote earlier, about near-real time physical memory monitoring, this application works, but sometimes it's just not available. This is the image I've faced 5 min before writing this post :

Friday, July 08, 2011

Adobe Flex Mobile On Android - first application

I think that new feature of Adobe Flex 4.5.1 Mobile is really cool stuff !!! I was able to convert existent application from my previous blog post to application for Android platform. Didn't tryed iOS support yet, but for Android platform things looks good.

So, few words about app. I have free CloudBees run@cloud account. Application from my previous blog post showing live data feed about physical RAM memory usage on server, one frame per second. Long AMF pooling used there to fetch fresh data. And Flex charting for visualization.

I have to admit that in mobile version plain AMF pooling is used, because I had no luck with long AMF pooling.

You can pick up this application from here -> Memory monitoring Adobe Flex Mobile App.

Maybe CloudBees will want to have it ? :)

Wednesday, July 06, 2011

Memory monitoring on CloudBees RUN@cloud instance

I've been playing with RUN@cloud for some time. In fact it's former Stax service provider. Friendly speaking I was disappointed a little because they decided to decrease amount of RAM available for Tomcat from 256 Mb to 128 Mb for free account. It doesn't affected my playground apps there, but the fact... From other point they have something good for Open Source projects. Maybe I'll try it soon.

So, regarding this changes, I've desided to write RIA application for live RAM monitoring. I mean not JVM memory, but actual RAM amount. I believe this makes sense, even for all things which are even not Java related. Yep, I know there are a lot of them available for free, some of them are commercial. The key feature of my application here is that we have near real time memory charts - 1 second between measuring times + some network roundtrips time + time for rendering. And all this in fancy Adobe Flex client.

Next thing I'm going to do is to create the same kind of application, but for Android. It's possible with Adobe Flash builder 4.5. Among tons of games, organaizers and other stuff this application could be useful for admins, support team, whatever.

You can take a look at my app here : http://monitoring.daoway.staxapps.net/

Any feedback apprececiated. Thanks.

Wednesday, May 25, 2011

Loading Spring Beans based on environment

In complex deployment environments it makes sense to keep web application monolith war file and provide appropriate instructions for deployment team. It's possible only if your application have flexible configurability strategy, so that at least you can switch from DEV to UAT, from UAT to PROD environments and be sure that only thing has been changed is environment specific stuff. Forget about unpacking and patching war-file, it could be prohibited at all because of non-trivial distribution mechanism.

For Spring-based applications all this could look a little bit scary for a first look, you have your applicationContext.xml with placeholders, which is ok, or JNDI for enviromnent-specific resource lookup. Great, but what you are going to do if some of the parts of resources are present in one env. and absent in another ? For example +1 in dev, -1 in uat and +1 in prod ? Maybe there is not only one way to do it, but I came across the following solution for this.

It's based on combination of servlet context init params and Spring bean re-definition at runtume. Thanks to Spring it's possible. In terms of configurability servlet init params is very flexible thing. You can define one in web.xml, and then redefine it in context setup. I mean externally, so, your war file is still don't need to be patched. Other stuff is technical details for this - key point to implement this is Spring bean which implements both ServletContextAware and ApplicationContextAware interfaces. First makes possible to access Servlet context, second - Spring application context. Runtime re-defintion is not so hard - take a look here.


Complete source code you can find here.

Thursday, April 28, 2011

How to turn off popup page preview in Google search results

Did you enjoyed this feature ? This fancy useless annoing popup page preview images ? Ohh yeess... There is no way to turn it of using some kind of chekbox, so here is the solution for this.

1) Install Firefox extenstion called Grasemonkey - https://addons.mozilla.org/ru/firefox/addon/greasemonkey/

2) Install script for Grasemonkey that will disable this shit - http://userscripts.org/scripts/show/90222 just click on install button at this page.

3) Say goodbye to page preview popups. Enjoy.

I believe it's hard to implement this kind of feature from technical perspective. I mean preview of web page. You have to deal with rendering HTML, CSS, JavaScript... a lot of work, but...

From user's perspective it's useless bullshit !