« Coherent clustering and talking past each other | Main | QCon Panel: Which clustering to use when »
November 21, 2008
Coherent clustering revisited
posted by ari
Ok. I thought more about it.
Remember the CAP theorem
Consistency
Availability
Partitioning
Flaw in my thinking is partitioning is all about "surviving a partial network outage".
So, Terracotta is Consistency + Availability with weak partitioning guarantees. Weak means that we will do our best to survive network outages by trying to find a path through the cluster from one half to the other. If we can reach through to the other side from some route, we will then shoot the smaller cluster in the head. If we cannot reach to the other side of a partition till after the network has been restored, we will shoot the smaller cluster in the head at the time of restoring the network. This being said, there are very few scenarios under which we end up partitioned. We do not care of your app nodes can reach each other; this is not a partition to us. We do not care if the stripe groups or partitions in our array cannot reach each other as they share nothing anyways. We only care if our mirror groups inside our array cannot reach each other--this gives a reasonable ability to survive network partitions.
In Coherence they provide Consistency + Availability with weak partitioning guarantees. Their definition of weak is different; simpler on the surface. They let partitions occur and simply merge back together when the network is healed. This is seemingly elegant and simple except "merge" is defined as last change in wins. While last one in semantics works great for WAN-based use cases where traffic is well partitioned in the first place, it doesn't help much for a single app cluster where an internal switch has failed and users are running roughshod all over your cluster, making some updates to one partition and other updates to the other. Your data is pretty much trashed if the system doesn't shoot one half of the cluster in the head. No?
So, if both are C+A what's the difference?
Terracotta's consistency guarantees are for app nodes trying to share data. Take a round robin load balanced web app with HTTP session replication. Terracotta will end up with the session resident in all JVMs and will broadcast deltas to sessions as sessions get updated. This means the session objects are _consistent_ in all JVMs at all times. (Yes it is fully ordered in this degenerate round-robin case.)
Coherence would have lots of trouble in this round-robin scenario because all changes would serialize up and broadcast the session. So they ask you to partition your data. Regardless which Tomcat instance I hit, my session is stored on a fixed node in the cluster and thus I pay higher latency update price to hop servers to my session, but there is no need to try to keep the cluster consistent inside Coherence which would be slow and not scalable.
Don't get me wrong. This is a degenerate case and round-robin will not scale well for either vendor. It just makes the point clear and tangible--the point is that consistency in Terracotta is provided for any an all use cases as designed in the app. Coherence meanwhile will never scale in a P2P broadcast everything everywhere mode and thus falls back to a model where objects get assigned to nodes and stay there.
Partitioning in CAP is all about network partitions. Partitioning in Coherence is about performance (divide up my data so I don't share too much info in too many places). partitioning in Terracotta is about performance as well but we ask you to load balance your traffic so that we can inherit your organic partitioning automatically.
In summary:
C+A == good hard problem to solve and the most valuable use case for most enterprise apps as surviving network partitions is something that quickly degrades to involving human intervention anyways.
C for Terracotta is all about pushing deltas amongst JVMs that are sharing objects
C for Coherence is all about asking the developer to partition the data set so that Coherence can efficiently go back to a fixed node for all updates.
A for Terracotta is all about keeping all data on disk at all times and mirroring data to a 2nd disk or third for backup.
A for Coherence is all about keeping all data on a fixed node plus a buddy in case of node failure. Lose both nodes and the data is lost.
Terracotta consistency is automatic and works in many use cases. Terracotta's availability is a much higher guarantee (on disk at high throughput). And Terracotta's partitioning resilience is easier to live with (Shoot the other node in the head rather than "last one in wins" merge).
I would pick Terracotta's balance of CAP almost every time. But you decide for yourself.
--Ari
Trackback Pings
TrackBack URL for this entry:
http://blog.terracottatech.com/cgi-bin/mt/mt-tb.cgi/77
Comments
Interesting post Ari. I could be wrong, but the last time I read the Coherence docs, in partition mode, coherence will copy each partition to 2 or more nodes. So even if 2 nodes go down, there may still be a third or forth node with a copy of that partition.
Posted by: Peter Lin at November 21, 2008 1:55 PM
Peter,
You are correct. Coherence can be configured w/o any custom code to make as many copies of each partition as you would like. The problem is you are trading off availability for performance. Those copies are synchronously made, after all!
--Ari
Posted by: ARI ZILKA at November 22, 2008 9:18 AM