IF YOU BUILD IT

... Will they Come?

November 2006

Subsequent to my previous post, I received a few questions about how to handle split brain. Before answering, here's a short recap:

When you have a master-slave scenario, there are cases where you rather have no active master rather than having two (or more) masters running at the same time (split brain, in other words). The question is, how can you make sure no more than one server will be master when the master and slave get disconnected?

The obvious answer is to use one of the commercial products like Microsoft Cluster Server, Veritas, Linux-HA, and so on. Going with a commercial product is especially recommended if you need complex scenarios like geographically dispersed clusters, and the like.

Nevertheless, if you do find yourself in situations where you need to roll your own solution besides the obvious bit you need to establish between the server, you also need to solve the split brain problem of deciding who is master.

There are two approaches I know of (that work in commercial products):

• Have some resource which everyone can see (communicate with). The first to obtain a lock on the resource is the master if a server doesn't see the resource, it doesn't consider yourself part of the cluster.
• Another approach is based on voting. If you have three computers, the one that gets two votes wins.

Note that these strategies work for situations where there no master is better than two masters.if your requirements are different you need to consider other options.

P.S.
I got one comment from someone who thought the previous post was offensive for that applicant.This perspective didn't occur to me when I posted it. I am not sure he reads my column at all -- but if you do and you've found that post offensive, I heartedly apologize. That was not my intention. By the way, the actual applicant I interviewed is a brilliant person who has great design skill, and great depth of knowledge in the technology (.Net in that case). I would hire him as a lead developer without thinking twice. I just didn't find him ready for an architect role -- yet -- for the reasons I pointed out in the post

Posted by Arnon Rotem-Gal-Oz at 05:50 PM  Permalink |

About a week ago I interviewed a bright individual who is currently working as an Architect. We had a nice interview and I was impressed with the depth of his knowledge. Everything went well until he raved too much about the optimization he did for over the default serialization of the platform (.NET in this case). [strike 1].

The next thing I asked him then was "let's say you have a distributed system with a master and slave, when the master goes down for any reason, the slave should wake up and become the master. How can we prevent both the master and slave from being master when the communication between them breaks [a problem which is a called a split brain]".

The applicant thought about it for a while and told me that it isn't possible. Since when the communication is down the hearth beat between the computers will fail in both nodes and both will become master (thinking the other node failed). [strike 2]

Well, I said, I know several commercial products that do deal with this problem and don't suffer from split brains

"Okay", he said, tell me about them, "and I'll show you where they are broken.." [strike 3, you're out!]

I did tell him about the solution, but I also didn't hire him because:

• If you find yourself at a point you need optimize serialization -- it is, in my opinion, a sure sign that you are moving too much data back and forth -- which is most probably, the real problem in your architecture (see the fallacies of distributed computing)
• Depth is not enough, you also need breadth of knowledge
• The more important thing here is that it doesn't matter how bright you are, you need to know there are other bright people walking around the plant who might be just as bright as you are. So that:
  • You don't need to reinvent the wheel every time
  • You are not always right -- others may have a better solution

I call this person "half baked" because he does have potential, but he is in the stage where he thinks he knows best. You always need to be open to learn from other people. The understanding that you don't know everything and that you always need to learn is crucial for an architect.

Posted by Arnon Rotem-Gal-Oz at 12:03 AM  Permalink |

I've just read Scott Ambler's "Scaling Agile Development Via Architecture". I was especially happy to see it was published in Agile Journal, since it is nice to see that the realization that architecture is essential (especially for larger projects) is sinking in with the agile community.

I've been saying that for a long time: Agile projects are not different from other projects -- you need software architects (see " Agile Architect", which I posted when I started this blog in April). I've also talked about many of the points Scott mentions in his article. For example:

• "Prove it with code." I wrote about Evaluating Software Architecture in Code
• "Think through the big issues up front". See the post I made on JEDUF (Just Enough Design Up Front).
• "Allow good architectures to emerge over time". See the "Architecture phase" which I posted a few days ago (by the way, you can download both this post and the second part as single whitepaper from here).
• "Model with others". I wrote about the benefits of Pair Architecting

The bigger the project, the more it is important to think about architecture -- setting your boundaries right will enable things like Scrum of Scrums that otherwise would be (in my opinion) impossible. Anyway, if you haven't done it by now go read Scott's article it is an excellent read.

Posted by Arnon Rotem-Gal-Oz at 01:11 PM  Permalink |

In partnership with Microsoft, IASA, the International Association of Software Architects, has initiated an interesting project. Based on the work of the taxonomy (of the software architecture) workgroup, IASA assembled 119 skills that are beneficial to architects (soft and hard). The project is to write a short paper that describes each of these skills.

Here is an excerpt from the original announcement by IASA:

This fall, the International Association of Software Architects (IASA) is developing a full skills package for the IT architect. The project fulfills a primary component of the IASA mission which is to allow the open architect community to guide the overall direction of the profession. The goal of the project is to deliver a series of 3-5 page white papers on the skills necessary to function as an IT architect. This material will be freely available to the community and will provide a solid foundation for an aspiring architect to understand the capabilities of a senior IT architect regardless of specialization.

Purpose
To create an overview of the state of the art of IT architecture, compiled from responses to targeted questions and discussion with, the world's dominant professionals and thought leaders on a series of particular, focused topics. The outcome of the overview will be a series of white papers targeted to the senior developer and aspiring IT architect

I will contribute one paper to this endeavor (as I hard have any free time left between my blogs and my book) but many other (probably more capable) senior architects will be delivering these papers. The project is due for January 2007, and I can hardly wait to see how it turns out. This is a good news for the architect profession in general, and for aspiring architects in particular.

Posted by Arnon Rotem-Gal-Oz at 02:12 PM  Permalink |

The business rationale behind going on the SOA road is the ability to increase the alignment of the business and IT. Consequently, we divide the business into a bunch of business services and everything is just fine. However, the minute we start diving into the SOA implementation details, we are swamped by a horde of technologies, cross-cutting concerns (auditing, security, etc.), and whatnot.

For example, in one project I was involved with, we implemented an SOA over a messaging middleware (Tibco's Rendezvous). Just when everything was fine and dandy, along came another project which could potentially use a few of the services. Well, almost. It needed a slightly different contract and it also used completely different wire protocol -- WSE 3.0 (Microsoft interim solution for the WS-* stack before Windows Communication Foundation). And that's just one simple example. Cross-cutting concerns and implementation details are everywhere. The question is then:

How can you handle cross-cutting concerns like multiple technologies, protocols, changing policies, etc., while keeping the service focuses on its core concerns -- that is, the business logic

You can get the full pattern from here.

[This is an early draft of one of the Service Structural Patterns from my SOA Patterns book.]

Posted by Arnon Rotem-Gal-Oz at 01:38 PM  Permalink |

In a previous post, I explained that architects need to stay around to make sure the architecture fits the solution (and that it would likely change and evolve in the first few iterations of the project).

In this post I examine the other issues that Eliyahoo's question raises:

• How do we know someone does not adhere to the architecture?
• What do we do once we find such a violation?
• How can we avoid architecture violations in the first place?
• How involved should the architect be?

Before looking into what can be done, let's take another look at the root cause of the problem -- why would anyone want to circumvent, change, or ignore the (hopefully) wonderful architecture you created, even if you kept it relevant (following my previous post, of course :-) )

Some people would do that because they have other interests which conflict with your's or they just don't care -- most people, however, are concerned with the project success just as much as you are but:

• They flipped the bozo on you -- they think they know better.
• They don't see the big picture and they try to optimize locally.
• They don't know any better, they just do what they always did.
• They believe they are following your design but they didn't really understand what you've meant (or alternatively -- you didn't explain yourself very well).
• They cut corners to meet deadlines (just to appease the project manager).
• Etc.

As was the prerequisite for understanding there is something wrong with the architecture, you need to stay involved with the project. Designing an architecture (as elegant and great as it may be) and disappearing will most likely (if not always) not cut it. As time goes by the architecture will deteriorate. How involved should you be? There is no magic number I know of, in my experience the first third of the project as well as the iteration(s) when approaching a release milestone are more prone to trouble vs. the rest of the project but your millage may vary.

Things you can do to find existing violations include reviews (code and design), pair programming (with the developer in the driving seat most of the time and the architect sitting with her), staying approachable (make people feel at ease talking to you), positioning yourself as a "mentor" or "coach" (someone people would want to come to when they don't understand or know something). You can see I only mentioned one command-and-control approach vs. three collaborative approachs -- even though the task is to enforce the architecture, achieve governance) etc -- getting there by collaboration (even though it takes more effort on your part) is always better. Even if you do get your way by pulling rank or whatever, people will not like working with you in the long run. (At least that was my experience until, I changed my ways.)

When you find that something is amiss, you sometimes need to be curt, right the wrong, and be done with it. As a general rule, however, try to avoid that and instead explain your decisions, listen to other views, and try to persuade you are right. Developers are smart people. They usually understand the bigger view (they just don't have the time to attend to that).

I don't think you can avoid all "violations", but to minimize violations you need to be aware that they can happen, be attentive for what happens in your project and try to work on your soft-skills, like leadership, organizational politics, communications and human relations. (I'll cover both communications and human relations soon.)

Both this and the previous posts touch the D(eployment) activity of my SPAMMED architecture framework. The common denominator between them is that you, the architect, need to stay involved in the project after the moment in time where you declare the architecture "ready". This deployment of the architecture may not take 100% of your time (though in a large, multi-teamed project it is not likely). You can use the "free" time to participate in the coding or work with additional projects -- anything goes -- as long as you don't neglect your responsibilities as an architect for the project and help maintain its designated quality.

Posted by Arnon Rotem-Gal-Oz at 11:31 AM  Permalink |

I got this question from Eliyaho:

[Everything is nice and dandy until the "architecture" gets into the hands of the architecture breakers. Many times I've found myself (just seconds from releasing all my acquired know-how) in hand-to-hand combat with some developer. As you said, the architect's role includes tight supervision and aggressive code review all along the way. More than once I've seen code like MyView.MyController.MyFacade.MyBL.MyDal.FooAccessor.GetValue or similar rubbish, and then I need to make the developer redo all his (hard) work. This is where the architect needs to have authority. Sometimes I wait for developers to take a vacation to refactor their code and I excuse this by saying we needed something urgent and he was not present (so that he will not take offense). My question is at what level of coercion should the architect use -- at the module and core guidelines level or at the "each and every class" level. What happens is that during the project, the architecture is not good enough -- it is lacking, either because requirements change, poor performance or over flexibility. These shortages decrease the value people see in the architect -- which is a problem.

Well, There are few issues we need to address here:

• How do we know someone does not adhere to the architecture?
• What do we do once we find such a violation?
• How can we avoid architecture violations in the first place?
• How involved should the architect be?
• How do we make sure "the architecture" is relevant throughout the project?

I'll address the last issue (ensuring the architecture is relevant) in this post and the rest of the issues in the next post.

There is no "architecture phase" per se. The architecture should stabilize in the first few iterations and it would probably not change much after that (unless you get major requirements changes -- which will prompt you to re-evaluate and possibly redesign). Note that the architecture is dependant mostly on quality attributes of the system and these tend to be more stable than functional requirement (even though, in a startup I worked for, we once made a wee change in our target customers --from SMBs to Fortune 500 which really turned almost everything, except maybe the splash screen, upside down.)

Thus, in my opinion the question should have stated "What happens is that during the project, the [initial] architecture is not good enough", which would be a reasonable statement since the initial architecture matches your initial understanding of the system. Unless the project is really trivial, I do find it useful to have an initial architecture phase (where you don't deliver features and concentrate on architectural decisions) -- I usually try to limit this initial phase to one iteration (usually 1 month) and have it end with a working skeleton that demonstrates how all the components play together.(e.g. by implementing one thread through a use case or implementing a central user story). But, again, this does not mean the architecture is "done" -- it means we (hopefully) have a sound foundation to start building the software.

The goal of the initial architecture is to leave enough leeway to allow changes without having to throw everything. The more you are familiar with similar projects the more you can expect the architecture to be close to final but the opposite is also true, if you are new to a project type you can expect the architecture to be far from final.

So how do you keep the architecture relevant throughout the project? You keep yourself involved in the project, validate the architecture against the requirements and be ready to evolve the architecture if needed. You should strive to stabilize the architecture after the first few iterations -- to do that the architect should try to find the use cases/user stories that challenge the architecture or that are risky and prioritize them higher. Note however that the customer has precedence and the initial iterations might be focused on delivering some business value that doesn't challenge you architecture; that is why I recommend you have architecture iteration(s).

Posted by Arnon Rotem-Gal-Oz at 05:23 PM  Permalink |

am going to present SOA in one of our internal forums next week, so I thought it would be a good opportunity to dust-off my SOA presentation and give it a little face lift.

You can download a copy from the papers and articles section in my site (or get it directly from here).

As always, any comments are welcome.

Posted by Arnon Rotem-Gal-Oz at 07:19 AM  Permalink |

Udi had a couple of comments regarding my second post on Queues Are Databases.

The first thing Udi said was:

"I don't like the term "Saga". I prefer a term that reflects a different perspective: Long-running workflow."

I think both terms have their place. In my upcoming SOA Patterns book I talk about a "Workflodize" pattern which is a service structural pattern; i.e. you add a workflow to a service to get benefits such as Udi describes.

However, there's also a Saga pattern which is service interaction pattern. The Workflodize pattern is a good option to support Sagas within each of the involved parties.

The second point Udi made was that the messages from any of the sources invovled should have low-coupling between them. (That is, if I send message A and message B it wouldn't matter if you recieve A and then B or B and then A).

I agree that it is preferable to model your message like this, however it is:

• Not always possible or it might not make sense
• You don't always control all the services involved in the saga

So, Udi, this is what I have to say to that. Now what :)?

Posted by Arnon Rotem-Gal-Oz at 06:11 AM  Permalink |