IF YOU BUILD IT

... Will they Come?

January 2007

You may have read my BI and SOA post where I suggested using EDA within SOA to solve the BI/SOA impedance mismatch. Jack Van Hoof made the following comment on that post:

Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0. Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other….

I think that "Synchronous RPC" is not a very good (or useful) definition for SOA (see my series on what is SOA anyway). Nevertheless, I also think that even if all you have is synchronous request/reply you can still implement both asynchronous messaging and EDA How can we implement Asynchronous Messaging?

Option 1 Duplex Channel
Let’s say you are a service consumer. You send me your request. Instead of a reply I just acknowledge you that I got the message. I put the message into a queue and process it on my "spare" time. I then call you with the answer.

Option 2 One way Channel
Again you send the request. Instead of a reply, I give you a token or a ticket for the answer. When you think it is time, for example when the time promised in the SLA elapse (or whenever), you call me again, give me the ticket, and I look up the answer and give you your reply. If we hide all this protocol inside some software infrastructure the applications can see asynchronous messaging even though we have synchronous request/reply on the lower levels.

Okay, so what about Events? How can we publish events just using request/reply. The previous example would not work since we can miss out on important events?

If you are reading this blog -- chances are you already have the answer working on your computer -- yes, it is RSS. Think about it using an RSS Reader that pulls the server that publishes this blog you reach out using synchronous request/reply and get all the posts (events) that were added since the last time you asked.

There are a few additional architectural benefits for working this way. For one the service does not have to manage subscribers. Secondly, the consumer doesn’t have to be there the moment the event occurred to be able to consume it -- and the management and set up is easier and simpler than using queuing engines

Posted by Arnon Rotem-Gal-Oz at 06:44 AM  Permalink |

I am sure many of you have seen this happen before: You are working on v0.9 version of project, making nice progress. Then out of nowhere, out steps this marketing whizz-kid or wonder salesperson who meets a client and tells him all about the wonderful features of the product (you were planning for v3.5), signs a deal, and leaves you pick up the pieces and somehow try make all that work. Hype is like the power of thousand such whizz marketers working against all of us -- making it almost impossible to live up to these "common truths" which are actually hollow promises. Let's just take a quick look at few of the more common hype-originated myths surrounding SOA.

The claims that SOA will increase reuse and/or will make it easy are common -- so much so that it has been adopted by serious technical people. For example, the article SOA: Separating Myth From Reality by Mark Potts (who is a CTO in one of HP's divisions) says:

One of the major benefits of the set of architectural principles that define an SOA drives organizations towards a greater level of reuse and consistency. The amount of achievable reuse increases over time once newer services are designed using existing services in an organization....

There is some substance to the reuse claim, if we take a broader definition of the term "reuse." As an organization moves to SOA, it can make the transition an evolutionary process -- and still use existing assets in the transition phase. This evolutionary process approach contradicts another SOA hype generated myth which states that SOA requires an all or nothing approach.

Consultants and technology vendors would have you believe that you either have to fully embrace SOA or it just won't work. I don't think so. I agree with Sohel Aziz that it is actually the other way around -- if you want to succeed with SOA you need a pragmatic step-by-step approach.

When you think about it, a Full Enterprise SOA initiative would mean replacing all the enterprise's systems. Do you really think the business will just freeze or halt while the SOA initiative is underway? Not quite. Rather, a comprehensive SOA initiative is like building a new interchange on a highway. You need to provide ways for the traffic to continue to flow (business to continue to operate) while work is underway. More so, you will make as much progress as you can without disturbing traffic.

Also SOA characteristics (which I'll expand more about later) can even allow you to take an incremental way for removing legacy systems by exposing existing functionality using new SOA interfaces before you actually replace the underlying system.

Another SOA myth is that it will make integration easy. SOA puts a lot of emphasis on the interface , which makes it a "easy" to just make services talk to each other. On top of that SOA is (usually) based on standard protocols like XML, WS*, http and the like, so it is easy to just tie services together and integrate them.

Fred Brooks told us ages ago:

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

SOA does not change these basic rules. To properly integrate services you need to model their contracts in a way that will make them usable for multiple business processes -- this is not a small feat and that is where you'd spend most of the time -- this is still hard and SOA does not solve it. Once you solve that, SOA does help you lift some of the burden of doing the actual integration - but that's just making integration easier and not making it easy.

Yep, it seems SOA is indeed in dire straits. In fact, I've almost managed to convinces myself that Fowler is right and SOA is beyond saving. Luckily it does seem that over the past year or two businesses are starting to actually build SOAs and we are starting to see some more consensus on what SOA is and isn't. It seems that today there are two prevalent ways to look at SOA -- from a business point of view and from a technical one.

Posted by Arnon Rotem-Gal-Oz at 02:08 PM  Permalink |

Service-Oriented Architecture (or SOA) has been with us for quite a while.

Yefim V. Natiz, a Gartner analyst, first Service-Oriented Ambiguity. Martin says that the question "what is SOA?" is impossible to answer because SOA means different things to different people. While it is true that the term has been overly used, at least few of these "definitions" are plainly wrong. Let's take for example the definition that says SOA is exposing methods through Web services.

A common misconception about SOA is that using web-services technology makes whatever you are using SOA. The core reason for that is the poor naming choice for methods that are exposed through http which were named web-services. For example, one of the popular books on SOA "Service-Oriented Technologies: A Field Guide to Integrating XML and Web-Services" by Thomas Earl gives the impression that SOA equals WS* (even though that buried in the book, there's an explanation hat WS* is not the only possible technology for SOA). There are many other sources that give the same impression.

Nevertheless, Just a bunch of web-services (JBOWS as Joe McKinderik named them) does not an SOA make. In fact that's not really making anything new. Using http and WS* protocols to expose a method isn't really different from Remote Procedure Calls (RPC) using any other technology be that CORBA, DCOM or anything else. The way I see it saying that SOA is using Web services is just plainly wrong and not away to define SOA.

Another definition Martin mentions is that another widely used definition for SOA is "EAI without all the expensive EAI vendors locking you in". EAI emerged as an attempt to solve the Enterprise integration spaghetti. While it managed to solve the problem of connecting applications and transferring data between them through a semi-standard way. I agree with Sandeep Arora who said that basically EAI failed to deliver its promise as it is.

• Ddata centric and not process centric.
• Can't keep up with business process change.
• Does not address the business process.
• And that EAI solutions are technically very complex, need specialized skills and are very expensive to maintain.

Whether Arora is right or even if EAI is the best thing since sliced bread, if we do EAI using Web-service technology to get "EAI without the expensive EAI vendors locking you in" (which I also have een called SOI, or Service-Oriented Integration) we are again not doing anything new, we are just using a new technology to achieve an existing way of thinking or architectural appraoch.

Martin claims that the SOA acronym is beyond saving and that the (sometimes) good ideas that are all called SOA need to have their own name and independent life. However the way I see it the examples Martin provides are "just" misuses of the term. I think that the fact that someone uses a term wrongly does not invalidate the correct term. Nevertheless, SOA troubles do not end here, since there is the hype also lures behind the corner to dilute the SOA term as well. I'll talk about that in the next installment.

Posted by Arnon Rotem-Gal-Oz at 04:01 PM  Permalink |

I received a few questions from readers regarding my BI and SOA discussion.

My approach for the visuals and how the presentation unfolds is based on a book by Cliff Atkinson entitled Beyond Bullet Points. You can also see Cliff's blog.

In a nutshell, the approach is to get rid of the boring text laden and bullet points ridden slides and exchange that with a story and enahnce the story with supporting visuals. I am not completly there myself -- but I can tell you that my presentations already have a different look-and-feel.

Posted by Arnon Rotem-Gal-Oz at 05:03 PM  Permalink |

We are currently running the first sprint for our next-gen platform., Since I like to open my projects with a few architecture iterations, or exploration iterations, we are now identifying architectural requirements and core components -- and making a few fundamental decisions.

In this particular project, it is relatively easy since this is a "next-gen" which means there was also a previous generation. Thus we have a better idea on what to expect from the platform. One type of decision we have face involves buy vs. make. Actually this should be called "buy vs. adapt vs. build from scratch" since we can also choose to extend existing assets. However, let's focus on the buy side for this post.

There are two distinct decisions here:

• One is to identify the components which you will want to buy instead of develop.
• The second is to decide what is it that you want to buy.

I divide this into two decisions because you can separate them in time. For instance, you can make a decision early on to use a database rather than develop a persistent solution of your own. (In most IT projects that's an obvious choice, but in other project this is not always true.) However, you may be able to postpone the decision on what RDBMS you choose to a later time and just hide the database behind an O/R mapper or a data access layer. The decision to buy is important though, since otherwise you can find yourself spending development hours and days on implementing persistence functionality which you would throw away later.

To give you two examples from the project mentioned above: We need a secure database, we are storing biometric data and privacy is very important as well as preventing tampering. We definitely want to buy something here and we're debating several options. The other example is the computation heavy parts of the application where we think we may need to add Grid computing to cope with extreme scenarios. However, we have many other requirements that are ranked higher in the product backlog, so we just identified the block, decided on the approach we'll take if we'll get there but we aren't doing anything else about it for now.

We will proably won't even add an empty block just now. We may just identify the point were we will refactor it in later. But the fact that we know it can be needed there means our porcesses will allow accomodating it later and it won't be too difficult to weave it in if it will be needed. Granted, that's not exactly YAGNI, but it isn't exactly big design either -- it is something in between. I think it is just enough

Posted by Arnon Rotem-Gal-Oz at 04:43 PM  Permalink |

I got email from the Software Engineering Institute (SEI) on its third SEI Software Architecture Technology User Network (SATURN) Workshop which takes places May 14-16 in Pittsburgh.

The SEI is still accepting submissions (until January 15, anyway) on the following topics:

• Case studies and lessons learned in applying SEI software architecture technology
Efforts to improve the state of architectural practices across an organization
• Clarifying and codifying the role of the software architect
• Future directions in software architecture technology
• Architecture design in distributed software projects
• The relationship of software and system architecture
• The value or ROI of software architecture practices

If you are interested in participating, submit your proposal at http://www.sei.cmu.edu/architecture/saturn/submissions.html

Posted by Arnon Rotem-Gal-Oz at 04:24 PM  Permalink |

It seems the topic of BI and SOA is more important than I thought, at least according to a recent Ventana Research survey on SOA and BI. You can see the results here).

All in all, the situation looks rather grim. To quote the opening of the article:

...only one-third of respondents reported they believe their internal IT personnel have the knowledge and skills to implement BI services.

If you recall, last week I talked both about the problems BI poses for SOA and suggested an approach that might help. I didn't know about the Ventana survey at the time, but it turns out this is really something organizations are struggling with. As another small effort in helping increase the understanding of BI implementation in SOA, I've made posted a presentation I prepared that also explains the BI/SOA conflict and its resolution. You can download the presentation from the papers section on my site or directly from here. (The download is about 3MB.)

By the way, most of the information is in the notes (the slides mainly contains visuals).

Posted by Arnon Rotem-Gal-Oz at 04:23 PM  Permalink |

The Open Group (the TOGAF guys) has scheduled what appears to be an interesting conference on Enterprise Architecture and SOA in San Diego at the end of January 2007. Looking at the program, it seems to be right on target. Unfortunately for me, I will not be able to attend -- I am swamped (you know, new year, new projects). Maybe next year.

Anyway, the Open Group also plans on officially launching its A\architect’s association -- the Association of Open Group Enterprise Architects, which is also good news. There are too few active architect’s associations out there and I hope this one fares well, in much the same way as IASA seems to be (and not turn out like WWISA ).

Posted by Arnon Rotem-Gal-Oz at 07:51 AM  Permalink |