IF YOU BUILD IT

... Will they Come?

September 2007

There were a few threads about whether SOA is about the technology or not.

In my opinion, SOA and architecture in general are never about the technology -- technology is important but it is just one variable in the equation. What we are looking for is a way to satisfy as many of the business needs as we can under all the constraints we face.

For instance, a few days ago I got a question in my email box from someone calling himself "coldplay". While I don't think the band has somehow got interested in event-driven architecture, the question itself looked interesting enough. Here is the situation:

Current Setup Let's take a e.g of a Inventory Stock Reorder point exception with in heterogeneous apps environment (No-SOA and integrations)...

The exception definition was built into the source apps and when the stock dropped below reorder....event registered and led to a exception. Exception was further handled by a rules based engine and a workflow
notification raised ...

Planned Setup
Same e.g as above ... Post SOA implementation.. Inventory management is composite service built by orchestrating collaborative services from SAP and Oracle...which have different data model supporting them...

The exception definition requires to be defined outside the native Oracle apps and might have to get some event related information from SAP web service also ... to arrive to a conclusion as to whether this really is
an exception or not ...

Possible Technical approaches:

• Data Persists somewhere in the processing the exception.
  
• Data doesn't persist
  
• In mem database used...
  

My question now:
1. What do you advise to be used in EDA? which would reduce network round trips, decrease apps server loads from the above 3 technical outlooks.
2. What is diff between in-mem db and usual processing of apps logic by an apps server.

I feel:

• Data persistence would lead to larger commit times and reduce operational efficiency
  
• If the data doesnt persist... and all validations are executed on the fly... don't you think the current apps servers would die processing ... or if its processing capacity is increased .. is it going to be
  economically viable alternative.
  

To be quite honest I can't really answer the questioned asked because the question lacks the business context -- what are the implications if events are missed or lost? What's the acceptable latency that would allow the business to operate properly ? If the Oracle bits and SAP bits need a lot of data from each other, then maybe the whole service partitioning is wrong and the services are not cohesive enough? How many business events are expected anyway? How often? and the list goes on.

Once you answer the business questions you can look at the available technology portfolio and ask whether you would want a in-memory database or maybe a datagrid would be a better option? and even then
the decision is not just technology driven since when you do cost/benefit analysis you need to take into account purchasing costs, operational costs, skillset of the dev team, time to implementation etc.

This is not to say that if you choose a technology that isn't aligned with your architecture you should reconsider the architecture (or technology). Also since each technology product brings into the table
its own architecture (with its own constraints and decisions) you probably need to verify the architecture once you make technology choices. but still, at the end of the day it is the business needs that sits in the driver sit, the rest is just tagging along for the ride.

Edit 09/27: in case you are wondering, I am not the only one who thinks this. for instance, I would urge you to also check out Sam Gentile's post SOA it's the business stupid and Steve Jones's SOA is a change programme, not just technology

Posted by Arnon Rotem-Gal-Oz at 04:48 PM  Permalink |

You may have read about the guy who after spending two years on a Ruby on Rails project switched back to PHP. Without getting into the debate of whether Ruby on Rails is better than PHP or whether Ruby is overhyped (probably -- but that doesn't mean it isn't any good either. By the way, it is the same with SOA, but I digress).

Reading his post, I saw a few quotes that raised a red flag for me, such as:

But at every step, it seemed our needs clashed with Rails’ references. (Like trying to turn a train into a boat. It’s do-able with a lot of glue. But it’s damn hard. And certainly makes you ask why you’re really doing this.)

and

#2 - OUR ENTIRE COMPANY’S STUFF WAS IN PHP: DON’T UNDERESTIMATE INTEGRATION

By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change...

and

Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables. I was always fighting against Rails and its migrations hiding my beloved SQL from me.

What these quotes say really is that this guy doesn't know about "Technology Mapping 101". Here is what I wrote about technology mapping* about two years ago (incidentally that's about the same time this guy started his Ruby adventure :)).

Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible.

Every technology or framework has its own architecture. This architecture poses constrains and makes certain things easy (like using the ActiveRecord pattern in Rails) and certain things harder (like not using O/R Mapping ) so, for instance, on the case mentioned above a better technology mapping might have been RBatis (iBatis for Ruby/Rails) which lets you map SQL statements to objects. It is important to note (in Rails case) that one of the core tenets for Rails is preferring convention of configuration when you don't do that you have to work hard(er) -- as you are working against the framework.

Another problem with the technology mapping here was his point #2. It is a pity he only saw it in after the fact. It can be justifiable to switch everything but you've got to allow this change to occur iteratively.

While I generally dislike the software architecture = building
architecture metaphor, using it here does make sense: building software for an existing business (vs. greenfield development) is like building a new intersection. You have got to think about building all the detours that would allow the traffic (business) to continue to operate, sure it might run slower in the intermediate phase but it can't stop altogether.

So again, Once you have an architecture that fits your business, take a look at the technology options you have. Try to choose the best fit. Whatever you choose -- take a look at the implications of that technology and think about the tradeoffs you may find that you either have to adjust your architecture or change the technology altogether -- if you don't you might find you waited two years of development effort or even more...

Posted by Arnon Rotem-Gal-Oz at 05:38 PM  Permalink |

This is another REST related post. This time I want to share a couple of observations I had after reading Roy T. Fielding's's presentation from RailsConf a couple of days ago (via Pete Lacey) and listening to Roy's presentations.

The first point has to do with a question which is sometimes raised whether you can do REST without HTTP; i.e., can you have a RESTful architecture if you don't use the http protocol, and furthermore not using the http verbs (GET/PUT/HEADER etc.) or as the unifier interface. I talked about it a while ago and I think you can. Listening to Roy's talk it seems that, at least in http architect's opinion the answer is "yes" as well.

Another point that occurred to me while watching Roy's talk, which is related to the REST magic post I wrote a little over a week ago. The use of a uniform interface is taunted by REST proponents (and Roy himself) as coupling reducing formula. After all if you use a uniform interface you are not coupled to the particular semantics of any resource/service you already know the capabilities (actually the maximal capabilities) they offer. What ensues is that instead of using a lot of verbs (ReserveRoom, UpdateOrder etc.) you use a lot of nouns (/rooms/, /orders/order1 etc.)

This works extremely well on the "human" web where my browser can navigate to any-ol'-site without any prior knowledge of what's the site about. When I navigate to Amazon I can buy stuff. When I navigate to New York Times I can read stuff, etc. The problem here is the browser is really dumb about what's going on. I, as a human using the browser, understand the context from the content (well, most of the time anyway;)) so the browser can remain decoupled. However when you translate it to the "programmable" web you usually don't have some mighty AI engine examining the response to understand the context. Instead what you do is trade the verb coupling, which with WS-* web-services would be defined in a contract, you are now coupled to the nouns (this is not to say the nouns aren't discoverable -- since they are due to the hypertext or document orientation communication REST encourages). The end result is pretty similar to what you get when you use verb-based contracts -- your software still needs to understand (where "understand" means some level of coupling) what it is doing with the "other" services. Not to mention that you still need to understand the content of the message (sorry, "response") to do anything useful with it.

In any event, while loose coupling is very desirable, we also need to remember that the only way to truly achieve complete decoupling is to not connect components. So some coupling is always needed if we want to produce meaningful systems.

What do you think?

Posted by Arnon Rotem-Gal-Oz at 05:42 PM  Permalink |

If there's one reason to go to ApacheCon 07 in Atlanta, then it's probably Roy T. Fielding's "A Little REST and Relaxation". Here is the abstract:

Representational State Transfer (REST) is an architectural style that I developed while improving the core Web protocols (URI, HTTP, and HTML) and leading them through the IETF standardization process. I later described REST as the primary example in my dissertation. Since then, REST has been used (and sometimes abused) by many people throughout the world as a source of guidance for Web application design. But is the REST that we hear about today the same as what I defined in my dissertation, or has it taken on the baggage that comes with an industry buzzword? This talk will provide a real introduction to REST and the design goals behind its evolution as the Web's arhitectural style. This is not about XML-over-HTTP as an alternative to SOAP, nor about "resource-oriented" frameworks that help simplify CRUD operations, but rather about the design goals and trade-offs that influence the development of network-based applications. I will also describe what happens when we relax some of the REST constraints, and how such relaxation is impacting the design of the waka protocol as a replacement for HTTP.

Now all I have to do is find an excuse for my boss... :)

There isn't a whole lot of information available  on WAKA (that replacement for HTTP Roy mentions in the end of the abstract). Below are a few links I managed to find:

• Roy's 2002 presentation on REST and WAKA (ppt).
• A blog post from Feb. 2007 by Robert O'Brien that talks a little about the new semantics offered by WAKA (such as MONITOR to get notifications if the resource changes)
• A podcast by Jon Udell interviewing Roy T. Fielding (from Aug 2006)
• A blog post by Mark Nottingham briefly contrasting WS-Transfer and WAKA (March 2006)

And there's a few others but not as interesting (to me anyway). Well, as we see this WAKA thing is in the works for a long time now. Also replacing something as ubiquiteus as HTTP is not a small feat. But I guess if anyone can pull this off it would be Roy... As always, only time will tell.

Posted by Arnon Rotem-Gal-Oz at 08:55 AM  Permalink |

From time to time, I read about the magic that is RESTful services and how they solve everything and anything, such as scalability, idempotency, simplicity, and the like. For instance, in RESTful Web Services Sam Ruby and Leonard Richardson say that "PUT and DELETE operations are idempotent. If you DELETE a resource, it's gone. If you DELETE it again, it's still gone..." (p.103) and "the safe methods, GET and HEAD, are automatically idempotent as well" (p.219).

Another example comes from Anne Thomas Manes who says:

The REST architectural style defines a number of basic rules (constraints), and if you adhere to these rules, your applications will exhibit a number of desirable characteristics, such as simplicity, scalability, performance, evolvability, visibility, portability, and reliability.

The basic rules are:

• Everything that's interesting is named via a URI and becomes an addressable resource
• Every resource exposes a uniform interface (e.g., GET, PUT, POST, DELETE)
• You interact with the resource by exchanging representations of the resource's state using the standard methods in the uniform interface

I think such claims are plainly wrong and misleading. Don't get me wrong, I like the REST approach, since it encourages better service design -- e.g. document oriented message exchange vs. the RPC-like message exchange which the so called "WS-death-*s" (or actually the tools that support them) encourages.

It also encourages the above mentioned traits -- however that's exactly the point -- REST encourages this thinking not solves scalability or other problems out of the box. You still need to design your services properly.

For instance, if you follow Anne's rules you can still end up with a service which is stateful, that performs heavy distributed transactions against multiple databases and systems -- i.e. a service that is neither simple, scalable or perfromant.

DELETE will only be idempotent if the resource is idempotent (e.g. a specific version of a resource) or the message is idempotent (e.g. requesting a deletion of a specific version) if you are deleting the "recent version" then it might have been recreated between your calls you are now deleting something completely different. heck, even a GET (read) message with a single reader can be made to be non-idempotent  if you decide to code something that alters the state of a resource significantly whenever it is read. When you have multiple readers and writers GET will not be idempotent "automatically" as two consecutive reads can give you two different representations as the resource might have changed (again unless the resources are idempotent)

REST is not different from other styles in this respect; for instance you can do object orientation in C but working in an OO language encourages object orientation (the opposite is also true; using an object-oriented language does not guarantee that you get an object-oriented design).

At the end of the day, architects should still think about the design if they want to ensure the results matches the quality attributes they want to achieve. Some environments/styles/tools will make some quality attributes more easy to achieve but nothing will solve the problems for you.

Posted by Arnon Rotem-Gal-Oz at 12:58 PM  Permalink |

In a previous post, I wrote that the RDBMS is dead. I didn't mean that it is dead dead, but rather that it isn't built to meet newer challenges like linear scalability, high availability, etc.

Well, it is one thing hearing it from me -- and it is another thing hearing it from someone like Michael Stonebraker.

Michael, you recall, was the main architect for the Ingres prototype project at UC Berkely just one year after Codd's paper, 9 years before Oracle was released, and more than a decade before the commercial version of Ingress was released.

Well that was in 1970. In 2007 Michael recently wrote:

In short, the world of 2007 is radically different from the world of the late 1970s. However, none of the major vendors have performed a complete redesign to deal with this changed landscape. As such they should be considered legacy technology, more than a quarter of century in age and "long in the tooth".

Among the new needs Michael cites are intelligence DBMSs (needs a lot of relations), textual, and semi-structural data, etc. He also said (promoting his own product) that 2007 customers expect high availability and linear scalability.

Michael's main point is that specialization can provide significant performance enhancements vs. the one-size-fits-all approach of RDBMSs. He gives his product (Vertica) as an example for how a column-oriented database (vs. the RDBMS row orientation) can outperform RDBMs by a factor of 50. Google's Big table is another example.

Interesting....

Posted by Arnon Rotem-Gal-Oz at 03:50 PM  Permalink |

Jeremy Miller recently tried to "answer hard questions about Agile development". One question he didn't really answer was that of agile and fixed bids:

I don't really think the Agile answer to a fixed bid is any different from any other process. I do think that Agile practices and project management can give you far more control and feedback on the "Iron Triangle" of resources, time, and features.

Jeremy says that this would enable the team to fail "softly" (i.e, with some functionality etc.).

I think -- well, actually, I know since I did that -- we can do better than just use Agile and hope for the best. For instance, here are a few strategies I've successfully used:

• Pre-budget the whole project with rough milestones and estimates (works best if you have prior experience in the domain).
• Coordinate expectations with the client and trade the single project for many smaller ones. What I did was to get a blanket purchase order for the estimated amount and each smaller order/project "ate" some of that budget until it was done.
• Since the project was for a CMMI level 3/ ISO 90003 we also added a few documents to the backlog --and let the product owner prioritize them vs. deliverables. For instance, the "Architecture document" was delivered after 6 iterations when the architecture stabilized.
• exchange requests instead of change requests. New functionality comes instead of old promised one.

Of course, we also did the other SCRUM practices of releasing frequently, demonstrating progress, retrospectives, etc.

I would love to take credit for the ideas above. However, Pascal Van Cauwenberghe details most of them (and more) in Agile Fixed Price Projects Part 1: The Price is Right and Agile Fixed Price Projects Part 2: Do You Want Agility With That. Go read them.

Posted by Arnon Rotem-Gal-Oz at 07:29 PM  Permalink |