What if everybody is wrong? Well, not everybody, but majority.

Reading Kelly Water’s Post about “Agile Project Management: Avoiding The Waterfall” I came to think about this. Almost nobody claims today that they are following waterfall, because of the very bad connotation that word now has, but not actually because they understand what are they doing. Same thing for Agile followers.

From the post, I rescue that any process that has phases reassembles a waterfall. From actual presentations, blogs and discussions, Agile often means just following methodologies like Scrum, Lean or XP. Now I question: are we right about this? Let me see if we are, or not.

Waterfall is a term, it seems nobody knows who coined it, related to a phased methodology. Dr. Winston W. Royce wrote about it (without given that name), describing an ideal form that was not so good. The phases were the requirements gathering, analysis on those requirements, system design, coding, testing and operation. The ideal was each step may iteratively cycle with the next down and the next up, in little steps. The flaw was it was usually the case that testing will not just go back to coding, but it may go up till design, and from there, jump back to requirements gathering again. That flaw (the large back jump) was caused not because the phases, but because the testing phase is where the actual product is finally validated against design and requirements.

The problem is, there is no solution to this. We can dilute or minimize the impact of the testing findings, using several techniques, but we cannot avoid it. At any level, team or individual, the same steps are repeated: I see what it is needed, I analyze what is possible, I think on how to do it, I do it and finally I check if it works. That can describe the process of developer creating a simple three line method! That is, phased development is there, everywhere, and that is not the problem. The problem is when that phased process is done in a way that the impact of facing reality with needs is too big that multiplies the work and the cost of the back jump.

So, the waterfall illusion is when you think that rain is not a waterfall, simply because the raindrops are so tiny you don’t see them fall individually. But they fall anyway. It means, people that think eliminating the analysis and design steps and jump into coding breaks the phased process, I’m sorry, it doesn’t, it may avoid a large body of water fall into one place, but will make millions of little bodies fall everywhere. See the point?

Ok, there is another idea into this, and that is to make the water flow up. You start coding, with no idea of what you need, and then the phases are completed backwards. (Hey! If you want to start really backwards, then start by testing! Oh, yes, we have TDD). The code will make a design “emerge”. The next logical steps if to create requirements that match the design. Well, I guess that does not work. In that case, the requirements are still presented before coding. So the new phases are: requirements, coding, design, testing. Or, requirements, testing, coding, design, testing, coding, design, etc. Anyway, we always have phases.

Pages: 1 · 2 · 3

Yep, my own. Although it may have some shared ideas, this is not a group agreement, nor the savior of what is now known as SOA (which is what is being done so far), nor something made for others to follow. It is just what I believe, what I follow, and what I teach each time I can.

So, here it is.

SOA MANIFESTO

1. Commons.
a. SOA is an acronym composed of the words Service Oriented Architecture, which relates to an architectural style that is based on the rules of the service metaphor.
b. A Metaphor is an application of the figure of speech concept to the IT design discipline. Using metaphors in architectural design actually refers to observing the rules of behavior of a particular entity in the business domain, and mimic them with an architectural element.
c. A Service is conceived as a cohesive (and coherent) business functionality, technologically neutral, offered through a uniform interface. A Service is a plain, homonymous metaphor of a business service.
d. For an architecture, Orientation describes the guidelines, principles and decisions that are based on the rules that govern the metaphor behavior.
e. The Architecture refers to an actual style that defines the architectural constrains, suitability and consequences of using it.

From (a) to (e) :

SOA is an architectural style whose components, constrains and principles
are driven by the service metaphor.

2. Delta
By definition, and in contra-position from some usual implementation, SOA IS NOT:
a. An Indirection Layer for interoperability.
b. A Decoration Wrap for legacy code accessibility.
c. A Component from a whole that represents one architecture instance
d. A Service Group or container of services
e. A Business Process Container.
f. A Modernization Agent
g. An Object Distribution Technology
h. An Antagonist of REST
i. A ROI salvation
j. A Way of Life, A Philosophy of Doing or a Trend (It is an architecture style!)

Corollary of (j): SOA cannot be killed, it cannot just die.
It can be suitable for a problem or not.

By definition, and in contra-position from some usual implementation, Services ARE NOT:
a. Necessarily Web Services as defined in the WSA.
b. Ways to implement RPC nor RMI.
c. Exposed Methods or Objects.
d. Defined by the process they execute
e. Necessarily Stateless (YES, they should not necessarily be Stateless).

3. Principles.
Whenever there is the opportunity when working with SOA, people should
a. Favor Business Value exposed in the business domain OVER Business Value exposed with IT domain.
b. Favor Composability OVER Integration through Interoperability, AND favor Integration through Interoperability INSTEAD of Distribution.
c. Favor Metaphor rules INSTEAD of Tool rules
d. Favor Pure Document Style INSTEAD of RPC/RMI style. (NO WRAP TRICKS ALLOWED)
e. Favor Messaging OVER Other communication options.
f. Favor Business needs OVER specific IT practices (like optimization or flexibility)
g. Favor Entity documents OVER commanding documents
h. Favor Versionable documents OVER Structurally static documents.

So it is written.

William Martinez Pomares

In a recent discussion at InfoQ about the news of the upcoming SOA Manifesto, I had discussion with great people, including Jean-Jacques Dubray and Steve Ross-Talbot both renowned guys in the SOA world. We all talk the same, maybe at different levels or on different realms. Still, I promised Steve to put my two cents about SOA. Well, here I have a couple of ideas about it, too high level, but trying to clarify at least the three words that conform the acronym.

Well, here they go.

1. Many people talk about SOA and Services, but I’ve found in presentations, articles, case studies, working with tools vendors and looking at real implementations, that not everyone understands the concepts quite right. So, I have a couple of posts (here, and here) and a short video in the Architecture Journal (SOA Myths), explaining the types of perceptions people have about the concepts, and why I think they are wrong. I assure you it is a good list, interesting to read. So, that would be my first contribution.
2. Now, let’s check on the Service concept. As I mention in the posts, to me a service is a cohesive (and coherent) business functionality offered through a uniform interface. And it is technologically neutral. What does that mean? Well, a service is a plain, homonymous metaphor of a business service, just like the loan service in a bank, or a delivery service of you mail office. As a metaphor, it is driven by the rules of common services, with contracts, processes, protocols, controls and evolutions. Its actual implementation may be whatever, but it should not surface the definition, meaning users of a service do not have to know how the service was implemented to use it, and should not have to learn anything else apart from the rules and protocols of regular business services interactions.
3. Now, Orientation. For an architecture, orientation describes the guidelines, principles and decisions that are based on the rules that govern the metaphor behavior. That is, the service one.
4. Architecture refers to an actual style that defines the architectural constrains, suitability and consequences of using it.
5. So, SOA is an architectural styles whose components, constrains and principles are driven by the service metaphor.
6. Service implementation is out of scope. Service metaphor implies business domain. The implementation domain should not surface into the business view, not affect the rules of the metaphor. This is something almost all of implementation fails to accomplish: to use a service you usually must know implementation details, and those are the ones that actually avoid flexibility and change.
7. The service contract requires a semantic agreement (standard shared business semantics are a must), functional specification, pre and post conditions and the description of side effects. All that is technologically neutral.
8. Lastly, and just for the records: to me, reuse is not a business concept, but an IT one. So, Services should not have as a goal the reusable property. In business, the property is composability. The key is to be able to create higher-level services from composing lower level ones. Reuse is a side effect, and should not be the base for ROI, because it will never make it!

Hope this is clear.

William Martinez Pomares

From a recent discussion with Duncan Cragg, related to his proposed FOREST, a get-only-rest-integration-pattern I came to think that it was good to create a little post to explain the difference between an Architecture Style and a Design Pattern. So, let’s do it.

It was Christopher Alexander, in his book A Pattern Language: Towns, Buildings, Construction. (Oxford University Press, 1977) that coined the Pattern term to denote ideas or solutions that were proven to be successful and widely used.

Well, that book was about buildings, and Alexander was an architect, of the building class. When mapped to the IT field, we have three levels at what we develop: Strategic, Tactical and Operational, or Architectural, Design, Implementation. So, we have those “ideas or solutions” in all those three levels. For Architectural level, we have Architectural Styles. For Design level, we have Design Patterns (or just plain patterns) and for Implementation level we have Idioms. You can read the definitions in POSA1 (Pattern-Oriented Software Architecture:A System of Patterns from Buschmann et al, Wiley 1996).

The style name is discussed by Roy T. Fielding in his dissertation about REST. He actually dislikes a little the name since he says it represents more a particular way of doing things (that singer’s style) rather than a general way of doing things. Coming from the art world myself, I can say it has that other meaning: a style is a way someone follows, and thus “followable”. Given someone is so original that creates a new style that others follow, does not take away that followable quality, only makes it richer.

So, as you know, the architecture of a system is the organization of its elements and their relationships, guided by principles. When defining a style, you define what types of elements would you use, what do they do, the relationships, and principles to guide their construction. A Simple example is the Client Server style. It identifies two architectural elements, the client and the server, indicates what each one does and who will the interact. Not this is a global solution: the whole system works Following this style.

BTW, Roy defines the Styles more like a set of constrains, coordinated, that define the elements and relationships. Actually, REST is defined that way in this dissertation.

When defining a Pattern, you work closer to implementation. There, you define a solution in the domain context, identifying actors and processes, relationships and results, aimed to solve a localized problem. The example here is the Factory. To solve the problem of needing different but similar objects depending on some parameter, the factory proposes the creation of the right type of object. Note this is in an object oriented context, and the solution is given in terms of objects, and it is to solve a particular problem in the whole system, not to define the whole system. Finally, note this is an idea that can be implemented in any OO language, and particularly in Java you can do it using inheritance or interfaces, meaning the pattern does not impose an specific implementation.

The Idioms are little patterns related to languages. These are proven ideas of how to accomplish things in one particular language. Say, to use StringBuffer in Java to concatenate strings.

As you can see, in general, patterns are good things, but quite different depending on the development level you are using. Now, to define a pattern, you need to provide at least five things, per Nick Rozanski and Eoin Woods, in Software Systems Architecture: Working with Stakeholders Using Viewpoints and Perspectives (Addison Wesley 2005).:

1. A name
2. A context where the problem and solution may be presented.
3. A problem to solve
4. The actual solution, given at the appropriate level
5. The consequences of applying that solution.

So, it is important to check all that before defining a new pattern, or before applying it. This last part is very important, since you may end up using a pattern that is not suitable for the context, for the problem or that has unwanted consequences.

Hope this helps clarify the concepts!

When creating a solution, much of thinking goes into the creation of functional chunks, database tables and generation of reports. Basically, the definition is made part by parts, and then it is all glued together.

Now, how do you glue it? At architectural level, it is important (and often overlooked) to define a flow strategy. What is that? Well, all the functional elements will have to execute at some time, and the execution may have at least start and one end. But before and after that what happens? How do you get to that start, and what do you after the end?

At one particular moment, the application may be doing several things, all coordinated. At the next moment some things will continue, other will stop and other will start. That “movement” is the flow, which is important. Think two applications with the exact same elements, but with different flows: they will behave differently, and even produce different things!

So, which flow strategies are there? Of course, your actual application does have one, maybe you don’t it yet but there it is, just like an undocumented architecture. Let’s review some of the strategies:

-> The Script. In this strategy, there is one script, like in the movies, where all is described: from the start, the time when one actor cries and the way another actor jumps out of the window. And there is a director, who is in charge of telling the actors what to do. Notice that the actors (architectural element, components, or even plain objects) know what to do and they do it, but the time when they must act, and the decision of who is next, is the director’s job. And this can be hierarchical, where one higher director will control other directors, which in turn control the actors.
Another important thing is the director may know all. That is, there is a continuum manager (the director or a helper). Meaning the state, persisted or not, is taken care of by the directional staff, and not by the actors.
This is perfect for concurrent processing, with highly pluggable and independent pieces of code.

-> The Production Line. If you center the flow on the processed entity, then you can have a production line. Here, the entity goes from one station to the next one, following one sequence (with forks in some parts and sub-roads). You recall the Pipes-and-filters here. The flow is actually very static, predefined. There are decision in the way too. The state in this case is managed by the actual entity, and in some cases you may need to avoid state at all.

-> Democratic Control (Chained, Distributed). This is a very common setting: the flow is in the actual executed code. That is, one piece of code not only executes its task, but also knows which other piece is called next. Of course, this means coupling and a bad separation of concerns. A change in the flow will require changes in the business code. Despite this, it is commonly used by developers (without much thinking in the strategy).
It may be use in very simple cases, with a very static flow, where the construction of a more complex flow strategy may not be worth it.

-> Workflow (or control container). Usually it uses orchestrators (I prefer to call them directors) in an engine, which will parse and interpret the flow. It seems similar to the Script, but here the engine is a generic orchestrator that executes a flow, instead of a functional element programmed with the flow itself. It is more flexible as a change to the flow may require simple adjusting the flow definition. The engine will not suffer modifications.
This is the regular workflow solutions out there, like WFF and the tools using BPEL.
The problem with this is that you require a control container, an engine to execute the flow.

-> Reactive (Event Driven). The other strategy is similar to the democratic control, as the control is associated to the functional pieces. Here, the pieces are no chained, and the control may be an aspect. Each piece is passive but listening to the environment, until something happens. Then, some pieces may decide to act, and perform something. That performance may be also detected by others, and so on. These are event driven systems, where there is an event engine and the pieces are the ones that act upon messages or events. Note the control here is in the pieces, not the engine.

So, now: which flow strategy do you use?

From many talks, blogs, case studies and reviews, I have collected a list of conceptions of what people think SOA is.

Yes, Service Oriented Architecture may mean different things for different people. I’m not talking about definitions, since those are more academic-theoretical things. Conceptions are more like the practical being of the concept.

Most conceptions differences are subtle, but enough to be important.

Let’s star by saying I conceptualize a SOA (yes, an instance of a SOA) as an architecture whose elements, in majority, represent (or are) services. SOA is an architectural style.

That said, here is the list:

SOA as an Acronym. Yes, SOA is just a name created by composing the first letter from Service Oriented Architecture. Thus, SOA is taken as yet another acronym that may be compared to other named technologies. Just another one of the bunch.

Academic SOA. In trying to explain the concept, many academics find several similarities with other concepts they had known from long time ago. Are they simple exposed objects? Are services similar to Complex Systems? What is the difference between a Service and an Intelligent Agent? Of course, there are differences. Services were never meant to be exposed objects or components (although implementation made that happen). Complex systems have evolution, self organization, chaos is involved and emergence (clearly, services are far much simpler). And of course, an agent is something that is not only autonomous, social (talk to other agents) and react and pro-act, but also have intelligence. Services do not.

SOA as an Indirection Layer. Now this one is more on the earth of implementations. SOA is seen just as a layer that receives commands or requests, and passes them down to the real working objects (that ones with the business rules). So, SOA is just a way to add indirection to the requests, encapsulating and hiding the actual how-to. Of course, this conception is wrong by a great deal. SOA is not a layer to begin with. Being an architecture, it should be an organization of elements, the majority of them representing services. As implementation, we may think of services as the layer before the actual implementation, but the service is really the complete thing: the service contract, interface and inner implementation.

SOA as a Wrap. This is very similar to the above. SOA is seen just as a decoration for already made business logic. Usually presented as a salvation for legacy code, SOA is depicted in the diagrams as a layer that wraps the old in-use code, and made it usable for the new generations. (Careful with that picture, it is not as easy to expose old functionality with new interfaces, old code may not be designed to handle new load or business flow). Of course, SOA cannot be a wrap. You can create a wrap that uses services, but then your architecture will have new elements on top, services in the middle and old code down below. That is a sandwich, not really an architecture oriented to services.

Pages: 1 · 2

Neal Ford from ThoughtWorks, recently wrote an article that is the first of a series related to architecturing and design called Evolutionary Architecture and Emergent Design. The first article is a revisit to the architecture and design concepts. In it, Neal describes the evolutionary architecture with these lines:

However, just because you can't allow architecture to emerge doesn't mean that it can't evolve. If you have created a flexible architecture and taken care not to create an irreversible decision, you can allow it to evolve over time as new concerns appear.

Let's talk about that idea...

Pages: 1 · 2 · 3

This one I owed it since a long time.

In the year of 2000, David Cavallo published his article "Emergent Design and learning environments: Building on indigenous knowledge". In this article, the term "Emergent Design" was coined, as an alternative of blueprints. Cavallo indicates that many analysts researching the situation of educational change projects failing, was actually because of the committee-generated blueprint that was created to drive the change. The result was always a reformed implementation of the blueprint that did not solve the problem.

So, Cavallo proposes another "way" of doing that process. He bases his discussion in two innovations: The Digital technology and the Administrative decentralization. Se second one does not change the management practices, but only lowers them to other levels in organization to be distributed.

Now, Cavallo proposes some principles to guide this. Constructionism (Papert, we build knowledge when we construct things), that is based on Constructivism (Piaget, we build upon what we already know). The other ones are Technological Fluency, Applied epistemological anthropology, immersive environments, critical inquiry, long-term projects and Emergent Design.

The idea was to allow children to be immersed in long period projects, so they can handle fluent technological language. With that, the students will start looking with criticism the new understood world, and using the epistemological anthropology the teacher may be able to detect the learning forces of the students freedom and design "on-the-fly" the next learning framework to explode the possibilities.

All this is handled using the emergent design approach. The learner focus on building new knowledge upon the existing knowledge, and the tools and guides are given to facilitate that. Thus, there is no preexisting path, but one that is built as it goes on. It is pure Discovering by Construction.

Now, that is a learning experience Cavallo says it is not the only one possible. It was mapped to software development by Kent Beck, and it is presented as an alternative to blueprints (that is, the dreaded BDUP).

Emergent design is built on top of a Make-Verify-Improve life cycle. Since the new world is not created yet, the developers usually give baby steps, one step at a time. Decisions are not taken at earlier stages, but a just-in-time approach is followed, and developers are the one making those decisions.

This can prove good points:

• Avoids Prediction. Decisions are taken upon knowledge acquired by constructing, when the developers know that it works, and at the time it is needed.
• Verifiable products. The created products are real, working, and thus they can be validated against the requirements. New development is done on top of this real thing and not on top of paper ideas.
• Decisions are backed up by information obtained from feedback of real products and not assumptions.

Still, there are a couple of bad points to mention:

• Refactoring vrs Redesign. Note these are two different things. Refactoring is about improving the code after it works, redesign includes changing what it is done to use another approach or structure simply because the first one does not work. So, emerging design may not be about creating and then improving, but creating and then recreating.
• Late decision at Macro level. The problem with too late decisions are that the ones that affect a level higher that code may require much more amount of rework, recreating functionality in another way. Remember, refactoring is work to improve, rework is work to make it work as it should.
• There is certainly more information (a developer immersed knows more about his local environment than a designer), but there is less higher view to make the correct decision. That is, the grass-level view Cavallo explains work for individuals creating a single knowledge, since that knowledge is ok to be incomplete (there are lots of data the learner will not know). When you have individuals working together, you need a second level view to understand how all the individuals are doing matches together.
• The validation is locally centered. When individuals validate their decisions taken upon the current local value, the isolated solution may work. When integrated, the correct parts may not add up to an integral sound structure.
• It is assumed the individual will make the correct decisions based on information, which requires a long time to acquire (Cavallo’s long process). Thus, initial development is a challenge since no individual has enough info to make certain decisions.

So, those and other pitfalls can be avoided by taking care of tactical and strategic decisions:

• Use guidelines, like principles, to driven the decisions in case of doubt.
• Create from the bones. That is, create on top of a secured and sound base that explains by itself the rationale of the solution. That may require to bring some critical decisions Up Front, always knowing those decision may be changed in the future.
• All decision has to have a rationale that supports it. The idea is that later on, when the decision proves to be wrong, the rationale may help to chose another idea.
• Multilevel Coding. I know this is something not much people understand, but actually defining the global structure, then defining each component using a tactical structure to bind the bones, and finally writing the code to add meat to them, is what I call coding at levels, since every description of the solution is code. This helps a lot to visualize when and how a decision of any level can be taken or improved.

Hope this helps to clarify a myth about Emergent Design: It is not an easy two step methodology of blindly doing and then step back to see what emerged from the disconnected decisions. And, it may not be good for all projects not for all people. Lastly, it not an alternative to architecture, but just another technique to construct the architecture itself.

William Martinez

During all these discussions, in this blog and other forums, I have found some semantic dissonance. This means some people use the same word and even the same concept, but actually understand different things from it.

So, words like architecture and design mean paper to some and process to others; they may be normal, unnecessary or even offensive. This is a little post that tries do define some of the words I use, just to be sure anybody that reads me understands what my meaning is.

Design: As a verb, is the process of decision making towards the creation of a solution. As a noun, it is the one word name of Design Description.

Design Description: The description of the decisions taken toward a solution and their rationale. It can be a document, image or even multimedia.

Code: Sentences, written in a particular language, that describe the solution. It can be at any level, and may (or not) be executable.

Strategic Design: Design that focus on architecturally significant issues, whose majority are global decisions described in an intensional way.

Tactical Design: Design that focus on tactical issues, whose majority are local decisions described in an intensional way.

Micro Design: Design that focus on operational issues, whose majority are local decisions described in an extensional way.

Domain: Is the set of concepts and relations that define a reality. It is not a description, since those concepts may be infinite or unbounded.

Abstraction: Is the result of abstracting: process of elimination of detail without loosing the essence of the concept being abstracted.

Model: Abstraction of a reality from an specific domain.

Modeling: Process of describing a possible reality in an abstraction level.

Architecture: System Structures, the externally visible properties and the relations between them. NOT PAPER.

System: Independent structure of components interrelated.

Software Intensive Systems. Systems where the majority of the components are software based ones.

Solution: Part or whole of a system that solves a problem from a domain.

Meta something: Meta means beyond, above of, self-referring. Using those concepts of the suffix, we may say meta-modeling is modeling models. Meta-data is data about the data. Meta-language is a language that allows us to describe languages. Meta-comments are comments about the comment. And so on.

Now some notes: from the above, we can say interesting things. For instance, modeling is not a paper generation process, but a description process. Modeling can be use to describe a solution, thus it can also be interpreted as coding. And code, it is not just writing executable lines (in fact, they are not executable, they need more work).

When time and other discussions require it, I will be posting more glossary terms.

William.

Working with developers, I often found them thinking of design as a generic word to indicate "sketch the classes you are going to create".

Of course, I don't share that definition. I think of design as:

"composing the sentences, in a particular design language, that describe the solution of the problem"

That said, I add that there are several design levels and types. That is, you may find out you are designing when you think you are doing something else!

Amnon H Eden and Raymond Turner had some publications where you can read about a proposal for an ontology (using the Intention/Locality). Part of their definition is that the software design should be divided into three categories: Strategic, Tactical and Implementation.

Of course, Strategic design sentences are those for Architectural design, where decisions will impact the whole. Tactical sentences are what we have always call the design, where the focus is local to one part of the problem, and that is abstract enough to be variable. Then the Implementation are sentences that are local, but also the actual execution.

That last one I call Operational design. I also call it Micro Design. That design is the one the developer does when creating the sentences that define the executable solution. Note that the decisions here are also important. Note also there are design patterns to this level too, that are different depending on then selected language (those are called Idioms).

There is another point to this definition of design: We can describe a solution at different levels, and it will be the same solution. The good point here is that each level is an abstraction of the lower level, where details was taken out of the way to clearly see broader requirements. Not all designs can be executed, and not all levels may have all the required details to be able to execute as it should.

Funny, huh?

William Martinez