Hi William,

I sort of agree. But I don't think that true design fits neatly into different stratifications and development phases. Expert designers/programmers like Kent Beck do everything all at once in small iterative cycles. TDD builds on this idea for micro design like you say, but there are other XP practices that do the same for macro design too.

The result is emergent design, where both micro and macro design decisions are deferred to the last possible moment and the whole design evolves over time in response to concrete feedback and in keeping with the principle of the simplest thing that can possibly work

Emergent design cannot occur if we are modelling obsessed, creating large unverifiable paper models up front. The only model we can verify is the code. Steve Yegge as a good post on this subject which I tend to agree with. It is called Portrait of a Noob.

Paul.

Hello Paul.
I just missed that post. It is a very rich post to comment on!
Buy I think we are talking about different things.

First, let’s put the modeling issue aside for a moment. The intention is to focus on the actual point where we both agree: there are design levels.

The micro design occurs when decisions are taken at coding level, when the problems you must solve are in front of you and very localized. There, TDD will help to think in cohesion and interface, and let you worry about actual implementation later on.
The macro design refers to other decisions, higher level. "When" you make those decisions is not the point, but how. My post (and Cedric’s) is TDD may not be suitable for that macro decision taking, since it is too localized. It may help you decide, but it may not be the best decision.

An analogy: if you want to design the traffic in the city, you may want to drive your car through all the streets and then decide which ways each must implement, or fly an helicopter and see all at once, bird’s eye. I would use the first one to decide turns and visibility, but the second one to decide entrances and exits to the city. Micro localized and Macro Non-local.

So, I’m talking of the difference in the design levels, and how to accomplish each. I’m not talking of "when" nor about paper nor about modeling. I am a defender of using the correct tool for the task, and thus using the correct language to describe things. Kent Beck may have methodologies to describe all three levels with only one language, and take decisions for all three levels from the street view, just in time. He may iterate adding functionality, building the onion from inside out. Some others iterate de-abstracting, building the onion from the outside-in. I grant there are others that draw the salad before even having the land to bury the onions seeds, and maybe those are the ones you refer as to “modeling obsessed individuals that create large unverifiable paper models up front”. I don’t like them either. Trust me.

So, I will write my post of noobs, that will be fun.

William.

"An analogy: if you want to design the traffic in the city, you may want to drive your car through all the streets and then decide which ways each must implement, or fly an helicopter and see all at once, bird’s eye. I would use the first one to decide turns and visibility, but the second one to decide entrances and exits to the city. Micro localized and Macro Non-local."


Hi William,

Agreed!

Emergent Design addresses both. The difference between Emergent Design and Big Upfront Design is that design decisions are deferred to the last responsible moment. I'm sure you've experienced getting to the end of a project and thinking if I knew at the beginning what I know now, I would have done things differently.

Emergent design addresses this problem, allowing you to evolve your design as you go along. I believe validating your design decisions with concrete feedback is a favorable design approach at all levels of abstraction (architecture, macro, micro). TDD is just one of a number of emergent design techniques. Although TDD is focused on micro design, it can also influence macro design and architecture too. Small iterative design cycles with feedback. I had a much longer comment describing how emergent design at all levels of abstraction works. Perhaps I'll turn it into a post on my own blog. It would be interesting to hear your views.


BTW. We tend to agree on most things. The learning opportunity for me is where we disagree. It will be interesting to hear an Architects take on Emergent Design.

Paul.

Sure!

I will write about Emergent Design. I just heard an interview with Michael Stal about Architecture Refactoring, and I want to post about it. And this is also related to Emergent Design's concerns for an Architect.

So, I owe you three posts: The NOOB one, the Emergent Design one and the Architecture refactoring one. Plus, one of Multi Domain design I have started.

I will try to work on this soon, for today I have a class to teach at university and so work to get paid meanwhile :D

Will.

Hi William,

Whoops! I didn't want to create work. Never mind competing with the day job :)

I look forward to what you have to say, but take your time.

Take care,

Paul.

In behalf of Paul Beckford.
Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.


How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.