Hi Will,

I think you slightly misunderstood Steve. He was talking about how much he originally relied on meta-data in his youth versus allowing the code to speak for itself which is what he tends to do now. He identifies this as a trait of a noob. I agree.


This doesn't make meta data good or bad. But it does identify a skills gap. We should all be able to listen to the code and let it speak as the one true model. The problem with relying on meta-data models is that they can become out of sync with the computational model and reality and hence can become meaningless and even misleading. They can also unduly constrain the design process.

In my first year of University I didn't have a television, so I was forced to experience the world first hand going out and talking to people in pubs and clubs. When I went home and started watching TV again, I didn't recognise the TV world as my world. This meta-world bared little resemblance to my reality and I could see that anyone who relied on this medium as their sole input on society would end up with a skewed view which was out of touch with reality.

How many square box and straight line 10 thousand feet paper models have you seen, where in reality the real model (the actual code) is a big ball of spaghetti?

This modeling obsession and detachment from reality has a number of side effects. See my latest blog post for a concrete example around N-tier architectures and SOA.

Paul.

Hi Will,

I didn't address your point about compilers, byte code VMs etc. The way to look at these things are that they are part of the machine. Humans need to tell the machine what to do. That is the definition of a program, a list of instructions from a human for the machine. The machine then translates the program and executes it. The translation phase is an implementation detail (especially if you are using an interpreter), so a more generic description is that the machine executes the program. So as far as us humans are concerned the code is the program. And the program is an executable model.

This is the main difference between meta-models and code models. A meta model is a model that tells you something about another model. If our meta models were executable then they too would be code, and their translation part of the machine. Languages like this do exist. Smalltalk and Ruby have a lot of meta facilities built in. The difference is that these facilities are not redundant. Say like type annotations are. These meta facilities allow these languages to be self describing, and hence programmer extendable, leading to DSLs.

Back to the main point. We need to know that our models are correct, and the only way to do this is to have executable models. An executable model is a program, and the language used to describe that model is a programming language, by definition.

My main point is about emergent design and feedback. Executing your model is a way of gaining feedback on the correctness of your model. I am saying that models that aren't testable are of less value then models that are.

The last point here is about abstraction. You could argue that programming languages do not provide an high enough level of abstraction to model certain concepts like modules, layers, tiers etc. For some languages this is true, and you need to augment the program with meta-data to describe such things. Other languages though like the Beta Language do allow you to declare abstractions that are larger than a single class. For these programming languages meta data as a way of declaring high level abstractions is not needed.

Paul.

Hi Paul.

I think you slightly misunderstood Steve. He was talking about how much he originally relied on meta-data in his youth versus allowing the code to speak for itself which is what he tends to do now. He identifies this as a trait of a noob. I agree

I guess I did misunderstood. The part of what he did in his youth I got it, and I tend to agree with his perception of "talking to ourselves" when writing comments. I think what we seem to understand different from him is the metadata definition.

Let's see from his words in the conclusion:

"Metadata is any kind of description or model of something else. The comments in your code are just a a natural-language description of the computation. What makes metadata meta-data is that it's not strictly necessary"

He explains more in the conclusion...

"Software engineering is hard to get right. One person's pretty data model looks like metadata-addiction to another person.

I think we can learn some lessons from code-commenting: don't try to model everything! You need to step back and let the code speak for itself.

For instance, as just one random illustrative example, you might need to return 2 values from a function in Java (a language with no direct support for multiple return values). Should you model it as a MyFunctionCallResult class with named ValueOne and ValueTwo fields (presumably with actual names appropriate to the problem at hand)? Or should you just return a 2-element array (possibly of mixed types) and have the caller unpack it?

I think the general answer to this is: when in doubt, don't model it. Just get the code written, make forward progress. Don't let yourself get bogged down with the details of modeling a helper class that you're creating for documentation purposes."

So, we need to agree in something to get forward: When does modeling happens? From Paul I get it happens before the code, and modeling is just paper, non-executable work. For Steve and the above lines, modeling can happen during the coding, and creating an object representation is modeling and since is it not needed, ergo it is meta-data!

Actually, Paul, you differentiate both by introducing the Meta-Model concept. That is the paper model I mention. So, for you, modeling in the language is ok. I understand that, for Steve, it is of no use unless strictly necessary since ultimately it is just documentation.

There is where I jump in, and say that Meta-something is not totally useless. It may not improve the code execution, but it will improve the coding process. Now, my own portrait of a NOOB is that one that actually comments everything but thinks there is only the programmer and the computer. Part 2 and 3 will refers to those things. In 2 I will address the globalization issue. And in 3 I think to address the issue you point about paper models out-of-synch with reality (Actually, there are two realities there, the code done and the problem).

Finally, my definition of coding is higher that just creating execution instructions for machines. But of course, it is totally different than what the guys are doing right now. Many modelers do the paper work because it is part of the discipline, but in their hearts those boxes mean nothing. That is useless waste of time. But for me the solution is not removing that step, but transform it into something useful.

I hope next post will clarify a little bit.

William

Hi William,

I believe Steve and I are saying the same thing. The code is the model. All other models are projections of the code. The meta-data Steve speaks of is data that tells you about the code and is hence redundant. Like comments that tell you about the code. UML that tells you about the code is redundant too. For this type of meta-data I introduced the term meta-model, since most people think of UML meta-data as a model, but like Steve says such a model is redundant too.

BTW. I also introduced the distinction between meta-programming and meta-deta. For me meta-programming tells you about the runtime computational model, and is not redundant. Like listing the methods in a class. This is not redundant since the number of methods can change dynamically. So meta-programming is not a redundant language feature, but a self descriptive feature providing information that you would not know otherwise.

Finally, modelling can occur at anytime. My focus is on the executable model, this is commonly refered to as code, but it doesn't have to be code in the tradditional sense. It just needs to be executable. For example the Self programming language is graphical and allows you to program visually using drag and drop, but the model is directly executed by the machine (Self VM) with no manual translation step, so Self is a programming language although the "code" is graphical.

I hope this clears things up.



Paul.

Humm.
Now I'm sure I don't understand.

1. I read your post and also Steve's and I find they are different. Subtle, you are using the same words but meaning different things.
2. Model is an abstraction of reality. Code can be used to model a reality, but that does not make any code a model. Very philosophical.
3. The main difference of your definition of "Code" and my definition of "Coding", is that mine is not executable directly by a machine, and that is Part 3 I mentioned: There is not always just the machine and the programmer to solve the world.
4. When we talk about modeling, I mean something totally different from making boxes in UML, Steve talks about creating classes as abstractions of simple grouped primitives, and you are talking about meta-modeling. To me, meta-modeling is when you model the domain definition for a model.

My head hurts. I think we need a glossary somewhere.

William.

Hi Will,

Just another point. Steve also identifies meta-data imposed by the compiler. This can be for a number of reasons such as aiding the compiler in optimising, or as a programming crutch, flagging potential type errors before runtime. This meta-data creates a programming model which is artificially constrained and some what different from the underlying computational model of the hardware. These types of languages lead to code models that are constrained and lack certain features.

Steve makes the point that such compiler imposed meta-data is not only redundant but limiting. I agree with him, but I am making a different point. Your programming model although constrained is still executable, the difference is the machine upon which it executes (including the compiler) is a somewhat limited one. You can still get concrete feedback on the correctness of your model though, so you can still perform emergent design. With a paper model there is no such feedback, and without concrete feedback emergent design is not possible.

Paul.

"My head hurts. I think we need a glossary somewhere."

I agree. I think the problem stems from the language we use which is "borrowed" from other professions and not well suited to what we actually do, which is produce working software.


In other professions they separate the design activity from construction. In computer science we try to do the same, but unfortunately we made the wrong distinctions and hence the headache :)

Programmers do not do construction. With computers construction is performed by the machine (Compiler, VM etc) which translates a design into machine code, placing it in memory ready to be executed by the CPU. Programmers do design. The output of the design phase is an executable model commonly known as code.

This is where the whole language gets it wrong. Programmers do architecture too, and analysis. In fact it is all there in the program. It may be difficult to see, which is why other representations are useful, but let us not forget that these other representations are just mere projections filtered in specific ways to focus on a given level of design abstraction.

If you create such a projection on paper before producing a concrete design (program), then you have no way of knowing that it is correct and whether the machine will construct what you expect. You do not know whether it will work once constructed.

Paul.

"2. Model is an abstraction of reality. Code can be used to model a reality, but that does not make any code a model. Very philosophical."


I think this is where the philosophical difference lies. I believe that all code is a model. The issue is whether it is an good model or a poor one.

I will replace the word "model" with the word "blue-print". All code is a blue-print for something we wish the machine to construct.

If you look at things this way then there is just the program and the machine. Nothing else.

I agree with you that the creation of the program is a collaborative effort that involves more people then just the programmer. But if you accept that the program is the final "blue-print" for the software, then it follows that the programmer is at the sharp end.

Paul.

That's fair.
I will chew a little bit that. It seems that there is enough data for several posts!

I need to win the lottery to have time to write more... :>>

Will

"I think the general point is when in doubt don't model it"


I see what you mean Steve and I are saying subtlety different things. I have not said this, but again I agree with the sentiment. What I think Steve is saying is don't "over elaborate" your code model for documentation purposes. I agree. Steve is focusing on what he calls the computational model which actually runs on the CPU at runtime. I am focusing more on the programming model. The actual code.

Steve's computational model is even more fundamental then the programing model, so he is taking the back to first principles idea even further. I think the sentiment behind what he is saying is keep your design as simple as possible and don't invent unnecessary abstractions that the computational model doesn't need.


The counter argument is that the people reading the code need these abstractions. To which I would argue that for skilled practitioners they don't. And there is a cost to over elaborate models. Over elaborate models increase the size of the code base, increase the time to market and increase the cost of maintenance.

Look at the concrete examples on my blog with regard to ORM, SOA and N-tier architectures where I try to make the cost clear.

I am arguing for coaching programmers so that they have the pre-requisite skills rather than relying on elaborate models, whether on paper or in code.

Unskilled programmers will not translate even well thought out paper models into code with any level of fidelity anyway. I am currently coaching a team where the architect has produced a decent OO domain model in UML, leaving the programmers to fill in the details. The unknowing programmers have gone on to treat his domain objects as value objects and have produced code in a procedural style. So the architects intent has not been reflected in the code.

This is yet another example of why trying to split up the programming role, in an attempt to dumb it down to just "coding", doesn't work.

Paul

Yes. I see what you mean.

One ideal world will contain only seasoned programmers. But we have to deal with juniors due to cost.

In the example you mention, the architect may not need to create an UML design (his job should be a higher view than that). But in the case the architect is also the designer and the technical lead, the he may be defining the principles and strategic decisions (he has to decide into using web services or CORBA), then the tactical decisions (define main tables and attributes, define patterns to use and main class interactions) and then hand off to a senior developer or such (which will take the challenges and leave the more trivial code to less experienced guys). As a technical lead, he needs to check the code that is produced does not go against the decisions taken in the tactical level (unless they have a very good reason).

To put this simple: I won't let a junior decide if using WS or REST. It will be my job. And my boss may not want me coding a protocol, or a DAO. And I do need a technical lead that validates all that is produced.

But, I may code a POC (to have a particulay point the way I wanted) and the designer to actually code the skeletons of classes and critical interactions. So, nobody escapes coding, but everyone does it at the level it will cause less disaster.

William

"One ideal world will contain only seasoned programmers. But we have to deal with juniors due to cost."

Seasoned and skilled are two different things. There are plenty of "seasoned" programmers who lack basic skills such as OO design. The issue is mentoring/coaching and placing value in people. In my experience hiring the most skilled programmers you can find is the cheapest approach. A crack team of 5 highly skilled programmers will beat a team of 25 mediocre ones hands down, at a fraction of the cost.

"To put this simple: I won't let a junior decide if using WS or REST. It will be my job. And my boss may not want me coding a protocol, or a DAO. And I do need a technical lead that validates all that is produced."

Your job isn't as easy as you may think. How do you know that you have made the right choice? For me the right choice is the simplest thing that can possibly work. WS may work, but it may not be the simplest thing for your project.

But you want to use WS across all projects as a strategic choice, but how do you know that your strategic choice maps well to the types of problems your customer is presenting? These types of decisions rely on feedback. Ease of implementation, relevance, etc. To get this type of feedback you need to be part of the development team, programming.


This is the difference between Emergent Design and BUFD. In BUFD you know that your intent is right and the whole development team get locked into a decision made at the beginning by the architect. In emergent design, you say that you don't know what the design should be at the outset and you allow it to emerge through refactoring over time.

You can take an emergent approach to strategic decisions too. So teams can choose WS or REST or what ever as long as their choice is the simplest thing that can possibly work in their context. Over time the type of technologies that best suite the types of problems in a given organisation will emerge. You may find that as an organisation, that the best option (cheapest, most productive, most responsive, etc) is to allow for a number of strategic distribution technologies to co-exist, integrated through a common backbone (ESB). The difference is that you learned this over time and the backbone that emerges will be ideally suited to your needs (the simplest thing that can possibly work).

I'm really enjoying this discussion. It is helping to clarify a number of ideas I have had for a while. I've just thought of another architectural pattern that is "over-elaborate" and not the simplest thing in most instances where it is applied: DAOs. I think I'll blog about this one too.

Paul.

Ok. I agree and disagree.

I see you like to go extremes, black and white, and sometimes you think I'm in one of those extremes. Probably it is because I'm not totally clear sometimes.

So, let's go one by one.
"Seasoned and skilled are two different things". I agree. And I think I cannot play only with winners, real world does not work that way.
Skillful kids need management and focus so they can learn. They still need experience. Some Seasoned guys may have a a better view for some decisions, but may not be able to beat a kid in coding the latest JSon thing. I have no use of skill java programmer if they fail to make critical decisions. A project will never work with wild programmers in the loose. It is far more complex.

"But we have to deal with juniors due to cost". This doesn't mean I will be hiring only juniors (an extreme). There is a very important task we need to perform and that is Teaming. I need to build teams for projects, that may require UI design guys, DB guys, Scripting guys, service guys, java guys, designers, architects, testers, requirement analysts, etc. I wouldn't like 5 Williams working in my project, they will not listen to me! But I would like some juniors beside seasoned guys, learning and doing the dirty work. And of course, I cannot have them at day 1, I need to understand the problem first and then propose the team that better fits.

"Your job isn't as easy as you may think. How do you know that you have made the right choice?". Sure! It is very complex. Architects (at least the ones that know what are they doing) will not throw a coin to make a choice. There is a study of restrictions, requirements (internals and externals), costs, goals, principles, politics, teams, stakeholders concerns, etc, all of them composing a problem we need to solve. My concern is not about a class interface, or if REST is popular or not. If I'm in an industry where all is WS, I cannot go upstream.

"These types of decisions rely on feedback". That is not totally correct. There is feedback, history, prediction, patterns, statistics, restrictions, politics, and a long etc. To validate a decision we study at least 5 methodologies at university (in my course) and only two of them involve coding. They are more precise, but also cost more.

"To get this type of feedback you need to be part of the development team, programming". Here is another misconception. There is a discussion out there about Architects programming. I'm clear about that: Architects should work on details when these are architecturally significant. I may need to develop on a POC, and I may need to learn new languages just to see if they fit the solution. But an Architect should not be painting a JSP, or creating a Hibernate XML, there are cheaper resources that can do that. Architects do code, the strategic design is a high level code. Tactical design is too. Architects SHOULD be active programmers, and must have experience is at least two languages, including scripting.
Finally, an architect does not live in the Ivory tower. It is hands on during development, verifying, coaching, mentoring, refactoring, obtaining feedback from stakeholders, managing expectations. A hard work someone has to do.

"This is the difference between Emergent Design and BUFD. In BUFD you know that your intent is right and the whole development team get locked into a decision made at the beginning by the architect". That may be the regular result, but that is not the idea (since it makes no sense). And, I'm not a BUFD practitioner (that is another extreme) And not all architects do BDUF. We work with the client, whose view is not in the details. We make decisions based on million things, including code feasibility. Then we design the solution at high level (which does not mean creating hundreds of boxes, the less the better). If there is a problem in java that does not allow us to use WS and I need to change my decision, then I didn't make my homework correctly. But, if that is the case, we need to go and change. No lock.

Now, my decision was on WS, but I'm not deciding on several other issues that the developer will deal with. My decision is a macro decision that solves an strategic point. More details are added by designers and even more by developers. So, there is no such thing (in this approach) as a BDUF. I'm giving the structure, the skeleton, the developers add the meat. Visibility is the key, not documentation. And my junior will not decide WS or CORBA, but he may decide how to process the data.

"You can take an emergent approach to strategic decisions too. So teams can choose WS or REST or what ever as long as their choice is the simplest thing that can possibly work in their context". Exactly. You can make strategic decision, but how do you test they are the best ones? (I'm returning a question here). Actually you can, in several ways, but not inmediately (you will need to wait for the system to grow), and the outcome is you will either refactor to adjust or keep as you did. That is not bad, but it is not the only way to work. You said it:"the simplest thing that can possibly work in their context", that simple thing can be a very different one in another context.

"I've just thought of another architectural pattern that is "over-elaborate" and not the simplest thing in most instances where it is applied: DAOs.". I hate them too (but not the idea). Just to clarify DAOs are not an architectural pattern, it is functional unit definition used in application architectures. It's java implementation is not the best one, and has lots of separation of concerns problems, but that is not the unit's definition fault.

The discussion is really good. Too many new themes for posts!

Thanks, Paul.

William

Hi Will,

I agree it isn't black and white, and I was being intentionally provocative :) Effective collaboration between Architects and Developers is possible.

"To get this type of feedback you need to be part of the development team, programming". Here is another misconception. There is a discussion out there about Architects programming. I'm clear about that: Architects should work on details when these are architecturally significant. I may need to develop on a POC, and I may need to learn new languages just to see if they fit the solution. But an Architect should not be painting a JSP, or creating a Hibernate XML, there are cheaper resources that can do that. Architects do code, the strategic design is a high level code. Tactical design is too. Architects SHOULD be active programmers, and must have experience is at least two languages, including scripting.
Finally, an architect does not live in the Ivory tower. It is hands on during development, verifying, coaching, mentoring, refactoring, obtaining feedback from stakeholders, managing expectations. A hard work someone has to do.

I agree. This is what I tried to do when I was an Architect. But is this the commonly understood role of an architect?

And if it is how many Architects do what you describe here?

You say it yourself:

Everyone talks about architecturing.. so why won't we?

Food for thought!

PS. The only post I curious to see is your take on Emergent Design.

Cheers Paul.

Ok,Paul. I promise I will write about it soon.

For now, please take a look at these slides from Coding the Architecture site.
This presentation analyzes the issue of when to make the strategic decisions.
Note, it is an art to know when to defer a decision, and also an art to know when to get down at code level so sustain an architecturally significant decision.

http://www.codingthearchitecture.com/files/20071003-architecting-during-implementation.pdf

William.

Hi Will,

Read the paper and I agree with everything in it. Especially the conclusion:

- Spike
- Prototype
- Proof of Concept

It also makes the point that you should defer what decisions you can, and make a decision on what you must. Common sense really.

The rest of the paper is a justification for the role of Architect versus that of Senior Development Lead. Is it all just a justification for the salary hike? :)

But the decisions still need to be made and they need to be made from a position of technical authority - otherwise what's the point in architecting?

So Lead developers can't make decisions or is it that they just lack the technical "authority" and can't learn? I guess this is the politics you speak of :)

I for one shouldn't talk about salary hikes, I make an obscene amount as a Consultant:)


Thanks for the link. If this is anything to go by it looks like the centre of gravity in the archtecturing world as moved significantly since I was doing it back in 2000.

Cheers,

Paul

Wow. Long discussion

1. Yes, architect should decide what they must (architecturally significant ones), other are deferred to more detailed design (including coding).
2. Senior Development Lead deals with development, Architects may play that role, but they need to also handle other things. SDL is a lot of work byt itself.
3. SDL will make authority decisions, of course!. Architects may need to decide another ones with a different authority. The point in that sentence I guess is that those decisions may not be taken by just anyone, they must be taken by someone who really knows.
4. Architecture is always moving. And will evolve even more, given the feedback received and these discussions we have :-D

William.