Products, Projects, Processes
Systems architectures combine software components with users and actual devices:
- Users (supposedly human beings) possess symbolic processing capabilities; they may also be granted with organizational status and responsibilities.
- Software components and actual devices cannot be granted with organizational status or responsibilities; the former do possess symbolic communication capabilities, the latter do not.
Assuming organizational contexts defined by agents and responsibilities on one hand, technical contexts set by technology, resources and legacy systems on the other hand, the objective is to rationalize the development process of software components.
As noted before, the outcome of engineering processes must meet different kinds of requirements: business goals, system functionalities, quality of service, and platform implementations. In a perfect Model Driven Engineering world there would be one stakeholder, one architecture, and one time-scale.
Unfortunately, those requirements are usually set by different stakeholders, governed by different rationales, and subject to changes along different time-frames. With regard to governance, the objective of engineering processes is therefore to manage those goals along time and arbitrage between overlapping or even conflicting agendas.
Engineering processes must be considered when tasks cannot be performed simultaneously and have to be divided and sequenced. That’s the case when, due to contexts or dependencies, developments cannot be carried out by atomic activities but call for more complex undertakings.
To begin with, building blocks must be clearly defined; for that purpose one can use Aristotle’s three unities for drama:
- Unity of action: a work unit should have one main objective, be governed by a single authority, with one single outcome.
- Unity of place: developments should be executed into a space within which all resources can be addressed directly and accessed independently of physical constraints, and communication carried out without mediation.
- Unity of time: work units should be set within fixed and continuous time-frames and, once started, should proceed until completion independently of what may happen elsewhere. That is a necessity if innards and synchronization of tasks are to be managed independently of context contingencies.
Factors associated with contexts may also affect the sequencing of work units:
- Decisions falling under the authority of different organizational units.
- Geographical constraints bearing upon execution.
- Unpredictability of contexts and execution making intermediate milestones necessary.
Those rules are applied to project objectives in order to define work units and plan their execution.
The reason for Processes: The Cooking Metaphor
The need of intermediate products is at the root of development processes: paraphrasing Einstein, the only reason for processes [time] is so that everything doesn’t happen at once.
The rationale of development processes may be illustrated by the cooking metaphor, with development artifacts equated to cooking ingredients. While there is an infinity of ingredients, most belong to four categories: water, proteins, carbohydrates, and fats. The ways they react to physical processing and can be combined are set by chemical laws. It is therefore possible to characterize cooking recipes and identify basic states and processing sequences independently of their specifics.
Taking as examples a few egg-based recipes:
- Poached Eggs (atomic, single flow): a single operation (put eggs into boiling water) whose outcome must be synchronized with users needs
- Omelette (composition, single flow): two sequenced (not necessarily synchronized) operations (beat eggs, fry) performed on the same contents (not necessarily by the same operator), under a single clock, with all resources at hand.
- Eggs Benedict (aggregate, joint flows) : joint and synchronized operations, executed in parallel by different operators, under a single clock, with all resources at hand.
- French Crepes (aggregate, single flow): two phases (making the crepes, adding savouries), the first one made of sequenced operations, executed successively by different operators, possibly under different clocks, within different working spaces.
- Crepes Suzette (restaurant): the same as above but necessarily executed within different spatial and organizational contexts: kitchen for the making and in front of the customer for the “flambé”.
Recipes 1 and 2 don’t call for engineering processes or intermediate outcomes. Recipe 3 must take into account intrinsic constraints on flows, as described by a process; intermediate outcomes don’t have to be managed. Recipes 4 and 5 should follow predefined processes describing tasks, flows, intermediate outcomes and associated constraints.
As illustrated by these examples, discontinuity can be intrinsic (nature of products) or extrinsic (nature of participants): for simple processes (e.g poached eggs), all products are delivered to customers, for structured ones products will also include intermediate outcomes to be instantly used (e.g eggs Benedict) or kept aside for a time (e.g crepes).
That is what model driven engineering is about: reasoned processing of models depending on the fundamental properties of their contents: computation independent, platform independent, platform specific.
With regard to software development artifacts, and whatever their form (textual specifications, test cases, models, code, etc.), intermediate products are introduced to support discontinuity in time or teams:
- Time discontinuity is set by the nature of development flows: when sequenced tasks can be performed asynchronously, output/input flows should be managed independently of associated producing and consuming tasks.
- Team discontinuity is set by organizational context: when sequenced tasks are to be performed by different teams (whatever the reason, technical or organizational), shared flows should be managed independently of the organizational units involved.
Obviously, time and team discontinuities are not at the same level, the former being governed by products characteristics and the latter by projects organization.
Products and Development Flows
Products involved in software engineering processes can be mapped to architectures set by the Actual/Symbolic and Persistent/Transient categories:
- Enterprise architecture deals with organizational and operational contexts: locations, agents and organization, business objects and activities, events.
- Functional architecture describe systems functionalities with regard to business processes: business domains and symbolic representation of persistent objects.
- Software architecture describe applications with regard to functionalities: business logic, persistency, user interfaces.
- Technical architecture: describe systems and components with regard to applications.
Taking into account those discontinuities, developments can be set along two axes:
- Operations/Applications: self-contained developments target applications that can be defined, built, and accepted under a single authority.
- Functional/Technical: shared developments target architecture components whose specifications, integration or acceptance cannot be set under a single authority.
As it happens, that distinction can be used to characterize projects according to their footprint (enterprise or system) and purpose (assets or users’ value):
- Purpose: management of shared assets (architects) vs specific requirements (analysts).
- Target: enterprise vs information systems.
Applications development can be directly driven by requirements (use cases, users’ stories, …) when these can be fully set under a shared ownership, otherwise cross dependencies will have to be consolidated by the functional architecture and managed through models.
Projects and Development Models
- Bureaucratic (aka waterfall) processes follow predefined sequences of tasks targeting standard outcomes and deliverables. Tasks and deliverables can be customized but adjustments don’t affect contents. Due to the one-fits-all philosophy such processes lack built-in mechanisms; as a consequence quality management must be performed from the outside and the effectiveness of the whole depends heavily on support and expertise. Their main advantage is with integrated projects involving different teams across organizations, domains and architectures.
- Democratic (aka agile) processes take the opposite view and let project organize work units and deliverables as befit the objectives and contexts. Milestones are set aside by direct collaboration between business and system analysts, and scheduling is replaced with iterative modelling and time-boxing. Whereas such processes still lack built-in mechanisms, quality management is performed from within, based upon responsibility and communication. Overheads are limited but expertise and best practices are essential. Agile approaches may bring substantial benefits to complex projects with well-defined perimeter and limited external dependencies.
- Automated processes (as promoted by model based approaches) are the holy grail of software engineering. Processes are organized around predefined models subject to transformation. Providing comprehensive and consistent semantics, built-in mechanisms can be used all along the processes, especially for quality management. Expertise can be concentrated on models and rules, with best practices taking a back seat.
Each approach addresses some part of the problem, and combining them within a unifying paradigm should bring cross benefits: milestones for integration across business units and architectures, agile for projects, automated transformation when analysis and design decisions are set.
If development models are to be set under a common roof, their respective understanding of work units must be consolidated.
- Activity-based work units (waterfall): work breakdown starts with one-fits-all organization, follows top-down with predefined activities, which are then entrusted with production objectives
- Team-based work units (agile): no breakdown but iterations managed collectively.
- Flow-based work units (Model based): work breakdown starts with production steps and builds processes bottom-up, depending upon constraints.
Manufacturing processes, built from material flows and engineering constraints, have demonstrated the efficiency of flow-based approaches. Despite advances made by model based initiatives, e.g MDA, software engineering hasn’t followed suit and work units, when explicitly defined, remain customized versions of the same catch-all activities. As a consequence, there is no way to map problems (what is to be developed) to solutions (how it should be done). That’s what processes are about.
Processes and Development Patterns
Just like software products are solutions to application problems, development processes are solutions to development problems, namely how to deal with dependencies between work units.
On that basis, standard work units can be associated to development flows themselves derived from development problems. Road-maps can then be organized along architecture tiers and model layers:
- Architecture tiers deal with products dependencies: standalone, shared and transient, shared and persistent, communications.
- Model layers take into account stakeholders and concerns along application life-cycle: business, functionalities, components.