A recent paper about performance engineering of the Community Earth System Model (CESM) discusses current approaches to load balancing and performance optimization for that model [1]. The authors describe a two-step process to performance optimization: First, some empirical data is gathered about the performance of each constituent at different processor counts for the given platform and problem specification. Second, the local performance information is used to decide how to assign processors to maximize the coupled model’s throughput. Like most climate models, CESM’s constituent models are parallel applications in their own right with a number of different parameters that can be tuned to optimize performance.

At the current time, climate model performance is dominated by computation, not communication. Therefore, model throughput is improved best by ensuring that computations of the constituent models overlap as much as possible so that processor idle time is reduced to a minimum. This tuning is done primarily by carefully choosing the number of processors to allocate to each constituent. The whole process is more or less a trial-and-error tuning guided by empirical performance data for each of the constituents.

For current processor counts, the “enlightened trial-and-error” procedure works relatively well. However, as we approach the exascale era, it is expected that the performance characteristics of the models will change significantly. At today’s typical processor counts, the coupler overhead does not introduce a significant performance penalty. A study by the Earth System Modeling Framework shows that for one typical configuration of CESM, time spent in the coupler is less than 3% of the total running time [2]. However, at very large processor counts, the number of messages required for both intra- and inter-model communication increases dramatically such that optimization of the coupler will become important to maintain high model throughput. The last set of experiments in the CESM report supports to this prediction: For the highest resolution runs, the percentage of time spent inside the coupler increases dramatically when compared to typical resolution runs. At some processor counts, the coupler accounts for nearly half of the execution time [1].

The current performance tuning approach is convenient because it allows the user to focus on local optimizations while still giving reasonably good global performance characteristics. Unfortunately, the changes introduced by the next generation of supercomputer will likely require a change in performance tuning strategies. Namely, more thought will have to be put into global performance optimization of the models because it is less likely that executing locally optimized models in parallel will give the best model throughput. Much of this cost will be due to a shift from computation to communication as the performance bottleneck. Effective performance tuning will require the assistance of tools such as automatic load balancers. Such tools should take as input a set of requirements (e.g., the set of constituent models) and local performance characteristics of each constituent, and generate a configuration that is globally optimized for maximum throughput. Unlike local optimization, global optimizations take into account knowledge of the system as a whole, such as the pair-wise communication requirements among the constituent models. In some cases, it might be that the globally optimized configuration is not locally optimal. For example, we might configure the decomposition of the land model to more closely match that of the atmosphere in order to reduce land/atmosphere communication. In this case it is possible that the land’s decomposition results in a reduced throughput for that constituent, even though it ensures higher throughput of the coupled system.

There are a number of “tunable” performance parameters for each constituent. For a global configuration of several constituents, the state space of possible solutions becomes too large to manage manually. Taking into account just one parameter, number of processes, let’s look at a global optimization problem for a fictitious coupled system with three models. This figure below shows the scaling curves of the three constituents: A, B, and C. The horizontal axis is number of processes and the vertical is hours of wall clock time required to simulate one year.

The optimization problem is this: how many processes should I assign to each model to maximize throughput? Note that if we draw a horizontal line, say at 2.5 hours, then we can automatically get a load balanced model because each model will take the same amount of time for its computation (assuming the empirical curves are good predictors, of course). Keep in mind, however, that this load-balanced configuration may not necessarily be the optimal solution in terms of model throughput.

Another parameter we have to play with is whether models are executed sequentially or in parallel. With three constituents there are 5 different configurations. The notation AB means that A and B run sequentially, one after the other in the same process. The notation A||B means that A and B run in parallel. Assume that the sequential operator takes precedence over ||, that is, AB||C means (AB)||C.

Possible configurations:

• A||B||C – fully parallel
• ABC – fully sequential
• AB||C – hybrid
• AC||B – hybrid
• A||BC – hybrid

As a simplification, we will assume that models executed sequentially have independent performance characteristics (i.e., we will neglect any gains due to the potential for these constituents to share memory). This allows us to sum the scaling curves of two or more constituents that execute sequentially and treat the set as if it were a single constituent. (This assumption should be relaxed!) We then wind up with one (fully sequential), two (hybrid), or three (fully parallel) sequential sets of constituents executing in parallel. The best throughput for a particular configuration can now be achieved by overlapping computation as much as possible for all the constituents running in parallel. The best global configuration can be found by taking the best throughput found among the five configurations.

In addition to relaxing the assumption that sequentially executing constituents have independent performance characteristics, both additional parameters (e.g., decomposition strategy) and additional constraints (e.g., memory available on each node) need to be taken into account to improve the quality of the optimization.

My question is if we add a realistic set of parameters to the global coupled model optimization problem, is it computationally tractable?

[1] Patrick H. Worley, Arthur A. Mirin, Anthony P. Craig, Mark A. Taylor, John M. Dennis, and Mariana Vertenstein. 2011. Performance of the community earth system model. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’11). ACM, New York, NY, USA, , Article 54 , 11 pages. DOI=10.1145/2063384.2063457 http://doi.acm.org/10.1145/2063384.2063457. (pdf)

[2] Peggy Li. ESMF Component Overhead in CCSM4.0. (pdf)

Separation of concerns has long been recognized as an important goal when building software. In an well-cited paper, Peri Tarr et al. point out that most software formalisms limit our ability to truly separate out all concerns due to what they call the “tyranny of the dominant decomposition” [1]. In other words, a single dominant dimension is typically selected as a decomposition rule, such as the object structure most evident in the application’s domain. Tarr et al. argue that single dimension separation is not ideal because the concerns of software systems are almost always overlapping in nature–that is, single objects or modules are often responsible for implementing multiple concerns. Therefore, the elements present in the dominant decomposition are responsible for handing multiple concerns such that separation of concerns is not really achieved. They propose an abstraction called a “hyperslice” which is a kind of module that encapsulates a concern other than the dominant one. Hyperslices are powerful because a given programming element (e.g., class, method) may appear in multiple hyperslices. Hyperslices are combined using composition rules to form the complete system.

I do not find it hard to argue that climate model software development has indeed fallen prey to the “tyranny of the dominant decomposition.” Informal architecture diagrams of climate model software almost always depict a single kind of functional decomposition: geophysical domain such as atmosphere, ocean, land, etc.  To see this, check out some excellent work comparing climate model architectures. (Take a look at the linked diagrams and notice what each of the bubbles represent regardless of the particular climate model in view.)

To be sure, it is hard to argue against decomposition by geophysical domain since it divides up the code nicely based on scientific expertise (i.e., those contributing to the code), encapsulates the data fields related to each of the domains and because scientists often want to run one of those components as standalone application (e.g., to isolate the atmospheric model for sensitivity analysis). The dominance of this decomposition rule can also clearly be seen by studying the reusable coupling technologies out there–to my knowledge, every one of them assumes a functional decomposition based on geophysical domain.

Traditionally, one-dimensional separation of concerns is seen as a Bad Thing. The issue is that the other concerns are diffused throughout the primary decomposition structure. This diminishes the ability to locate, comprehend, verify, and reuse the diffused concerns.

To make the discussion concrete, what “concerns” do we see in climate models that are candidates for isolation? Answering this requires some degree of expert input or at least careful consideration of the software requirements of a climate model. Some general purpose guidelines can be found in the literature about how to select concerns (e.g., definitive membership, finite domain, etc.) but these do not really help to identify the concerns of a particular domain or software system. Based on my experience thus far studying climate model software, here are some possibilities:

• grids / numerics / domain discretization
• interpolation / accumulation / averaging
• coupling
• domain decomposition
• driving / execution schedule
• configuration
• I/O
• error handling
• logging

What’s not included in the list above are the myriad scientific choices involved.  Is the ocean dynamic? Are cities modeled? What kind of dynamical core is used to drive the atmosphere? All of these scientific choices are indeed software concerns because they have to be implemented in software and their implementations are deeply inter-connected.

Different Meanings of Modularity

At this point, I should mention a recent discussion paper regarding modularity of features in software systems by Kästner, Apel, and Ostermann [2]. The paper describes an ongoing debate among members of software engineering and programming language communities about the costs and benefits of modularizing software features. (For now, let’s just assume that software “features” and the “concerns” I mentioned above are roughly equivalent, although traditionally a “feature” is a more user-centric term while a concern is a more developer-centric term. To fully appreciate the paper, read up on Feature Oriented Domain Analysis first.)

Part of the debate revolves around two different notions of modularity: Let’s call the first type informal modularity.  The paper describes this kind of modularity as meaning cohesion and locality. In other words, all code artifacts related to implementing a particular feature are placed into a separate structure, such as its own class, file, folder, or module. This kind of informal modularity seeks to provide many of the practical benefits (ease of maintenance, comprehension, evolution, etc.) without incurring the costs of an overly burdensome interface mechanism. As an example of this type of modularity, imagine gathering all the #ifdefs related to a particular configuration option into a single place instead of leaving them scattered throughout the codebase.

The second kind of modularity, which we’ll call formal modularity, seeks to achieve true information hiding by separating modules into an internal part and an external part. The external part, called the interface, establishes a contract between the module and the rest of the system. Details about what’s behind the interface are hidden. As pointed out in [2], this more formal modularity makes modular reasoning possible such as automated type checking (i.e., ensuring that module compositions are safe and correct) and separate compilation (ensuring that the module itself is at least syntactically valid). If semantic interfaces are available, more advanced kinds of reasoning can be used to test properties of composed modules.

There are tradeoffs involved with these two views of feature modularity:  The informal approach provides practical development benefits, but lacks the rigor required for automated reasoning. The formal approach provides many benefits such as reuse (e.g., an open-world view) and independent development and testing [2]. The benefits of formal modularity have to be weighed against the costs: Kastner et al. point out that granular and/or crosscutting feature modules may end up with interfaces that essentially contain the whole of the feature implementation (so that nothing is hidden). Furthermore, a feature may have a lot of interactions with other features requiring a large number of micro-modules to encapsulate pair-wise (or higher cardinality) feature interactions.

To Modularize or Not?

Some questions then:

• Would climate science benefit from modularity of the kinds mentioned above?
• If yes, which definition of modularity is most appropriate: modularity as cohesion and locality or modularity as true information hiding?
• What would a feature-modularized climate model look like? What climate model architecture would support multi-dimensional separation of concerns?

Some community discussions need to take place on these issues.  Probably the issue of most general interest is that of climate model verification. In a previous entry, I pointed out that software complexity is the enemy of climate model verification. As more sophisticated science is introduced into the model, the problem will only get worse. Now is the time to make some decisions about how to tame model complexity and I argue that improved modularity is perhaps the most important step the climate modeling community can take to increase verifiability of the models.

There is very interesting software engineering research on whether the cohesion/locality or information hiding viewpoint is most appropriate. In a previous experiment, I attempted to separate out the coupling interface from the rest of the model in the style of Robert DeLine’s Flexible Packaging. This was only a very simple case, but in reflecting on it I realize that much of the difficulty lies in the fact that the scientific parts of the model are highly dependent on the coupling infrastructure. In other words, were we to define a formal “information hiding” type of interface for the coupling infrastructure, it would be a large interface with many parameters. One way to characterize this would be to count the number of API parameters to coupling technologies like ESMF and OASIS/PSMILe.

For now, there are many open questions about how to best realize the benefits of modularity in climate models while still negotiating the costs of modular implementations. More to come on this topic!

[1] Peri Tarr, Harold Ossher, William Harrison. “N degrees of separation: multi-dimensional separation of concerns.” ICSE 1999.

[2] Christian Kästner Sven Apel Klaus Ostermann. The Road to Feature Modularity? SPLC 2011.

VampirTrace automatically instruments C, C++, and Fortran programs so that important events are recorded as the program executes. VampirTrace is designed to work with high performance applications, such as distributed memory parallel codes based on MPI. This works well for us since the vast majority of ESMs are based on this paradigm. When a program compiled with the VampirTrace compiler wrappers is executed, certain events will be recorded and, after program execution, the trace data will be available in a standardized format called Open Trace Format (OTF).

Lots of different events can be traced with fine grained timing data:

• MPI events such as message send, receive, and collective calls
• Function entry and exit
• System calls such as memory allocations and I/O calls
• Other user-defined events (requires manual code instrumentation)

A major advantage of using ESMF is that it provides a number of numerical modeling abstractions that would otherwise have to be coded by hand. Many of these abstractions hide the details of rather complex interactions, such as repartitioning field data from one decomposition (processor layout) to another and interpolating field data between grids with different resolutions. Because we are interested in what is going on within ESMF (not just within the user code that calls ESMF functions), I have instrumented ESMF itself using VampirTrace.

Instrumenting the ESMF library using VampirTrace

These are the software packages I am using:

• ESMF 5.2.0r
• Intel Fortran and C compilers (Composer XE 2011 SP1.6.233)
• OpenMPI 1.4.3
• NetCDF 4.1.3 (only required for I/O operations)
• VampirTrace 5.11.2

Although VampirTrace ships with the OpenMPI distribution, it seems that the version of VampirTrace packaged with OpenMPI lags behind the latest stable release.  So, I use the latest version of VampirTrace and ignore the version packaged with OpenMPI. VampirTrace uses compiler wrappers for automatic code instrumentation. So, the ESMF build environment must be set up to use the VampirTrace wrappers instead of the usual C and Fortran compilers.  The Intel compilers, OpenMPI, NetCDF, and VampirTrace packages are all set up using the standard build/install instructions that come with those packages.

Here are my ESMF environment variables:

export ESMF_COMM="openmpi"
export ESMF_COMPILER="intel"
export ESMF_DIR="<some directory>/esmf"

export ESMF_CXX=vtcxx
export ESMF_F90=vtf90
export ESMF_CXXCOMPILEOPTS="-vt:cxx mpicxx -DVTRACE"
export ESMF_F90COMPILEOPTS="-vt:f90 mpif90 -DVTRACE"

With these environment variables, ESMF will be compiled using the VampirTrace wrappers vtcxx and vtf90 instead of the OpenMPI compiler wrappers mpicxx and mpif90. Make sure that VampirTrace, OpenMPI, and Intel compilers are all in the PATH or specify absolute locations for ESMF_CXX and ESMF_F90.

Now build ESMF and install it.

$ gmake
$ gmake install

I do not recommend executing gmake check because all of the ESMF unit and system tests will be executed with tracing turned on.  If you wish to check the build, set up your environment using the OpenMPI compiler wrappers, execute gmake to recompile and then run gmake check.  If everything checks, then rebuild with the VampirTrace compiler wrappers.  The only difference will be that ESMF is linked against the VampirTrace library instead of the regular MPI library. So, we’ll assume that the unit and system tests would pass in this case. If they did not, the bug would be within VampirTrace itself.

Okay, now that we have an ESMF build linked with the VampirTrace libraries, let’s run a small ESMF application and take a look at the resulting trace data. The easiest way to do this is to execute one or more of the system tests that ship with ESMF.  For example:

$ cd $ESMF_DIR/system_tests/ESMF_ArrayRedist
$ gmake
$ cd $ESMF_DIR/test/testO/Linux.intel.64.openmpi.default
$ mpirun -np 6 ./ESMF_ArrayRedistSTest

A couple notes:

• Line 2 will compile the ESMF_ArrayRedist system test.  It will probably fail with an error File not found:  ’opari.tab.c’.  To get around this, I’ve had to execute opari -table opari.tab.c manually.  I’m not sure why it is not working automatically.
• The directory location in line 3 will change depending on the os, compiler, and MPI implementation you are using.

Line 4 will execute the instrumented system test. When it finishes, you should see several files produced in the same directory with the ESMF_ArrayRedistSTest executable: a series of files ESMF_ArrayRedistSTest.X.events.z (one for each of the six MPI processes), a file named ESMF_ArrayRedistSTest.0.def.z, and a file name ESMF_ArrayRedistSTest.otf.  This set of files contains the trace data for the run.

Analysis with Vampir

Vampir is an analysis and visualization package capable of reading trace files in OTF format.  It is not open source, but a demo license is available.  Let’s examine the trace files from the array redistribution system test.

1. Install Vampir (download here)
2. Start Vampir
3. Open the file ESMF_ArrayRedistSTest.otf that was generated from the system test run

This view shows the number of MPI messages grouped by message size. This particular plot is for a zoomed-in section of the timeline–it only covers messages passed during the ESMF_ArrayRedistStore function. The total amount of data transmitted can also be determined as well as the data throughput. This shows us that ESMF requires 152 MPI messages to calculate a repartitioning for the particular scenario in the system test (4 processors to 2 processors, 4×1 decomposition to 1×2  decomposition, 100×150 grid resolution on both components).

The communication matrix view shows properties of messages sent between individual processes.  Sending processes are on the vertical axis and receiving processes are on the horizontal axis. Again, this particular plot is only for the ESMF_ArrayRedistStore function.

For now, this just gives an idea of the kinds of analyses supported by VampirTrace + Vampir.  I have been impressed by the initial experience with the software. Potential issues may arise as the number of processes increases and the overall length of the run (this run took 7 seconds wall clock time), but so far so good.

One reason that the mapping problem can be difficult to solve is that most programming languages are not architecture-aware. Eichberg et al. have done some work on maintaining constraints on structural program dependencies defined over arbitrary groupings of code elements called ensembles [2]. They rightly point out that the module system built into a programming language is orthogonal to the module system in the architecture. I would agree with this whole-heartedly, especially when it comes to connectors which are by their very nature prone to being highly distributed at the implementation level. It is perhaps a bit easier to form one-to-one mappings between architectural components and program modules, but even this should not be assumed. In the Eichberg work, ensembles are defined using a domain-specific language based on Datalog and myriad different Java constructs may be identified as being part of an ensemble (e.g., class, interface, method, field, method call, etc.).  This means that an ensemble definition may cross-cut Java modules. This is similar in spirit to the way that pointcuts are used in aspect-oriented programming to select “where” in the program (statically or dynamically) an aspect’s code should be executed.

Architectural integrity, then, can be maintained by creating ensembles that represent architectural concerns (e.g., a component or a layer) and then constraining the dependency relationships among ensembles (e.g., only factory classes can execute certain object constructors or classes in layer N must only “use” classes in layer N-1). Through the use of Datalog-like expressions, a wide range of constraints can be specified.  It’s not clear to me exactly how to quantify the expressiveness of this language without a formalization of possible architectural constraints, but at first blush it seems expressive enough to write even complex static structure-based constraints. Kudos to the authors for including a visual language with notions of in-ports and out-ports that can be connected with arrows. Also included are Java annotations which are used to define ensemble membership right in the code. This work is primarily targeted at maintaining architectural integrity as an application evolves. Through integration with the Eclipse IDE, constraints can be checked with every incremental build and any violations of the architecture are immediately made known to the developer. At this point, it is not clear to me how this approach would work for a development methodology in which an architecture is designed up front before any coding constructs exist (i.e., it is hard to define ensemble memberships unless the code structures already exist). Also, there is lack of support for ensembles based on dynamic properties of the implementation.

The Archface work in [3] is another technology designed to bridge the gap between architecture and implementation. An “archface” is a type of interface that represents an architectural component or connector. The building blocks of Archface are pointcuts from aspect-oriented programming. The pointcuts identify architecturally-relevant program points (e.g., method execution, method call, field read/write, etc.) and they can be exposed as ports (in or out). Connector interfaces are used to link ports–that is, they coordinate the program points exposed via ports. So, just like the Eichberg work described above, a pointcut-like mechanism is used to expose program “locations” that are architecturally-relevant. Although somewhat hard to compare, there appears to be a difference in what you can do with the exposed program points: In the Eichberg work, constraints can be defined over the dependency relationships among the ensembles in an architecture. In Archface, a kind of mediation happens when connectors are linked–for example, the call of a method in one port is linked to the execution of a method in another port. This is a kind of indirection that is mediated by the connector using the AOP notion of advice. I think a major reason for this difference is that Eichberg et al. do not recognize a first-class connector. That being said, it is not clear to what degree the Archface connectors correspond to connectors as defined in the classic software architecture literature. Perhaps I get this feeling only because the connector examples given in the Archface paper [3]  are only used to connect ports and do very little processing on their own. I guess, then, that the expressiveness of the connectors boils down to what advice mechanisms are available. Another major difference between Eichberg and Archface is that Archface inherits dynamic pointcuts from AspectJ whereas the former is strictly static.

But, bottom-line takeaway from both of these papers is that some of the the latest thinking about spanning the gap between architecture and implementation involves some kind of mechanism to abstract and expose architecturally-relevant programs points and then manipulate them in some way either by constraining their dependency relationships or mediating their interactions. Both of these mechanisms seem very powerful allowing for the expression of all kinds of architectures. A concern I have with both–but especially Archface–is whether they offer intuitive mechanism for specifying the relationship between architectural concerns and the implementation. For example, even in the simple examples given in [3] you have to do a kind of mental code-weaving to get a feel for an architecture described by Archface. It could get pretty ugly if a complex architecture were described.

[1] Richard Taylor, Nenad Medvidovic, Eric M. Dashofy. Software Architecture: Foundations, Theory, and Practice. Wiley, 2009.

[2] Michael Eichberg, Sven Kloppenburg, Karl Klose, and Mira Mezini. “Defining and Continuous Checking of Structural Program Dependencies.” ICSE 2008.

[3] Naoyasu Ubayashi, Jun Nomura, Tetsuo Tamai. “Archface: A Contract Place where Architectural Design and Code Meet Together.” ICSE 2010.

The ability to predict the degree of code-level coupling introduced by architectural connectors is important because many software qualities are determined by the dependency structures of the implemented system. For example, if a connector implementation requires code changes throughout several modules, then a program’s maintainability and understandability may be diminished.  Or, if the connector’s interaction protocol requires a large amount of inter-component communication (especially data copies) then overall system throughput could be diminished.  Unfortunately, it is often hard to determine the code-level modularity characteristics that will result from choosing a particular architectural connector.

One way to mitigate the inability to determine degree of coupling from architectural descriptions is to make code-level impacts of architectural connectors explicit in the architectural description. Some things we’d like to know:

• What code modules are affected when a particular connector is chosen?
• To what degree are the changes localized within the affected modules?
• What are the properties of the required module interfaces in terms of number of parameters, size of communicated data, number of messages passed, degree of data localization, complexity of communication protocol, etc.?
• What external resources does the connector require (e.g., files, network, database, etc.)?
• How many code-level dependencies will be introduced and what is the nature of those dependencies?
• What kind of control structures does the connector require?  Is control uni-directional or bi-directional?

It is possible that an abstract connector might be implemented multiple ways.  In this case we want to know the things listed above for each of the possible implementations.

A well-known paper by Mehta et al. [1] notes that architectural connectors may be primitive or complex.  Primitive connectors are typically implemented via programming languages (e.g., procedure call, shared memory access) while complex connectors are composites made up of primitive connectors, other composite connectors, or even entire components.  Composite connectors are often implemented by libraries or frameworks. Mehta et al. point out that it is important to establish some conceptual framework to support reasoning about the low-level interactions present in composite connectors.

Good modularity can be achieved by reducing the amount of coupling (inter-dependence) among the modules in a system. In some of the earliest work on measuring coupling, Meyers provides six distinct levels [2]. These were later ordered by Page-Jones [3] according to their effects on certain qualities such as understandability and maintainability.  Meyers’ six-level scheme was extended by Offutt et. al to include several new levels of coupling and directionality [4]. I have reproduced that below. In the definitions each variable use is classified as a computation-use (C-use) in which a variable is used in an assignment or output statement, a predicate-use (P-use) in which a variable is used in a predicate statement, or an indirect-use (I-use) in which a variable is a C-use that affects some later predicate in the module. More recently, a slew of coupling metrics have been introduced that relate specifically to object-oriented systems. We’ll ignore those for the time being because most of the scientific code I work with is procedural in nature (go Fortran!).

The higher degrees of coupling are undesirable in software systems because they indicate that modules have a lot of dependencies (or a few particularly ugly dependencies).  Too many dependencies can reduce program understanding and cause maintenance headaches down the line. So, what we’d like to do is give a system architect some idea of the level of coupling introduced when choosing connectors without having to build the whole system first.

Most of the work on measuring coupling came about before the software architecture community honed in on the idea of connectors as loci of interaction that should be give first-class status. But, there is a clear relationship between the coupling levels defined above and primitive connectors. Most of the coupling levels imply a simple interaction mechanism: the procedure call. For example, coupling level 1 is a procedure call in which no parameters are passed. This corresponds with the Mehta Procedure Call connector type with certain constraints on the connector dimensions: no parameters, explicit invocation, single entry point, etc. (Mehta defines connector dimensions as “architecturally relevant details” of the connector.) The global coupling level (10) in which two modules share a reference to the same global variable corresponds to the Data Access connector with a “transient” availability (i.e., variable is located in memory, most likely on the heap).  Data Access with “persistent” availability (e.g., file or database) would correspond to coupling level 8, external coupling. The analytic power of linking connectors with coupling levels allows a system designer to make informed decisions about how different connectors will affect inter-module dependencies. For example, based on to the Offut levels defined above, changing a Data Access connector from a file-based communication mechanism to a database mechanism reduces the level of coupling.

Things will likely get more interesting when we consider composite connectors. For example, if we know the coupling behavior introduced by the primitive connectors that make up a composite connector, how can we leverage that knowledge to say something about the level of coupling introduced by the composite connector?

More recently, some work has been done on bridging the gap between architectural descriptions and implementations. One approach is to add architectural abstractions to a programming language. This is the approach of ArchJava which, for example, adds component, port, and connect keywords to the Java language.  The idea is to give programmers the ability to communicate the architecture by explicitly identifying architectural details right there in the code. Although this approach is shown to be effective for ensuring communication integrity (i.e., only architecturally connected components should communicate directly), I think this approach suffers from something like the observer effect.  The observer effect says that when you observe a phenomenon, the observation procedure itself often ends up effecting the behavior of the phenomenon. The Wikipedia example is a good one: when you measure the air pressure of your tires, you probably let a little air out while taking the measurement. The tie in with ArchJava is that in specifying architectural connection concretely in a programming language (the observation), some decision has to be made a priori about how to implement the connectors (the effect). In the case of ArchJava, a port is essentially a container for method signatures. So, quite literally, all architectural connectors implemented with ArchJava are object-oriented method calls!  This approach is therefore not useful for describing existing systems in which we want to illuminate the component-to-component interaction mechanisms that are already in place.

Stay tuned… more to come on this topic.

[1] Nikunj R. Mehta, Nenad Medvidovic, and Sandeep Phadke.  ”Towards a taxonomy of software connectors.” Proceedings of the 22nd international conference on Software engineering, 2000.

[2] G. Myers, Reliable Software Through Composite Design. New York: Mason and Lipscomb Publishers, 1974.

[3] M. Page-Jones, The Practical Guide to Structured Systems Design, 2nd ed. New York: Yourdon Press, 1988.

[4] A. J. Offutt, M. J. Harrold, and P. Kolte, “A Software Metric System for Module Coupling,” Journal of Systems and Software, vol. 20, pp. 259-308, 1993.

To be sure, there are kinds of analyses that are fully realizable even at the architectural level. However, these kinds of analyses are often theoretical in nature and many of them are predicated on having an architectural specification written in an Architectural Description Language (ADL) based on a formal conceptual model.  The formalism provides the semantic foundation for the set of supported analyses.  For example, the Wright ADL, which is based on Communicating Sequential Processes (CSP), relies on the notion of process refinement to automatically check compatibility between a component’s port and a connector’s role.  Another analysis allows you to confirm that the composition of roles in a connector is deadlock free.  While these and similar analyses are theoretically sound, you still get the feeling that they are somehow a bit too far removed from the actual software systems that they describe.  For example, in most cases port/role compatibility is likely checked by the developer by comparing APIs, ensuring that some common datatypes or suitable exchange mechanisms exists, and, in the case of a particularly careful developer, checking pre- and post- conditions, etc.  Often (but not always), the behavior of an external component that you wish to interact with is not so complex that you require a formalized description of it to determine if it is compatible with the rest of your system.

ADLs could be placed on a spectrum like the one pictured below.  There are some quite generic ADLs (e.g., ACME) that, although broadly applicable, have limited analysis capabilities.  On the other hand, some ADLs are highly-domain specific (e.g., MetaH). These languages have limited applicability, but provide richer analysis capabilities for systems that fall within the scope of the language.  A good majority of the languages fall somewhere in the middle to left-hand side of the spectrum.  For example, the Wright ADL is targeted at systems in which it is useful to describe the component and connectors in terms of their abstract behaviors.  For systems with straightforward behavioral models, it might not be worth the effort to describe them using Wright.

An “ideal” ADL would have both broad applicability (i.e., could describe a wide-range of different kinds of systems) and would offer a wide range of targeted architectural analyses.  One way to accomplish this would be to design a modular, extensible ADL such that analysis tools and different kinds of semantic frameworks could be “plugged in” to the language. This kind of thing is offered to a limited degree by ACME because you can annotate architectural elements with properties that, although ignored by the ACME tooling, could still be interpreted by external tools. The problem is that these external tools tend to be proprietary “one-offs” instead of reusable analysis modules made available to a large audience.  One could imagine a market of architectural analysis modules that are well documented, downloadable, and easily integrated with existing ADL tooling.

[Bosch] Jan Bosch.  Design and Use of Software Architectures: Adopting and Evolving a Product-Line Approach. Addison-Wesley Professional, 2000.

One question we might ask is if we can gain any significant benefit from formalizing the architecture of the scientific software such as numerical climate models and other high performance computer simulations. From my experience with the climate modeling community, there are very few informal architectural descriptions to be found, let alone descriptions with enough semantic precision to enable meaningful architectural analyses.

There are a number of different architecture description languages (ADLs) out there. [1] is a good survey paper. Each ADL enables certain kinds of analyses and makes other analyses impossible or difficult. Most ADLs support notions of components, connectors, and ways to configure instances of these into systems. Most ADLs sit on top of a conceptual framework (e.g., CSP, finite state machines) to provide semantics for the system elements and their interactions.  This allows us to do some analysis of a system at the architectural level.

Wright Architectural Description Language

One popular ADL is called Wright (it is 10+ years old, but it looks like much of the hard core ADL work petered out after 2000 or so–no offense to people named Peter). The Wright ADL has a formal semantics based on CSP. Wright’s architectural descriptions are based on describing components, which have ports of interaction and connectors, which have interaction points called roles.  Ports and roles are defined by providing abstract behavioral descriptions–these are basically possible sequences of events that the port or role can participate in.  Wright allows you to specify which events are initiated by a port or role and which events are merely observed.  When putting together a system, connector roles are filled by compatible component ports. In what follows I will give an informal treatment of how to describe an ESMF-like architecture using Wright. If you want to know more about the details of Wright, you should read [2] to familiarize yourself with the language.

Here is an attempt at a Wright description of an ESMF-like architecture:

 Style ESMF

//events prefixed with "do" are initiated by that process (not merely observed)
 Connector ESMF
     Role Driver = doRegister -> doInit -> DriverRunPhase
         Where {
             DriverRunPhase = doRun -> DriverRunPhase |~| doFinalize -> TICK
         }
     Role Coupler = register -> init -> CouplerRunPhase
         Where {
             CouplerRunPhase = run -> CouplerRunPhase [] finalize -> TICK
         }
     Role CompA = register -> init -> CompARunPhase
         Where {
             CompARunPhase = run -> CompARunPhase [] finalize -> TICK
     }
     Role CompB = register -> init -> CompBRunPhase
         Where {
             CompBRunPhase = run -> CompBRunPhase [] finalize -> TICK
     }

     Glue = Driver.doRegister -> RegisterPhase ; Glue []
            Driver.doInit -> InitPhase ; Glue []
            Driver.doRun -> RunPhase ; Glue []
            Driver.doFinalize -> FinalPhase ; Glue []
            TICK
                Where {
                    RegisterPhase = ; c:{Coupler, CompA, CompB} @ c.register -> TICK
                    InitPhase = ; c:{Coupler, CompA, CompB} @ c.init -> TICK
                    RunPhase = ; c:{Coupler, CompA, CompB} @ c.run -> TICK
                    FinalPhase = ; c:{Coupler, CompA, CompB} @ c.finalize -> TICK
                }

Constraints
 // no constraints
End Style

A few things to notice:

• I am describing ESMF architectures as a style (note the first line of the Wright description above). As you might suspect, styles represent an entire class of architectures, not just one specific architecture.
• For better or worse, Wright does not give much guidance on on how to determine what should be a component, what should be a connector, and how to determine which interaction fragments should be called a ports and which should be called a role.  Figuring out how to break down a software artifact into these structures is left to the writer (Wrighter?) of the architectural description.  Since connectors are the locus of interaction of two more components, I have considered ESMF itself a connector (lines 4-32).
• The ESMF connector has four roles: Driver, Coupler, CompA, and CompB.  This represents a coupled model with two components, a coupler in between, and a driver sitting on top.  You might visualize it like this:

ESMF-like architecture with two components, a coupler, and a driver

• Each role is defined by an abstract process description.  Think of this as a series of events that the role participates in.  Some events may be initiated by the role.  (Normally this is shown with an overbar, but here I have prefixed initiated events with “do” such as “doRegister” and “doInit.”)   Notice that the Driver role has four different events that it initiates, doRegister, doInit, doRun, and doFinalize. The way to interpret the Driver role behavioral description is: The Driver first initiates the doRegister event followed by the doInit event and then it behaves like the process DriverRunPhase.  The DriverRunPhase process, then, makes an internal choice (|~|) between two options.  The first option is to initiate the doRun event and then to behave like the DriverRunPhase process (recursive). The second option is to initiate the doFinalize event followed by TICK.  TICK (think of it like a check mark) means successful completion.
• The Coupler, CompA, and CompB roles can be interpreted in a similar way.  A major distinction is that all of their events are observed (i.e., register, init, run, finalize) meaning that each of these roles waits on some other process to initiate the events (in this case the Driver–well, technically the Glue does this, but more on that below).  Also, the Coupler, CompA, and CompB roles all feature an external (deterministic) choice (via the [] operator) meaning than an external process makes the decision  (e.g., such as whether the coupler should run again or should finalize).
• The last part of the ESMF connector description is the process Glue. Every connector has a Glue process which describes how all of the roles work together to form a coherent interaction. The Glue is the most complex of the process descriptions in the ESMF connector, however, I will briefly describe how the “register” interactions are coordinated and the others follow in like manner. The Glue process contains five external-choice alternatives. The first alternative observes the Driver.doRegister event and then behaves like the RegisterPhase process. The RegisterPhase process (described in the Where clause) then initiates a sequence of three events, Coupler.register, CompA.register, and CompB.register.  The ; operator is the sequence operator and, in this case, means that the three events may occur in any order.  This indicates that when the Driver initiates the doRegister event, the Coupler, CompA, and CompB will all respond (in some order) with their own register event. This is an abstract representation of the ESMF component registration process.  Notice on line 22 that after the RegisterPhase process completes, the Glue process makes a recursive call to itself, thereby waiting on the next event to be initiated by the Driver.
• The whole ESMF connector terminates successfully after the Driver initiates the doFinalize event, the Coupler, CompA, and CompB finalize, and then all process synchronize on the TICK event.

Given the quick and dirty ESMF example above, we see some of the benefits of using Wright to describe a climate model architecture:

• A set of distinct roles are identified, made explicit, and their behaviors are described in an abstract way. This gives model builders a precise understanding of the expected behavior of any components fulfilling the roles.
• The use of overbars (or the less preferable doXXX as I have done above) shows which roles are responsible for initiating events and which roles are waiting for events. This is an important distinction as the current set of coupling technologies differ with respect to which parts of the coupled model are driving (controlling) and which parts of the coupled model are passive (waiting to be called).
• The internal vs. external choice operators make it clear who is deciding on the next action.  At the same time, the reason for choosing an action is abstracted.  For example, from this particular description above, we have no idea why the driver would choose doFinalize instead of doRun.  We’d have to look at a different representation to know that.

These are all nice advantages, but the real power of Wright comes into play through its support of some automated architectural analyses.  The formal underpinnings of Wright are based on CSP.  This means that Wright specifications can be translated into CSP and analyzed using CSP processing tools, the most popular of which is FDR2.  Some of the more interesting analyses made possible by this approach are:

• Automated port/role compatibility checking.  Compatibility can be determined using the notion of process refinement.  Informally, a process P is refined by a process Q if Q respects all of P’s obligations to the environment and therefore Q can be safely substituted for P. In other words, we can automatically check which component ports are going to fulfill which connector roles. So, we might use this to answer questions like:  Can I plug in physics package X with atmosphere Y?  Of course, a number of steps have to be taken before we can answer that question automatically. The real work here is in describing the abstract behaviors of the physics process and the atmosphere process in a way that enables the analysis.
• Deadlock freedom.  This is a kind of sanity check that verifies that when all of the components are hooked together into a system, there is a viable path of events that will terminate successfully.  It is possible that this kind of analysis could be used to guarantee that the set of models that participate in a coupled simulation indeed fulfill each other’s data dependencies. Events might correspond to specific field data exchanges and only a non-deadlocking set of processes (models) will mutually satisfy each other’s field requirements.

Architectural Abstraction ≠ Low Resolution

A parting thought.  Computer simulations are scientific tools.  The interactions among components of a coupled model produce hard numbers about the physical processes under study. It is important to remember that the architectural perspective is not a kind of “low resolution” representation of the full model–at least not in the way that scientists think about low resolution. You actually have to run the full code to get low resolution output.  You just change the grid size to meet your resource constraints.  (In some cases, you might choose to ignore some physical processes which would not be resolved by not calling certain subroutines.  But this is an optimization…) Instead, the architectural view is a high level, abstract view of how components interact with much of the details about the internals of the components completely ignored. So, at this time, it does not seem to me that an architectural representation of a coupled model could be used to produce some kind of scientifically meaningful output.  Instead, it can be used to enable other kinds of analyses that gauge properties and qualities of the software system.

[1] Nenad Medvidovic and Richard N. Taylor. “A Classification and Comparison Framework for Software Architecture Description Languages.” IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 26(1), 2000.

[2] Robert J. Allen and David Garlan. “A Formal Basis for Architectural Connection.” ACM Transactions on Software Engineering and Methodology, July 1997.

While diversity at the scientific level is generally seen as a good thing, architectural variability is not necessarily desirable because it impedes interoperability among scientific components. Work has been done in the software engineering community on dealing with architectural mismatch. I have recently applied some of those concepts to climate model components to see what it would take to separate the infrastructure code from the scientific code. In its present state, my approach is heavily inspired by DeLine’s notion of Flexible Packaging [1], although it is already clear that some adaptations must be made for the approach to be viable in a high performance setting. Very briefly, Flexible Packaging is based on the idea of separating a software component’s “ware” from its “packaging.”  Informally, the “ware” is the functionality of the component–what it does.  The “packaging” is the way that the component interacts with the outside world (e.g., as an ActiveX component, a browser plugin, a standalone app, etc.).  DeLine argues that packaging mismatch is a problem that arises because engineering decisions are made too early.  Most software components today are pre-packaged in a certain way and changing the packaging is not feasible in most cases. “Flexible” packaging means that a given ware (piece of functionality) can be packaged in multiple ways, and moreover, that the decision about which packaging to use can be delayed until shortly before the software is deployed.  For more details about this, read the DeLine paper below [1].

Before getting into the details, a few words about motivation.  Why think about climate modeling components in terms of a ware and a packaging?  Here is what I’m thinking:

• The first advantage is the obvious one: interoperability.  This is the primary reason given for DeLine’s Flexible Packaging work.  Interoperability in the climate modeling context means the ability of a diverse set of scientific component to “talk” together because they share the same infrastructure layer.  Architectural mismatch of climate modeling components is an artificial barrier to scientific progress.
• The ware/packaging dichotomy forces a separation of concerns.  That is–all the code in a component needs to go into one of these two boxes.  This is hard because of the intimate connection between the science and the infrastructure code.  For example, the order in which the component models are called is a scientific choice (because it affects the numerics and the output), but the mechanism for actually making the call is an implementation decision.  It is not clear how to separate these two so that decisions about them can be made independently.
• The separation of packaging (which is largely an architectural artifact) from functionality enables studying the impacts of different architectures on software qualities–such as modularity and performance–because the same scientific code can be “outfitted” with different architectures.

What would a flexibly packaged climate model component look like?  A first step is to identify what is the “ware” and what is the “packaging” of a climate modeling component.  One way of thinking about this is:  ware = science code, packaging = coupling infrastructure.

The DeLine work is part of a group of related work in which separation of concerns is achieved via the use of separate processes–that is, each concern is coded in a separate process (which may be implemented as a thread) and the processes communicate when there is a data dependency between them.  In the DeLine work, the “ware” process and the “packaging” process each exhibit “channel signatures” (described using CSP notation) that describe the required communication protocol.  Wares and packagers must have compatible channel signatures in order to be compiled together into a working component.

I want to get right down to some nitty-gritty details to show one way that a flexibly-packaged climate model component might be implemented.  There are two Fortran modules below: simple_ware and simple_packaging.  Taking a look at the ware, you will notice that it is a standard Fortran module with two public subroutines, init() and run().  I have highlighted a few of the lines with subroutine calls that look like fp_out****() and fp_in****().  These calls are communication channels between the ware and the packager–the fp_out****() calls send data and the fp_in****() calls receive data.  These are rudimentary versions of the channel abstraction in the DeLine work.

module simple_ware

    implicit none
    private

    public init, run

    contains

    subroutine init()

        print *, "Inside ware init()"

        call fp_out_int("rank", 2)
        call fp_out_int_array("minIndex", (/1,1/), 2)
        call fp_out_int_array("maxIndex", (/100,150/), 2)
        call fp_out_int_array("regDecomp", (/1,1/), 2)

        print *, "Leaving ware init()"

    end subroutine

    subroutine run()

        real(8) :: pi
        real(8), pointer :: farrayPtr(:,:)
        integer :: i, j
        integer :: arraySize, dim1, dim2

        print *, "Inside ware run()"

        call fp_in_int("arraySize", arraySize)
        call fp_in_int("dim1", dim1)
        call fp_in_int("dim2", dim2)

        allocate(farrayPtr(dim1,dim2))

        pi = 3.14159d0

        ! Fill source Array with data
        do j = lbound(farrayPtr, 2), ubound(farrayPtr, 2)
            do i = lbound(farrayPtr, 1), ubound(farrayPtr, 1)
                farrayPtr(i,j) = 10.0d0 &
                    + 5.0d0 * sin(real(i,8)/100.d0*pi) &
                    + 2.0d0 * sin(real(j,8)/150.d0*pi)
            enddo
        enddo

        call fp_out_double_array("arrayData", farrayPtr, arraySize)

    end subroutine

end module simple_ware

Below is another Fortran module that serves as a packaging for the above ware. This packaging is an ESMF-based packaging. (ESMF is one particular coupling infrastructure).  The packaging has the customary interface for ESMF component, including register, init, run, and finalize subroutines.  The packaging also contains calls to fp_in****() and fp_out****() subroutines for communicating with the ware.

How do the ware and the packaging interact?  First, notice that the packaging references the ware init and run subroutines (e.g., see the use statement on line 4 of the packaging).  Ideally, these references would be automatically added later (by a code generator) so that the packaging does not reference any particular ware.  But for now I have hard coded in a direct link to one particular ware.  Also notice lines 47 and 87 of the packaging have subroutine calls to pthread_create_f(…).  When an external component calls the packaging’s init and run subroutines, the corresponding init and run subroutines from the ware are called in separate threads.  Then, the packaging and ware threads run concurrently synchronizing on the fp_in****() and fp_out****() communication calls.

module simple_pack

    use ESMF_Mod
    use simple_ware, only: ware_init=>init, ware_run=>run

    implicit none
    public init, run, final, register

contains

    subroutine register(comp, rc)
        type(ESMF_GridComp) :: comp
        integer, intent(out) :: rc

        ! Initialize return code
        rc = ESMF_SUCCESS

        print *, "User Comp1 Register starting"

        ! Register the callback routines.
        call ESMF_GridCompSetEntryPoint(comp, ESMF_SETINIT, userRoutine=init, rc=rc)
        call ESMF_GridCompSetEntryPoint(comp, ESMF_SETRUN, userRoutine=run, rc=rc)
        call ESMF_GridCompSetEntryPoint(comp, ESMF_SETFINAL, userRoutine=final, rc=rc)

    end subroutine

    subroutine init(comp, importState, exportState, clock, rc)
        type(ESMF_GridComp) :: comp
        type(ESMF_State) :: importState, exportState
        type(ESMF_Clock) :: clock
        integer, intent(out) :: rc

        type(ESMF_ArraySpec)  :: arrayspec
        type(ESMF_DistGrid)   :: distgrid
        type(ESMF_Array)      :: array
        type(ESMF_VM)         :: vm
        integer               :: petCount

        integer :: rank
        integer, dimension(2) :: minIndex, maxIndex, regDecomp

        integer :: theThread

        print *, "Inside simple_pack init()"

        call fp_init()
        call pthread_create_f(theThread, ware_init)

        ! Determine petCount
        call ESMF_GridCompGet(comp, vm=vm, rc=rc)
        call ESMF_VMGet(vm, petCount=petCount, rc=rc)

        call fp_in_int("rank", rank)
        call fp_in_int_array("minIndex", minIndex, 2)
        call fp_in_int_array("maxIndex", maxIndex, 2)
        call fp_in_int_array("regDecomp", regDecomp, 2)

        call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=rank, rc=rc)
        distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, regDecomp=regDecomp, rc=rc)
        array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, indexflag=ESMF_INDEX_GLOBAL, rc=rc)
        call ESMF_ArraySet(array, name="array data", rc=rc)
        call ESMF_StateAdd(exportState, array, rc=rc)

        rc = ESMF_SUCCESS

        call pthread_join_f(theThread);
        print *, "Leaving pack init()"

    end subroutine

    subroutine run(comp, importState, exportState, clock, rc)
        type(ESMF_GridComp) :: comp
        type(ESMF_State) :: importState, exportState
        type(ESMF_Clock) :: clock
        integer, intent(out) :: rc

        !real(ESMF_KIND_R8)    :: pi
        type(ESMF_Array)      :: array
        real(8), pointer :: farrayPtr(:,:)   ! matching F90 array pointer
        integer               :: i

        integer :: theThread
        integer :: arraySize

        print *, "Inside simple_pack run()"

        call pthread_create_f(theThread, ware_run)

        print *, "User Comp1 Run starting"

        ! Get the source Array from the export State
        call ESMF_StateGet(exportState, "array data", array, rc=rc)

        ! Gain access to actual data via F90 array pointer
        call ESMF_ArrayGet(array, localDe=0, farrayPtr=farrayPtr, rc=rc)

        arraySize = (ubound(farrayPtr, 2) - lbound(farrayPtr, 2) + 1) * (ubound(farrayPtr, 1) - lbound(farrayPtr, 1) + 1)

        call fp_out_int("arraySize", arraySize)
        call fp_out_int("dim1", ubound(farrayPtr, 1) - lbound(farrayPtr, 1) + 1)
        call fp_out_int("dim2", ubound(farrayPtr, 2) - lbound(farrayPtr, 2) + 1)

        call fp_in_double_array("arrayData", farrayPtr, arraySize)

        call pthread_join_f(theThread)
        print *, "User Comp1 Run returning"

    end subroutine

    subroutine final(comp, importState, exportState, clock, rc)
        type(ESMF_GridComp) :: comp
        type(ESMF_State) :: importState, exportState
        type(ESMF_Clock) :: clock
        integer, intent(out) :: rc

        type(ESMF_DistGrid) :: distgrid
        type(ESMF_Array) :: array

        print *, "Inside simple_pack final()"
        rc = ESMF_SUCCESS

        call ESMF_StateGet(exportState, "array data", array, rc=rc)
        call ESMF_ArrayGet(array, distgrid=distgrid, rc=rc)
        call ESMF_ArrayDestroy(array, rc=rc)
        call ESMF_DistGridDestroy(distgrid, rc=rc)

        call fp_final()
    end subroutine

end module simple_pack

Okay, given the example code above, you should notice a few things.

First, notice that there are no references to ESMF anywhere in the “ware” module. The ware is a framework-agnostic module (well, almost–more on that below) that must be paired with a packaging in order to interact with the outside world.  Of course, the output calls from the ware must match with input calls to the packager (and vice-versa).  This is formalized in the DeLine work by doing channel signature matching, but the basic idea is that the ware must provide what the packager needs and vice-versa.

Also notice that the ware is shorter than the packaging.  This is because the example ware doesn’t really do anything that interesting except fill up an array with some dummy data (lines 41-47).  A “real” component would have some actual science code there.

Currently, the fp_in****() and fp_out****() routines are based on data copies.  This introduces a significant overhead if the array sizes are large and/or if the number of iterations is large (both of which are true for a legitimate model).  A smarter implementation would use pointer manipulation such that both the ware and packaging are referencing the same memory addresses.  This is possible since the two are implemented as threads in a shared address space.

Also, remember I mentioned that the ware is “almost” framework-agnostic.  Even though there is no ESMF code inside the ware (e.g., the ESMF_Mod does not have to be imported) there is something a bit more subtle going on.  The issue is that the ware code still has an “init” and “run” subroutine which are derived from the ESMF interface standard.  So, using this ware with a non-ESMF packaging still requires some explicit knowledge about when to invoke the init and run threads in the ware…  It is not clear how to make the ware truly architecture neutral–indeed, the ware will have to adhere to some architecture.  The question is, which architecture is best suited for pairing with different kinds of packagings?

[1] DeLine, Robert. “Avoiding packaging mismatch with flexible packaging.” IEEE Transactions on Software Engineering: 27(2), February 2001.

Designing and implementing coupled Earth System Models (ESMs) is a challenge for climate scientists and software engineers alike. Coupled models incorporate two or more independent numerical models into a single application, allowing for the simulation of complex feedback effects. As ESMs increase in fidelity and sophistication, model developers are increasingly faced with the issue of software complexity. Furthermore, the desire to add more components to ESMs means that the complexity problem will likely become worse unless steps are taken now to improve how independent models are coupled.

Although coupling models is fundamentally a scientific endeavor, a substantial software engineering effort is required to compose the underlying software components into a single application that is at once efficient, numerically correct, and scientifically valid. Much of the difficulty comes from the fact that every coupling is unique—each coupling is intimately dependent on properties of the constituent models that are to be linked together. Therefore, when a model developer begins work on coupling models, there is a tendency to create a context-specific, ad-hoc solution. Unfortunately, the literature describing geophysical coupled models and their challenges says little about how one should go about designing and implementing couplers—the software components used to link together constituent models. The lack of coupler design principles reflects the relatively immature state of the ESM coupling domain and results in duplication of effort and continuous re-learning of the same design principles.

The large amount of legacy code, the wide range of coding conventions, and the breadth of architectural variability among numerical model software means that a completely automated solution to coupling is well beyond the state-of-the-art. However, in lieu of complete automation, a set of rigorous design heuristics can serve as a kind of handbook for building couplers that are likely to meet the desired set of functional and performance requirements. We propose to perform an architectural analysis of the state-of-the-art in ESM couplers and encode the expert knowledge found therein into a resource for coupled model developers.

To achieve this goal, we first identify a coupler design space, a multidimensional space of functional and structural design choices relevant for devising and constructing ESM couplers. Formulating the design space itself is a non-trivial task due to the range of different coupling philosophies and architectures currently employed in production quality ESMs. Furthermore, the design space dimensions reflect the design choices of primary importance when building couplers. Therefore, there is increased pressure to identify the “right” dimensions in order to provide the best possible guidance in coupler design. A point in the design space represents a particular instance of a coupler design.

Secondly, we perform an architectural tradeoff analysis of ESM couplers to assess software qualities of couplers within the design space. Although architectural analyses can be used to evaluate a wide range of software qualities, this work focuses on two fundamental qualities that are often antagonistic: performance and modularity. High performance translates to higher resolution simulations, increased capacity for multi-model ensemble runs, and decreased time to dataset analyses and publication of scientific results. Modularity is a proxy for other software qualities that are hard to measure directly, such as modifiability, flexibility, reusability, and intentionality. Although modularity in software systems is linked to a number of advantages, there are performance costs involved with modular designs due to potential efficiency loses at module interfaces. The antagonistic relationship between modularity and performance has been recognized as a general principle in many kinds of engineering systems and it is a fundamental tradeoff that must be considered when designing coupler architecture.

Thirdly, the results of the architectural analysis form the foundation of a set of coupler design guidelines. The design guidelines are an extrapolation from the architectural tradeoff analysis that can be used by model developers to predict the effects of making architectural choices when designing ESM couplers. They are presented in a practical and problem-oriented form that can be applied quickly by model developers who are faced with a particular coupling problem.

The design guidelines are evaluated by encoding them in a code generator capable of automatically generating ESM couplers with desired functional and non-functional properties. Generated couplers can be fed back into the architectural analysis allowing for iterative refinement of the design guidelines themselves. On a practical level, the automated coupler generator serves as a valuable tool for the geophysical modeling community by reducing development costs and time-to-solution for building couplers.

Here’s a thought experiment:  How do we separate the science code from the infrastructure code?  Why should we do this? Well, because we want scientists to focus as much as possible on their field of expertise, not on the accidental aspects of writing code. Can this really be done?  Maybe, maybe not.

But for now, let’s assume we could distinguish the infrastructure from the science and actually separate them to a large degree.  One way to approach this is to generate the infrastructure and let the scientists write the rest.  There are at least two ways to think about doing this:

1. Write “infrastructure-neutral” code using only plain programming language constructs and wrapping and/or adapting that code to whatever infrastructure is desired.  This approach is appealing because it minimizes the software dependencies of the scientific code–ideally, the only dependency would be a standard compiler.  (Note the contrast with the SIDL/Babel work which attempts to divorce the science from the programming language.) While this approach does not automatically guarantee interoperability, it does at least ensure that two potentially coupled models do not have incompatible dependencies (e.g., one uses coupling framework A, another uses coupling framework B and there are no readily available adapters that translate between A and B).   A pithy, but not entirely accurate statement made an the recent coupling workshop sums it up this way: “the programming language is the coupling framework.”  External metadata plays an important role here because the code generator needs to know how to interact with the existing code–that is, how the infrastructure should be woven in around the science code.
2. Write code using high level abstractions that can be mapped onto specific infrastructure implementations.  The high level abstractions could be formalized in a domain specific language.  The language constructs must be general enough to apply to all target infrastructure implementations.  (Could Jay Larson’s paper on principles for coupling be a start for defining the set of constructs?)

A major distinction between the above two approaches is what is formalized.  For the “infrastructure-neutral” approach, the external configuration metadata must be formalized.  For the “high level abstractions” approach, the abstractions themselves must be formalized.  At some point the two actually blur together.  As pointed out in my previous post, the first approach corresponds largely with the philosophy behind BFG.  The second approach, to my knowledge, has not been tried.

Kinds of Infrastructure Code

It is helpful to divide up what kinds of code might be considered “infrastructure.”  A high level breakdown is the Superstructure/Infrastructure distinction from ESMF and FMS.  In this case the term “Infrastructure” takes on a more specific meaning than how I have been using it previously, so I’ll use a capital “I” when speaking of this more specific kind of infrastructure.

The Superstructure represents the larger architectural pieces that are incorporated into a coupled ESM.   These are the high level structures that we expect the ESM users and developers to refer to.  These are the structures that have names, are connected together, and have interfaces.  This is closely related to the software architecture, although it will be helpful to think of the architectural pieces as having some domain-level semantics, not just arbitrary software modules.  We want the Superstructure to have high visibility.

The Infrastructure represents capabilities commonly required when building coupled ESMs.  This includes things like grid-to-grid interpolation, parallel domain decomposition, and time management.  Interestingly, these represent things that we want to to have low visibility–that is, we want them to be there working in the background, providing utility functionality, but they should not be too prominent or consume a substantial amount of development time.  To be sure, they are highly important, but they are a distraction from the real work.

What is the relationship between the two?  I think it’s a hard question to answer, but here’s one take.  The Superstructure can be viewed as an information-broker for the Infrastructure.  In other words, the Superstructure provides the information that the Infrastructure needs to do its job.  Let me be more concrete here by giving an example.  One of the primary functions of ESM couplers is to communicate data from one model to another.  Of course this may be handled in a number of different ways (memory-to-memory copies, message passing, argument passing).  The data communication utility is part of the Infrastructure.  However, this function must first know something about where the data is coming from and where it is going.  Where the source data is located and how it is encapsulated is part of the Superstructure, as is the location of the target and how the delivered data should be encapsulated.

How is information communicated from the Superstructure to the Infrastructure?  Either the Infrastructure queries the Superstructure (pull) or the information is provided to the Infrastructure (push) when the Infrastructure is invoked. If the Superstructure and Infrastructure are part of the same package, then the relationship between the two can be hard coded into a reusable framework (i.e., the Infrastructure can assume that a certain Superstructure is there).   In this case, the Infrastructure can pull architectural information from the Superstructure.  On the other hand, if the Superstructure is separate from the Infrastructure (e.g., because the code has been architected separately), then the knowledge required by the Infrastructure must be parameterized and passed (pushed) to the Infrastructure.

To generalize the notion of Superstructure, we can identify a coupling technology as either an architecture provider or architecture neutral.  Architecture providers inform and/or constrain high-level structural aspects of the coupled model. These technologies are an example of architectural reuse. How does this reuse occur? A common method is for the coupling technology to provide a set of abstract classes with interfaces that the user implements. The architecture, therefore, is encoded into the abstract classes and their predetermined interactions (this is the basic idea behind object-oriented frameworks).

Architecture neutral coupling technologies say little about what the high-level components should be and what kinds of interfaces they should have.  This means that the coupling technology cannot make architectural assumptions about the constituent models, but must instead be informed of relevant architectural characteristics using some external mechanism.

Another way to break down infrastructure code is by what requirement or capability the infrastructure code fulfills.  The coupling technologies feature model communicates the kinds of features that coupling infrastructures support.  (Arguably, “infrastructure” is much broader than “coupling” although at the current time there is a blurring of the two–for example, the coupling technology may provide a haloing operation which is used within the bounds of a single model.  It is not a “coupling” task per se, but handling the halo operation here is natural because some of the same abstractions used for data transfer can also be exploited for haloing.)  Using the feature model as a guide, we can see several kinds of infrastructure capabilities:

• Data transfer — via message passing, subroutine arguments, shared memory, etc.
• Interpolation (regridding) — these all have to do with the fact that models often have different grids.  This includes weight generation and global conservation.
• Accumulation and averaging in time — because coupling frequency differs from model timesteps
• Domain discretization — breaking up the physical domain into grid cells
• Domain decomposition — distributing those grid cells onto computing resources
• Driving — moving the whole coupled model forward in time, scheduling execution order, handling concurrency
• Setup and configuration — preparing for a run
• I/O — generally considered separate from the coupling technology, but clearly a part of the infrastructure

What is involved in generating these parts of the infrastructure?  To answer this, we must first ask how each of these are manifest in a coupled model.  It would also nice to know which ones are dependent on which other ones.  How localized is the capability (e.g., can the capability be implemented in one location in the code, or is it dispersed throughout the code)?  What is the relationship of the science code to each of these?  If that relationship is deep, then it will be harder to tease out the infrastructure.  For example, scientific constraints may determine the sequencing of the components.  But, since defining the sequence is a common requirement among all coupled models, we’d like to pull out as much of it as possible (perhaps representing it declaratively) while still respecting that the sequence is determined by the science–it’s not an arbitrary technical decision.  Domain discretization is also a common requirement in coupled models, so we’d like to offload it as much as possible to reduce the number of custom implementations.  However, the field calculations at each timestep are completely dependent on the discretization method.  Herein we see the tightly coupled nature of the infrastructure and science parts of the code.

De-coupling

To get a better handle on how to de-couple the infrastructure from the science, I’d like to do an analysis of how these various infrastructure pieces are represented in real models and characterize the relationship between the infrastructure and the science parts of the code.  This is a beast of a task, frankly, because the models are so large and complex.  I’m not sure if the analysis could be automated in any way.  However, the results could be very interesting and might open the door to better separation of concerns and improved composition of coupled models.