Honest disagreement is often a good sign of progress.

-- Mahatma Gandhi

Appendix A

Examples of Multiple Representations

Multi-model applications in a number of domains maintain multiple representations or views with some degree of concurrence and consistency. In §2.1, we presented these applications briefly and evaluated whether they satisfy the MRM requirements R1, R2 and R3 (Table 1). Here, we evaluate these applications in detail. Briefly, we discuss how an approach based on MREs may benefit these applications.

A.1 Multi-Resolution Graphical Modelling

Multi-resolution graphical modelling involves maintaining multiple representations, or levels of detail , of the same object [Clark76]. For example, a lamp may be rendered in full detail when a viewer is close to it, but as the viewer moves away, successively coarser levels of detail are rendered. As the viewing distance increases, the lamp occupies a smaller portion of the viewed screen space, and coarser levels of detail for the lamp are sufficient to cover this portion. The coarser the level of detail, the fewer the polygons required to render it. The system always maintains multiple levels of detail for all objects, and selects the appropriate level of detail depending on an object's distance from the viewer. The challenges in graphical MRM are to generate the multiple representations such that each captures sufficient detail as to be visually appealing, and to transition among representations smoothly [Gar95] [Heck94] [Heck97] [Luebke97] [Puppo97].

Typically, users cannot change multi-resolution graphical entities, although they may issue view interactions , which essentially read the values of attributes such as position and colour. Moreover, concurrent interactions to multiple levels of detail of the same object are not supported. Since interactions cannot change entities' representations and cannot be concurrent, R1 is violated. R2 is satisfied trivially after the levels of detail are created because the representations do not change. A few multi-resolution graphical models permit a single user to change entities dynamically, but do not support concurrent multi-representation interaction [Berm94] [Lee98] [Zorin97]. Multi-resolution graphical models satisfy R3 because multiple levels of detail reduce simulation cost. As long as interactions cannot change the representations of objects, consistency cost is not an issue.

An MRE for a multi-resolution graphical model would incorporate all levels of detail. Designers can create the multiple levels of detail using refinement or simplification [Heck97] [Luebke97]. Refinement and simplification methods can be the mapping functions among the multiple levels of detail. When changes to any levels of detail occur, these mapping functions can ensure that the other levels of detail are changed so as to keep the MRE consistent. Consequently, an MRE for a graphical object may interact concurrently and consistently at multiple representation levels.

A.2 Hierarchical Autonomous Agents

An autonomous agent is an actual or simulated robot that attempts to fulfill a goal by performing actions from its basic skill set. Traditionally, there have been two approaches regarding the manner in which an agent fulfills its goal. In the deliberative approach, an agent constructs a plan to fulfill its goal by composing actions from its skill set before beginning any action [Sacer74]. The agent may form optimal or provably correct plans; however, unexpected occurrences can sabotage any plan easily. In the reactive approach, an agent forms no plan at all, relying on reactions to external stimuli to fulfill its goal [Agre87]. This approach leads to extremely robust behaviour in the presence of urgent or unexpected circumstances; however, the agent may become trapped in local minima.

Multi-layered architectures for autonomous agents incorporate a deliberative layer (a planner) and a reactive layer (a perception-action or PA layer) with some intermediate layers. Multi-layered architectures balance varying requirements and capabilities of different layers, e.g., level of abstraction, amount of inference, time-scale and bandwidth [Albus97] [Bon97] [Firby87] [Gat92] [Laird91] [Hanks90] [Sim94] [Was98a].

Multi-layered, or hierarchical, autonomous agents satisfy R1. Such agents execute deliberative and reactive models jointly in order to take advantage of both. The planner and PA layer representations1 are linked epistemologically, i.e., subsets of representations encode knowledge that depends on or is derived from knowledge encoded in other subsets [Brill96]. Hierarchical agents do not satisfy R2 because dependencies among planner and PA layer representations can give rise to inconsistencies. For some desired agent behaviour, the paradigm of executing both models jointly is more cost-effective than executing only one model. Hence hierarchical agents satisfy R3.

Provided designers can agree on what must be represented at each layer of hierarchical autonomous agents, an MRE for such agents would incorporate the representation for each layer. Typically, we can capture dependencies between the representations by simple relationships, such as has-part and is-a [Was98b]. By ensuring that the individual relationships hold, we can maintain consistency between the representations.

A.3 Blackboard Systems

Hearsay-II is a layered system for translating spoken sentences into the corresponding alphabetic representation. In Hearsay-II , many processes access a single data structure, called a blackboard [Erman80]. Processes are data-driven , i.e., a process activates itself whenever appropriate data appears on the blackboard. The lowermost layer of the system interprets parts of sound waves as silence or non-silence. The next layer interprets non-silence as phonemes and predicts the sound wave corresponding to the next likely phoneme. The next layer composes phonemes into syllables and predicts the next word. The hierarchy of layers continues with the topmost layer composing phrases into sentences and predicting the next phrase.

Hearsay-II 's blackboard is a multi-representation system; each layer is a different model of the entire spoken sentence. R1 is satisfied because for each sentence fragment, the interpretation of the current layer and the prediction of the layer above are multi-representation interactions. Hearsay-II resolves conflicting interactions -- different predictions or interpretations of the same sentence fragment -- by retaining each as a version of the sentence. Each version is consistent -- the wavelets are consistent with the phonemes, the phonemes with the syllables, and so on -- thus satisfying R2. The system ranks all versions by a credibility metric; the version with the highest credibility is the best translation of the sentence. However, retaining all versions may be impractical in a general sense, since many multi-representation systems may not tolerate multiple outcomes.

Each version of a sentence in Hearsay-II is similar to an MRE. However, Hearsay-II violates R3 because it resolves conflicting interactions by creating new MREs that subsequently execute concurrently with existing MREs. In effect, each MRE executes in a "parallel universe" in which it is the most credible version. The greater the number of versions, the greater the number of MREs in execution, putting a strain on available resources. Our technique of resolving concurrent interactions within a single MRE may miss the best possible version of a sentence when local minima occur. However, when many objects or processes are present in a system, Hearsay-II 's technique of creating a new MRE for every possible outcome of conflicting concurrent interactions can cause a combinatorial increase in consumption of resources.

A.4 Cache Coherence

In a multi-processor configuration, individual processors may access a small amount of fast memory locally in order to reduce accesses to main memory, which tends to be slow. The fast memory, called a cache , may store copies of data items stored in main memory. Processors may read and modify data items in their caches. Ensuring that processors read correct versions of the data in their caches is known as the cache coherence problem [Henn96].

Cache coherence is a form of the MRM problem. The main memory copy and each cache copy of a single data item are concurrent representations of a variable. Processes issue interactions in the form of read and write operations to the copies. Processors may interact with cache copies as well as the main memory copy, the latter when a processor's cache copy is absent or stale or, in the case of write-through policies, whenever the processor writes to the variable. Concurrent interactions at multiple representation levels are assumed to have independent effects. Since multi-representation interactions may occur, but dependent interactions are not supported, R1 is satisfied partially. Cache coherence involves combining detection mechanisms such as snoopy bus or directory-based protocols with write policies such as write-back and write-through to maintain consistency among cache and memory copies. Although cache coherence solutions maintain consistency, typically, the relationships among memory and cache copies are simple relationships of equality. Therefore, cache coherence satisfies R2 partially. Various cache coherence strategies have different costs associated with them [Arch86]. However, accessing caches is more cost-effective than accessing memory. Hence caches satisfy R3.

We do not forward any new solutions for cache coherence. Rather, we use the cache coherence example to highlight the benefits of maintaining concurrent representations.

A.5 Abstract Data Types and Object Inheritance

Polymorphic languages may associate multiple types for a single data item. For any data item, the operations that are valid on it, the contexts in which it can be used legally and the manner in which it is allocated memory are determined by its type. If a data item has multiple types, the operations valid on it and the contexts in which it can be used is the union of the operations and contexts respectively for the individual types. Typically, the memory allocated to the data item is such that the data item has a single representation.

Some abstract data types present multiple views of the same data item, thus exhibiting a form of MRM. A data item defined as a union in C [Kern88] and C++ [Strou91] or as perspectives [Gold80] [Stefik86] can have multiple types, thus displaying ad hoc polymorphism [Card85]. Consider the definition of a union in C:

union {

int a;

char b;

} X;

The data item X has two types, int and char , corresponding to X.a . and X.b respectively. X.a and X.b are different views of X . They occupy overlapping bytes of memory, i.e., if an int is stored as two bytes on a particular system and a char is stored as one byte, then X is allocated two bytes of memory. One byte holds the value of X.b as well as part of the value of X.a , while the other byte holds the remaining part of the value of X.a . X.a and X.b are accessed jointly by any operation accessing one of them, i.e., if an operation changes the value of X.b , it changes the value of X.a as well and vice versa . A type is a representation level; therefore, operations of different types constitute multi-representation interactions. However, these interactions are assumed to be independent of one another. Therefore, R1 is satisfied partially. Changing the value of any type automatically changes the value of other types for a data item. However, unions cannot capture general relationships, such as those among attributes of an MRE. For example, a union cannot present two views such that a value in one view is an accumulation of values in another view. Hence, ad hoc polymorphism satisfies R2 partially.

Inheritance in object-oriented languages is an example of MRM, since a data item that inherits from one or more types has multiple views. Object-oriented languages such as Smalltalk-80 [Born82], Simula-67 [Dahl66] [Birt73] and C++ [Strou91] support inclusion polymorphism [Card85]. Consider the C++-like example below:

class Mammal { ... }

class Oviparous { ... }

class Platypus: Mammal, Oviparous { ... } Bill;

Here, Mammal and Oviparous are base classes for the derived class Platypus . Bill is an instance of the class Platypus , and by inheritance, also an instance of the classes Mammal and Oviparous . Inheritance results in Bill having multiple views: one, as an instance of a base class and two, as an instance of a derived class. One view is subsumed by another; the view as Mammal is a subset of the view as Platypus . Multiple inheritance results in Bill having even more views. However, the views do not subsume each other; the view of Bill as a Mammal has no relation to the view of Bill as an Oviparous . The representation for Bill is the union of the representations defined by each of the above classes, assuming name conflicts are resolved. Likewise, the set of methods applicable to Bill is the union of the set of methods defined by each class. A class is a representation level; therefore, methods of the multiple classes constitute multi-representation interactions. However, these interactions are assumed to be independent of one another. Therefore, R1 is satisfied partially. Any operation that is performed on an instance of a derived class is performed on an instance of the base class as well. Therefore, the instance of the derived class is always self-consistent. However, inheritance is only one kind of relationship among attributes of an MRE; for example, inheritance does not capture the accumulation relationship mentioned earlier. Hence, inclusion polymorphism satisfies R2 partially.

A.6 Views in Databases and Integrated Environments

Views, as defined for databases and integrated environments, are a form of MRM. A view in a database is a subset of the information contained in the system.

Database views are derived from the complete database by specifying relations that restrict the items displayed. In relational database applications, data are abstracted into relations, which essentially are tables of tuples and their values [Codd70]. Relational databases have been used for many applications [Astra76] [Stone76] and programming environments [Linton84]. In object-oriented databases, data are abstracted as behavioral entity relationships [Chen76] [Balzer85]. Hybrid approaches that maintain relations as well as attribute relationships have been used for editing programs [Hor86]. A view is a set of relations derived from existing relationships (in an object-oriented database) or relations (in a relational database) [Cham75]. Changes to a view must be translated to changes to the database in order to maintain consistency in the database [Ban81]. Since all views are derived from one database, this approach is a form of selective viewing, which violates R3. Each view is constructed after the entire database has been constructed. Users may update data in any view; however, all updates are assumed to be independent. Hence, views in databases satisfy R1 partially. When users update data in a view, the system updates the database automatically, thus maintaining consistency and satisfying R2. Views in databases require the database to be the repository of all possible views, thus making them unsuitable for MREs, wherein multiple representations may have been designed independently. Moreover, relations are powerful but not intuitive for some kinds of relationships [Hor86].

In some integrated environments, the complete database is constructed by conjoining all the views [Gar87]. Individual tools may maintain views of their data. Users may update data in any view as well as in the database; however, all updates are assumed to be independent. Hence, views in databases satisfy R1 partially. Users' updates are interactions. The system updates the views and databases to remain consistent with one another, thus satisfying R2. In database views, if each view is an independently-designed model, then requiring a single database for all the models put together violates R3. In contrast, in integrated environments, if each view is an independently-designed model, then the complete database is just the conjoining of the models, which satisfies R3. The latter approach is related closely to MREs. Each view may be considered a concurrent representation and the conjoining of all the views is the MRE. Garlan's work envisioned the multiple views to be used by tools that change databases. In MREs, other entities, other models and the environment change a representation.

A.7 Nested Climate Modelling

An increasingly popular approach to climate modelling is to nest the execution of Limited Area Models (LAMs), which predict regional climate, within Global Circulation Models (GCMs), which predict wide-ranging climate changes [Giorgi90] [Giorgi91] [Risbey96]. GCMs model synoptic or large-scale climate changes. The resolution of these models is usually in the hundreds of kilometres, which means that regional climatic variations are modelled poorly. LAMs model mesoscale or medium-scale climate changes. The resolution of these models is in kilometres, hence they model local climate well but at a huge computational cost. Of late, small sub-areas of the larger area modelled by the GCM are taken over by LAMs which discard all the GCM modelling information except at the edges of the sub-area modelled by each LAM. Subsequently, each LAM runs its own computations to predict local climates. The GCM-LAM linkage produces more accurate predictions than either a GCM alone (since the LAM usually has more detailed topographical and orographical information) or just a LAM driven by empirical data (which assumes that future climate will be very similar to past climate).

Nested climate models satisfy R1 but not R2. The nested models interact at multiple representation levels since climatic data at either level is incorporated. However, while researchers have had success translating GCM data for LAM input, the reverse is an open problem. As a consequence, global factors such as temperature fronts, monsoons and large mountain ranges can influence local climate models, but it is extremely hard to make local factors such as fires, nuclear waste build-up, small mountain ranges and anthropogenic pollution influence global climate models. Nested climate models satisfy R3 because they are more cost-effective than executing any one model individually.

A climate model MRE would incorporate GCM and LAM representations for a particular area. As a result, the climate of the area would be influenced by global factors as well as local factors. Nesting the LAM within the GCM would be one way to reconcile concurrent climate changes. However, as discussed earlier, this tends to make the execution of the LAM dominate the execution of the GCM, particularly close to the center of the area modelled by the LAM. In terms of accuracy of predictions, the MRE approach can do no worse than nesting; the potential to do better lies in the ability of the MRE approach to capture dependencies between the two representations that are ignored in nesting. However, the limiting problem in either approach is the lack of techniques to translate local factors into global factors.

A.8 Integrated Molecular Modelling

When theoretical studies on the potential energy surfaces for chemical reactions of large systems are carried out, low-detail low-computation models, such as molecular mechanics (MM) models, are used for most of the system and high-detail high-computation models, such as molecular orbital (MO) methods, are used for a small part of the system. An MM model for the entire system is usually fast but inaccurate since the level of detail does not capture all interactions among atoms. An MO model for the entire system is accurate but computationally expensive. Integrated models such as IMOMM [Matsu96] and ONIOM [Sven96a] strike a balance between resource usage and accuracy. These approaches integrate MM models, such as MM2 , MM3 , CHARMM , AMBER and UFF , with MO models such as Møller-Plasset second-order perturbation ( MP2 ) and Hartree-Fock ( HF ), in order to compute potential energy surfaces. Some approaches, for example IMOMO , integrate an high-detail MO model with a low-detail MO model [Humbel96] [Sven96b]. In all the integrations, the reported accuracy is comparable to a full-scale high-detail model, while resource usage is markedly below such a model.

Integrated molecular models satisfy R2 and R3. The models incorporate interactions at multiple levels of detail and are remarkably consistent. Also, reported costs are lower than running a detailed model for the entire system. However, these models satisfy R1 partially because the multiple models are executed one after another. Therefore, multi-representation interactions are assumed to be independent of one another.

The integrated models for the reactions under study are MREs. Although experts in molecular modelling strive for better correlation between the MM and MO models, the high level of consistency already achieved suggests that the integrated approach is very well-suited for applications involving models at different representation levels.

A.9 Multi-Level Computer Games

A number of commercial computer games present a player with multiple views of the world inhabited by the characters controlled by the player. In games2 like Civilization , WarCraft , SimCity , Doom , Heretic , Hexen , Quake  and Duke Nukem , a player may view the playing area at multiple levels of resolution. In some games, the player may control characters at any resolution, while in others, the player may control characters only at the highest resolution level, with the game pausing when the player switches to a lower resolution level.

Multi-level games satisfy R2 but not R1. Merely displaying information at multiple resolutions amounts to processing read interactions that do not change the representations. Even games that permit changes to be made at either representation rarely permit concurrent changes, or concurrent interactions, thereby completely avoiding the hardest problem in MRM. Most games adopt the approach of selective viewing, wherein all processing takes place at the highest resolution level. The player may request high-resolution information or may ask for low-resolution information. In the latter case, high-resolution information is aggregated and presented as low-resolution information. Selective viewing violates R3.

MREs for entities within such games would incorporate the representation at each resolution level. Players could be permitted to interact at any resolution level, and in the case of multi-player games, at multiple resolutions concurrently. Mapping functions that translate changes to one resolution level to changes to other resolution levels will keep the multiple resolution levels consistent.

A.10 Battlefield Simulations

In the domain of battlefield simulations that are used for training as well as analysis, MRM relates to resolving conceptual and representational differences arising from multiple levels of resolution in simulations that are joined for a common objective, particularly where the simulations were designed and implemented independently. The crux of the problem can be appreciated by considering what is required to simulate accurately an object and its constituents concurrently. For example, the abstraction convoy may have attributes such as position, velocity, orientation and state of repair. At a more detailed level, the convoy may be viewed as trucks that have attributes such as position, velocity, orientation, state of repair, fuel level, gross weight, carrying capacity and number of occupants. If the convoy abstraction and its constituent trucks are modelled concurrently, all interactions with the convoy abstraction and its constituents in overlapping periods of time must be reflected accurately at both levels.

Many battlefield simulations satisfy none of R1, R2 or R3 fully. Typically, battlefield simulations employ aggregation-disaggregation to force entities to interact at the same resolution. Aggregation-disaggregation can preclude concurrent multi-representation interaction, can give rise to inconsistencies, and incur high resource costs. MREs for battlefield simulations would incorporate multiple representations of the same object. Typically, the object would be a hierarchical unit such as a corps, division or platoon.


1. Although Brooks argues against representation in an agent [Brooks86], Brill has shown that agents with representation can be effective [Brill98].

2. Civilization is a registered trademark of Sid Meier games. WarCraft is a registered trademark of Blizzard. SimCity is a registered trademark of Maxis. Doom , Heretic , Hexen and Quake are registered trademarks of id Software. Duke Nukem is a registered trademark of 3D Realms Entertainment.