The purpose of computing is insight, not numbers.

-- Richard Hamming

Chapter 9

Evaluation

Our framework, UNIFY , is a sufficient and practical approach for effective MRM. UNIFY is sufficient because it satisfies three requirements for MRM: multi-representation interaction (R1), multi-representation consistency (R2) and cost-effectiveness (R3). We described these requirements in §1.3 and §3.4. UNIFY is practical because it offers techniques and processes for designing a multi-model. Designers can apply UNIFY in conjunction with a model specification methodology such as OMT to construct effective multi-models. In §9.1, we evaluate UNIFY in terms of the MRM requirements. In §9.2, we discuss briefly how UNIFY can be applied to existing applications to achieve effective MRM. In §9.3, we present limitations of our work.

9.1 Evaluating UNIFY in terms of MRM Requirements

We evaluate UNIFY with regard to our three sufficiency requirements R1, R2 and R3: multi-representation interaction, multi-representation consistency and cost-effectiveness. Since the joint execution of multiple models is intended to capture their combined semantics, an MRM approach must permit the execution of the individual models. Therefore, the MRM approach must permit entities at all representation levels to interact. An MRM approach must maintain consistency among the representations of jointly-executing models. If the representations of jointly-executing models are consistent, the behaviours of the models can be consistent, thus leading to effectiveness of the multi-model. Consistency can be maintained among multiple representations by propagating changes from one representation to another. Lastly, an MRM approach must keep simulation and consistency costs low. We reiterate the definitions of R1, R2 and R3 here.

UNIFY satisfies these requirements. In the following sub-sections we evaluate UNIFY and alternative MRM approaches such as selective viewing and aggregation-disaggregation in terms of these requirements.

9.1.1 Multi-Representation Interaction

UNIFY satisfies R1 by permitting interactions to occur at all representation levels. Let ModelM be a multi-model constructed from low-resolution model, ModelA , and a high-resolution model, ModelB . Recall from Chapter 3 that .

Alternative approaches, such as selective viewing and aggregation-disaggregation, do not satisfy R1. In selective viewing, interactions at only the most detailed representation level are permitted. In other words, in selective viewing, IntM = IntB at all times. Therefore, selective viewing does not satisfy R1. In aggregation-disaggregation, interactions at different representation levels are permitted, but at only one level at a given time. In other words, at time ti TM , IntM ( ti ) = IntA ( ti ) but at some time tj TM , tj ti , IntM ( tj ) = IntB ( tj ). Since at any given time, interactions at only one level are permitted, aggregation-disaggregation does not satisfy R1.

In contrast with selective viewing and aggregation-disaggregation, UNIFY permits concurrent interactions at multiple representation levels. In UNIFY , IntM = IntA IntB . Since interactions at all representation levels can occur at all times, UNIFY satisfies R1.

9.1.2 Multi-Representation Consistency

UNIFY satisfies R2 by maintaining consistency among jointly-executing models. A Consistency Enforcer (CE) maintains consistency among the concurrent representations within an MRE. A CE propagates a change caused by an interaction to all dependent attributes. A CE consists of an Attribute Dependency Graph (ADG) and application-specific mapping functions. An ADG captures dependencies among attributes at different representation levels. Mapping functions translate changes to attributes before the next observation time occurs. Consequently, an MRE is always internally consistent.

In alternative MRM approaches, such as aggregation-disaggregation and selective viewing, multi-representation consistency is not satisfied because cross-model relationships do not hold at all times. For a valid model, , Rel must hold at all observed times. For a multi-model, ModelM = ModelA ModelB , RelM = RelA RelB Relcross-model . If the models, ModelA and ModelB , are not related to one another, Relcross-model = ∅, i.e., cross-model relationships are null. In such a case, cross-model relationships hold at all observation times for any approach, including UNIFY . However, for typical jointly-executing models, Relcross-model ≠ ∅. Selective viewing forces Relcross-model to be null, since only one representation level exists. Likewise, aggregation-disaggregation forces Relcross-model to be null except during transitions from one representation level to another. Forcing cross-model relationships to be null ensures that they hold trivially, but does not capture relationships among jointly-executing models at all observed times. Therefore, selective viewing and aggregation-disaggregation satisfy R2 partially only.

In UNIFY , Relcross-model holds at all observation times. ADGs and mapping functions capture Relcross-model completely. A CE, which consists of an ADG and mapping functions, ensures that changes to attributes of an MRE propagate to all dependent attributes before the next observation time. Consequently, no two entities can receive inconsistent views of an MRE at overlapping simulation times. Therefore, an MRE exhibits temporal consistency. Mapping functions ensure that attributes in an MRE do not change in a manner inconsistent with model requirements. As a result, the MRE exhibits mapping consistency. Since an MRE is always internally consistent, UNIFY satisfies R2.

9.1.3 Cost-Effectiveness

UNIFY satisfies R3 by reducing the total cost of executing a model. A sufficient approach to MRM must achieve multi-representation interaction and multi-representation cost-effectively. Simulation cost is the cost of executing multiple models. Consistency cost is the cost of maintaining consistency among concurrent representations. Together, simulation and consistency costs constitute the total cost of executing a model. Simulation and consistency costs can be translated to resource consumption costs. For example, simulation cost can be translated to the amount of processing required to apply the primary effects of interactions. In other words, when an interaction occurs, the processing required to change the values of attributes affected directly by the interaction is a simulation cost. Likewise, consistency cost can be translated to the processing cost incurred in order to keep entities consistent. In other words, when an interaction occurs, the processing required to apply the secondary effects of the interaction is a consistency cost. Simulation and consistency costs tend to be trade-offs, i.e., an approach with low simulation cost tends to have high consistency cost and vice versa . UNIFY enables reducing the two costs, i.e., their sum is lower when UNIFY rather than aggregation-disaggregation or selective viewing is the MRM approach. UNIFY satisfies R3 by reducing simulation and consistency costs.

We compare simulation and consistency costs for selective viewing, aggregation-disaggregation and UNIFY . It is hard, if not impossible, to change the MRM approach dynamically for an application in order to measure costs fairly. Hence, we construct a synthetic application for which we can change the MRM approach. We present the assumptions we make in our cost comparison.

9.1.3.1 Assumptions

The semantics of our synthetic application are unimportant; we merely count simulation and consistency actions undertaken by the application. Each action reflects a processing or communication operation with an associated application-specific resource cost. For a fair comparison, each approach should permit interactions at all levels. However, aggregation-disaggregation and selective viewing do not permit interactions at non-simulated levels, whereas UNIFY permits interactions at all levels. Accordingly, we compare UNIFY with hypothetical variants of aggregation-disaggregation and selective viewing that permit interactions at non-simulated levels. Comparing UNIFY with these variants does not bias our cost analyses because the variants have the same remaining characteristics as their corresponding original approaches.

In our hypothetical aggregation-disaggregation approach (AD), an entity is simulated at its lowest resolution or most aggregate level. As long as interactions occur at this level, the entity is represented at this level alone. However, when an interaction at a higher resolution occurs, the entity is disaggregated into sub-entities at the level of the interaction. After the effects of the interaction have been applied to the appropriate sub-entity, all sub-entities are aggregated back to the lowest resolution. AD can be improved; partial disaggregation and pseudo-disaggregation are improvements over AD (see §2.2.2). However, as it stands, AD captures the essence of the aggregation-disaggregation approach. AD has low simulation cost since only a few entities are simulated.

In our hypothetical selective viewing approach (SV), an entity is simulated at the highest resolution level. The entity is disaggregated initially into its sub-entities at the highest resolution. Each sub-entity exists throughout the duration of the simulation. When lower-resolution interactions occur, they are translated into their highest-resolution equivalents. If a low-level interaction affects a single low-resolution entity, we translate the interaction to many high-resolution interactions that affect a corresponding number of high-resolution entities. SV has low consistency cost since only one level is simulated.

In UNIFY , an MRE is constructed for an entity at multiple resolution levels. In our synthetic application, we maintain attributes at all resolution levels at all times. The effects of an interaction are applied at the appropriate resolution level and propagated to other resolution levels. Computing simulation costs for an MRE simulated at all resolution levels would bias our analysis against UNIFY unfairly. An MRE simulated at all levels permits concurrent interactions at different levels, which none of AD, SV, aggregation-disaggregation and selective viewing permit. Therefore, for our analyses, we simulate an MRE at any one of its levels at a given time. Simulating the lowest resolution level would incur low simulation cost. However, we choose the simulated level uniform-randomly to reflect the capability of an MRE to be simulated at any level.

The model for our synthetic application consists of one entity (shown in Figure 49) represented at multiple resolution levels. The entity may interact at any level. In order to satisfy R2, the representations of the entity at all resolution levels must be consistent with one another. We make some assumptions about our model for our analyses:

  • · There are L resolution levels, level 0 being the lowest (most aggregate) and level L −1 being the highest (most disaggregate).
  • · A sub-entity at a resolution level j consists of N identical sub-entities at level j +1 if 0 ≤ j < L −1, and zero sub-entities if j = L −1. We refer to N as the fan-out.
  • · All sub-entities at all levels have exactly a attributes. All of the attributes of a sub-entity at a particular level are modified by every interaction at that level.
  • · Interactions may occur at any resolution level.
  • · All interactions are independent of one another. Therefore, concurrent interactions are serialized.
  • · An entity executes progress interactions to advance in the simulation. These interactions do not change attributes, but involve processing on the part of the entity. An entity receives R interactions before receiving a progress interaction.

We define Ψ as a function on X and Y such that: . If an entity is represented at L resolution levels with a fan-out of N , it has Ψ( N , L ) sub-entities. In AD, an entity may be disaggregated down to level L −1, thus requiring Ο(Ψ( N , L )) memory. In SV, only level L −1 sub-entities are simulated, thus requiring Ο( NL -1). In UNIFY , all sub-entities at all levels are present, thus requiring Ο(Ψ( N , L )) memory. The memory consumption for all three approaches is of the order Ο( NL -1).

9.1.3.2 Consistency Cost

Consistency Cost (CC) reflects the number of actions required to maintain consistency when interactions at different resolution levels occur.

Aggregation-disaggregation : In AD, an entity is always simulated at level 0. If an entity receives an interaction at level r (0 < r < L ), the entity disaggregates to level r , applies the effects of the interaction at level r and re-aggregates to level 0 . Aggregation and disaggregation maintains consistency among the multiple representations because the effects of an interaction propagate to attributes at the simulated level. In order to disaggregate to level r from the current level 0, or aggregate from level 0 to level r , the costs incurred are Ο( a × Ψ( N , r )). Thus, CCAD (Figure 50) = Ο(2 a × Ψ( N , r )).

Selective Viewing : In SV, an entity is always simulated at level L −1. There exists only one level of resolution, namely, the highest. Consistency is maintained only within one level. All interactions occur at level L −1, where L = 1. Therefore, CCSV (Figure 51) = Ο( a ).

UNIFY : In an MRE, an entity is represented consistently at all levels of resolution. If an interaction occurs at level r (0 ≤ r < L ), the entity applies the effects of the interaction at level r and propagates the effects to all other levels. In order to propagate the effects to higher resolution levels, the cost incurred is Ο( a × Ψ( N , L r )). The cost incurred in propagating the effects to lower resolution levels is Ο( ra ). Thus, CC UNIFY (Figure 52) = Ο( ra + a × Ψ( N , L r )).

9.1.3.3 Simulation cost

Simulation Cost (SC) reflects the number of actions required to simulate an entity. In AD, an entity is simulated at level 0. Therefore, SCAD = Ο( a ). In SV, an entity is simulated at level L −1. Therefore, SCSV = Ο( a × NL −1). In UNIFY , at a given time, an entity is simulated at one of the multiple levels. If the entity is simulated at level r (0 ≤ r < L ), SC UNIFY = Ο( a × Nr ). Figure 53 shows SC for AD, SV and UNIFY .

9.1.3.4 Expected Costs

Table 13 compares the expected costs for the different approaches. Figure 54 shows a rough diagram of expected simulation and consistency costs for AD, SV and UNIFY .

Cost Comparison among MRM approaches

 

CC

SC

AD

Ο(2 a × Ψ( N , r ))

Ο( a )

SV

Ο( a )

Ο( aNL −1)

UNIFY

Ο( ra + a Ψ( N , L r ))

Ο( a × Nr )

As Figure 54 shows, simulation and consistency costs are trade-offs. Consistency costs decrease with approaches that execute more in the disaggregate. However, simulation costs increase. An approach executing mostly in the aggregate has low simulation costs, but high consistency costs. UNIFY lies between extremes of multi-resolution approaches, i.e.,

SCAD ≤ SC UNIFY ≤ SCSV

CCAD ≥ CC UNIFY ≥ CCSV.
Therefore, UNIFY enables reducing the sum of simulation and consistency costs.

9.1.3.5 Experimental Costs

We constructed a simulation to measure and compare SC and CC for AD, SV and UNIFY . The simulation confirmed our predictions about how the costs grow as factors such as number of levels and fan-out grow. Also, the simulation confirmed our expectation that the total of simulation and consistency costs can be reduced in UNIFY .

All costs were measured in terms of the number of actions. SC was the total number of actions to execute a progress interaction (SCP) and apply the primary effects of an interaction (SCI), i.e., SC = SCP + SCI. SCPAD and SCIAD were one per interaction. For each interaction, SCPSV was equal to the total number of entities at the highest resolution, and SCISV was equal to the number of sub-entities affected by the interaction (after translating a low-resolution interaction into high-resolution interactions). For each interaction, SCP UNIFY was equal to the number of entities at a level chosen uniform-randomly when a progress interaction occurred, and SCI UNIFY was one. CCAD was the number of times sub-entities were created and destroyed per interaction. CCSV was the number of sub-entities created and destroyed initially. CC UNIFY was the number of actions required to propagate the effects of each interaction to all sub-entities and to each parent.

We measured costs by varying four independent parameters one at a time:

  • · T: total number of interactions during the simulation. T = 10, 100, ..., 100000.
  • · R: number of interactions between progress interactions. R = 1, 2, 3, ..., 10.
  • · F: fan-out, or the number of sub-entities per entity. F = 1, 2, 3, ..., 10.
  • · L: number of levels. L = 1, 2, 3, ..., 10.

The canonical case was L = 3, F = 2, T = 1000, R = 5. The graphs that follow should be interpreted for trends rather than actual numbers. The relationship between costs of simulation and consistency and the above parameters are as follows:

  1. 1. As the number of interactions increased, primary effects on sub-entities increased, and more progress interactions occurred (since the number of progress interactions was T÷R). Therefore, SC increased with T for all approaches (Figure 55). SCSV increased the most since all interactions were translated into equivalent highest-resolution interactions. The translation usually resulted in more interactions being generated since a low-resolution interaction affects many sub-entities at higher resolution levels.
  2. 2. As the number of interactions increased, secondary effects on sub-entities increased. Therefore, CCAD and CC UNIFY increased with T (Figure 56). No consistency maintenance is required for SV since only one level is present.
  3. 3. As R increased, progress interactions occurred less frequently, since the number of progress interactions was T÷R. Accordingly, SC decreased with an increase in R for all approaches (Figure 57).
  4. 4. The increase or decrease in R did not change CC since the progress interactions were purely simulation interactions. Accordingly, CC was unaffected by R for all approaches (Figure 58).
  5. 5. As the number of sub-entities for each level increased, SCSV and SC UNIFY increased polynomially. SCSV increased because an increase in the number of sub-entities increased the number of translated interactions. SCSV and SC UNIFY , increased because a greater number of sub-entities resulted in a greater number of actions when progress interactions occurred. SCAD was independent of F because the effects of all interactions were applied at level 0 (Figure 59).
  6. 6. As the number of sub-entities for each level increased, CC increased polynomially for all approaches (Figure 60). An increase in F resulted in an increase in CCSV because more sub-entities were created initially and destroyed finally. CCAD increased with F because more sub-entities were created and destroyed during aggregation and disaggregation. CC UNIFY increased with F because more effects were propagated to other resolution levels.
  7. 7. As the number of levels increased, SCSV and SC UNIFY increased exponentially. SCSV increased because the greater the number levels, the greater the number of translated interactions. For SCSV and SV UNIFY , a greater number of levels resulted in an greater number of actions for progress interactions. SCAD was independent of L since the effect of all interactions, including progress interactions, were applied at level 0 (Figure 61).
  8. 8. As the number of levels increased, CC increased exponentially for all approaches (Figure 62). CCSV increased with L because more sub-entities were created initially and destroyed finally at level L −1. CCAD increased with L because more sub-entities were created and destroyed during aggregation and disaggregation. CC UNIFY increased with L because more effects were propagated to other resolution levels.

In Figure 63, we plot SC, CC and Total Cost using each approach for the canonical case (L = 3, F = 2, T = 1000, R = 5). Total Cost is a weighted sum of simulation and consistency costs. The weights for SC and CC are application-specific; in the graph in Figure 63 we assign equal weights to them, i.e. Total Cost = SC+CC. UNIFY incurs the least total cost in this case. Other cases in which the values of the above parameters were varied indicate similar trends.

9.1.3.6 Summary of Cost-Effectiveness

UNIFY satisfies R3 by enabling reductions in the costs of simulation and consistency maintenance. Although selective viewing incurs low consistency cost and aggregation-disaggregation incurs low simulation cost, both approaches fare poorly when both costs are considered. In contrast, UNIFY achieves lower total cost than either aggregation-disaggregation or selective viewing. An approach that achieves MRM at a high cost is ineffective because it does not satisfy R3. UNIFY enables the total of simulation and consistency costs to be reduced, thus satisfying R3.

9.1.4 Summary of Evaluation in Terms of MRM Requirements

UNIFY satisfies our three requirements for effective MRM: multi-representation interaction (R1), multi-representation consistency (R2) and cost-effectiveness (R3). These requirements must be satisfied by any approach in order to achieve effective joint execution of multiple models at reasonable cost. Alternative approaches such as aggregation-disaggregation and selective viewing do not satisfy all of R1, R2 and R3. Therefore, by these criteria, UNIFY is better than the popular MRM approaches.

9.2 Applying UNIFY to Existing Models

We have applied UNIFY to four models. Three of them are military models specified using OMT. The fourth is a hierarchical autonomous agent that is a research project at the University of Virginia. For all four models, we constructed an MRE from attributes at multiple representation levels. We constructed an ADG for each MRE. We classified the interactions in each model according to our taxonomy. For each model, we assumed reasonable mapping functions and policies for resolving concurrent interactions. For each model, we worked only from specifications, since pursuing the project to implementation would have been an unreasonably large undertaking.

9.2.1 Military Models

The three military models we considered are part of the Department of Defense's High Level Architecture (HLA). They are: Joint Task Force prototype (JTFp) [JTFp97], Joint Precision Strike Demonstration (JPSD) [JPSD97] and Real-time Platform Reference (RPR) [RPR97]. These models have been the basis of many examples that we provided in this dissertation to explain techniques in UNIFY . The process for applying techniques in UNIFY to these models is shown in Chapter 8:

  1. 1. Construct a Multiple Representation Entity (MRE) from the OCST.
  2. 2. Capture relationships among the attributes with an Attribute Dependency Graph (ADG) constructed from the APT and the ART (see §8.2).
  3. 3. Select mapping functions for each dependency in the ART.
  4. 4. Determine the effects of interactions from the OIT, and classify interactions according to our taxonomy.
  5. 5. Resolve the effects of concurrent interactions from policies specified in the CIT (see §8.2).
  6. 6. Construct a Consistency Enforcer and an Interaction Resolver for the MRE.

The results of our experience with these models are a proof-of-concept for UNIFY . Designers of jointly-executing battlefield models can achieve effective MRM by applying UNIFY . For each of these models, we were able to apply UNIFY , thus avoiding pitfalls encountered with alternative MRM approaches. Details of how we applied UNIFY to JTFp, JPSD and RPR appear in Appendices B, C and D respectively.

9.2.2 Autonomous Agent Model

We applied UNIFY to a hierarchical autonomous agent model [Was98b]. The autonomous agent model we considered is part of a research project undertaken by the Vision Group at the University of Virginia. The autonomous agent, Marcus, has been programmed to construct complex arrangements from basic building blocks. Figure 64 shows Marcus with an example arrangement, an archway.

Marcus is a hierarchical autonomous agent that has two models, one corresponding to a planner and the other corresponding to a perception-action (PA) system. Typically, the planner maintains long-term or abstract representation, whereas the PA system maintains immediate and detailed representation. Each model may have its own representation of the world in which Marcus operates. Accordingly, each model may represent building blocks, partially-completed arrangements, obstacles, doors and pathways by a number of relevant attributes such as position, orientation and colour. Marcus considers relationships among blocks that are stacked or placed next to each other as an arrangement.

We constructed an MRE for Marcus's planner and PA system and captured dependencies among attributes with an ADG. In the current implementation of Marcus, interactions occur only at the PA level through sensors and effectors. Planner-level interactions originating from user directives are envisioned as future work. Therefore, we classified interactions at only the PA level. Figure 65 shows a partial ADG for an MRE constructed from the planner and PA representations for Marcus. The MRE contains all of the objects (and their attributes) that the planner considers important for the current task, and all of the objects (and their attributes) that the PA system senses and affects. For brevity, we show only objects represented by the planner and PA, but not their attributes. We show dependencies that exist among objects when Marcus constructs an arrangement. Wasson shows how representations can be constructed for the models in Marcus and how consistency can be maintained among the representations [Was99].

Our experience with the hierarchical autonomous agent model indicates that the techniques in UNIFY can be applied to multi-models in different domains. A valid concern with any framework-based approach is whether the framework is general enough to be useful to applications in many domains. Applying UNIFY to applications in many domains would be a convincing, but time-consuming, argument for the applicability of UNIFY . We chose one domain -- that of hierarchical autonomous agents -- to show that UNIFY can be applied to many domains. Details of how we applied UNIFY to a hierarchical autonomous agent appear in Appendix E.

9.3 Limitations

A fair evaluation of any research must include the known limitations of the work. The underlying feature of our work is a design decision to maintain concurrent representations of jointly-executing models to enable effective MRM. In order to support this decision, we constructed a framework, UNIFY , consisting of techniques and processes for achieving effective MRM. However, in order to make UNIFY a viable approach for MRM, we made some assumptions about jointly-executing models. These assumptions are the limitations of UNIFY . T hese limitations, individually and together, neither make UNIFY unusable nor outweigh its benefits.

UNIFY is limited to models in which representation exists for objects and processes that are part of a model. We assume that designers can describe properties of objects and processes in a model, i.e., they can represent a model. UNIFY is not applicable to models wherein representation does not exist. Our assumption about representation is reasonable because a large number of practical models represent objects and processes.

In UNIFY , we assume that individual models meet their users' requirements. UNIFY permits designers to capture the combined semantics of multiple models of the same phenomenon. Whether the individual models meet their users' requirements or not is an important issue, but outside the scope of our work. Our work addresses the effectiveness of the joint execution of multiple models alone.

In UNIFY , we assume that multi-models progress in compatible time-steps. We discussed compatible time-steps in §5.2. We regard our assumption of compatible time-steps as the most critical assumption in UNIFY . General techniques for achieving compatible time-steps would be a desirable addition to UNIFY .

UNIFY requires appropriate mapping functions to translate attributes from one representation to another and appropriate policies for resolving the effects of dependent concurrent interactions. We do not regard our assumptions about the presence of mapping functions and interaction policies as critical assumptions because:

  1. 1. Mapping functions and interaction policies capture semantic information about an application. Semantic information is specific to an application and can be provided by a designer.
  2. 2. Alternative approaches to MRM also require similar mapping functions and interaction policies (see §5.3).
  3. 3. We guide designers in the selection of mapping functions and interaction policies (see §6.2 and §7.6).

Despite these limitations, UNIFY is a viable approach for MRM. Its benefits outweigh its limitations. It eliminates or reduces many problems with alternative MRM approaches (see §5.5). It provides designers with techniques for resolving concurrent interactions (see Chapter 7) and applying their effects consistently (see Chapter 6). It provides designers with a process for achieving MRM (see Chapter 8) effectively and practically (see §9.1 and §8.2). Hence, UNIFY enables designers to achieve effective MRM.

9.4 Chapter Summary

UNIFY is a sufficient and practical approach for effective Multi-Representation Modelling (MRM). It is the first known approach to MRM that satisfies R1, R2 and R3. Its limitations are not serious. We have applied it to four practical applications and established that it supports MRM exactly as we have claimed it would. Next, discuss contributions of our work and present future directions for research.