Fortran is the most widely used programming language for high-performance scientific computing applications. In the high-performance computing community it has received more attention than any other single language. Indeed, many parallel Fortran dialects exist, from Fortran M to Fortran 90 and HPF. In the past, the Legion project has not attempted to support Fortran in its native form, opting instead to concentrate on object-oriented languages more closely matched to the Legion programming model. The effort to provide similar compiler support for Fortran that we provide for C++ was beyond the scope of the project.
Instead, we have an alternative to full-fledged compiler support, called Basic Fortran Support or BFS. BFS is a compromise between full-fledged compiler support and no support. The basic idea harks back to the pseudo-comments used in early parallelizing Fortran compilers such as Paraphrase . Rather than creating an extended Fortran dialect that includes coarse-grain objects and attempting to parse and perform data-dependence analysis on Fortran programs, we provide a set of Legion directives that can be embedded in Fortran code in the form of pseudo-comment lines. The programmer uses these embedded pseudo-comments to call member functions on Legion objects and to obtain the results of these member functions. The pseudo-comment syntax is intentionally similar to Fortran, and is easy for Fortran programmers to learn.
The pseudo-comments are translated by a simple preprocessor, legion_bfs, into calls to a BFS support library, which in turn directly invokes the Legion run-time system. The pseudo-comments direct the preprocessor to invoke member functions on Legion objects. Keywords in the pseudo-comments indicate whether the invocation is a subroutine-style invocation or a function, whether the call should be made synchronously or asynchronously, and whether the call is to a stateless object or to stateful object. The execution model is that of Legion -- each "remote" member function invocation executes in a separate address space. Thus, all communication between the caller and callee is via parameters and returns values -- there is no global memory. In the case of stateless objects, the back-end run-time system will create new object instances to service function invocations as needed.
In addition to invocations on Legion objects, the Fortran program can define Fortran-based Legion objects using a Fortran-like interface description language (IDL). The IDL is translated by the legion_bfs filter, which generates Fortran skeletons that can be linked to Fortran code in order to implement the object interface. The Fortran implementation code may itself be the output of the legion_bfs preprocessor, allowing Fortran code to act as both client and server in a Legion system.
The Legion BFS programming interface consists of two basic parts: client-side method invocation through pseudo-comment directives, and server-side object development through the BFS IDL. In "client" code (i.e. Fortran code that will invoke methods on Legion objects) pseudo-comments are used to describe method invocations. These pseudo-comments are translated into calls to a BFS support library, which in turn uses the services provided by the Legion system to support method invocation (see Figure 4). Fortran code that will be used to implement Legion objects (i.e. Fortran code that will be called through a Legion member function interface) must be described by programmer-supplied BFS IDL interfaces. These IDL interfaces are translated by the legion_bfs filter, which generates MPL  skeleton code. This skeleton code is compiled using the legion_mplc compiler and linked to Fortran implementation code (see Figure 5).
It is certainly possible that "server" code -- code that will be called through Legion member functions -- may need to call the methods of other Legion objects.In such cases, the server implementation code includes the appropriate BFS pseudo-comment directives, is translated by legion_bfs, and is then linked to the appropriate IDL (also translated by BFS), as depicted in Figure 6. Thus, a single Fortran subroutine or function might both be called through Legion, and call other objects through Legion. In the following two sections, we examine the server and client programming interfaces in more detail. The combination of these two (as depicted in Figure 6) is straightforward, and is thus left to the reader.
We first consider the problem of implementing Legion objects in Fortran, the process of taking subroutines and functions defined in Fortran and making them available as Legion methods. Fortran code wrapped in this manner can be called in parallel from anywhere in a Legion network. In section 4.1.2 we will examine the programming interface used to invoke functions on Legion objects -- the client side interface.
BFS supports the definition of two basic kinds of Legion objects: stateless and stateful. Stateless objects are useful for making purely functional services available. Stateless objects support methods whose outputs are pure functions of their inputs -- the operation of a stateless object is not affected by previous method invocations or evolving object state. Because of this, methods invoked on stateless objects are automatically load-balanced among all available instances of the given stateless class, thus allowing good performance in variably loaded or heterogeneous networks. Despite this advantage, it is often more natural to deal with objects that maintain state between method invocations. To enable this programming style, BFS supports stateful classes. Instances of these classes have normal method invocation semantics: methods are invoked on a specific object, and the object "remembers" its state from one invocation to the next.
To make Fortran subroutines and functions available in the form of a Legion object, we must "wrap" the Fortran code that implements the functions in a C++/MPL skeleton, invoke the MPL compiler, and link the generated MPL objects with the Fortran implementation code provided by the user. This process is automated through the use of a simple IDL that follows the same form as the caller-side specification of the function (discussed in Section 4.1.2). The user specifies the name and parameters of all functions in an interface specification (i.e. IDL) file. The legion_bfs filter parses this interface specification file and generates an MPL program that matches the interface. The user then compiles and links the program using legion_mplc. No changes to the Fortran code, nor any C++ or MPL programming on the part of the user is required.
Suppose we wish to make a Fortran module containing a function called function1 and a subroutine called subroutn1 available in the form of a stateless Legion object. The following IDL interface specification file would be used:
To compile an object, first translate the IDL file using legion_bfs. Given a file with a ".bfs" suffix, legion_bfs will generate a ".c" and a ".h" MPL file. The resulting MPL file should be compiled using legion_mplc. The object file containing the Fortran implementation code should be specified on the legion_mplc link line. For example:
In this section, we examine the client-side (i.e. method invocation) programming interface. The client-side programming interface is based on the use of pseudo-comments. The programmer embeds BFS directives into normal Fortran code using comments that begin with the prefix LEGION. This "annotated" Fortran code is then translated by the legion_bfs filter, which generates standard Fortran code. This resulting code is then compiled and linked against a BFS support library and the Legion library.
Before methods can be invoked on objects of a given class, the BFS translator must have a description of the class's interface. This interface description will be used by legion_bfs to generate the appropriate calls to the Legion run-time system. The mechanism for communicating object interfaces to the translator is the INCLUDE directive, which is used to "include" BFS IDL files in BFS Fortran code. For example, the following statement
In this section we cover the BFS syntax used to invoke methods. We start with a very simple example: a blocking remote method invocation that takes a single parameter and returns a single result. To call remote method function1 defined by stateless class my_class, which takes an integer parameter and returns a real result, the following syntax could be used:
This would perform a blocking remote procedure call (RPC) on an instance of my_class: calling function1, passing the integer parameter I as an input, and placing the real return value in the variable X. Note that the complete method invocation is contained within a single Fortran comment line. Note also that this method is invoked on a stateless Legion class, whose instances are therefore pure functional units. Thus, this method will be serviced not by a specific named object but by any available (or newly created) instance of my_class.
The above example performs a standard, blocking RPC on a Legion object. To achieve parallelism between remote method invocations, a non-blocking RPC mechanism is necessary. In BFS, non-blocking RPCs are achieved through the use of the ASYNCH specifier.
This statement causes the same function to be executed but does not wait for the return value to be placed in X. When this statement is executed, the method function1 begins to be processed at the remote object, and the caller immediately proceeds. The caller can later block for the result using the statement:
Since the variable X was previously named as the result of an asynchronous method invocation, when this statement is encountered the caller blocks for the result and assigns it to X. The use of asynchronous method calls allows methods to execute in parallel. For example, if function1 is time consuming, a caller of this function might perform other work in parallel before blocking for the result. This parallel work could include the invocation of other Legion methods, such as additional calls to function1.
Unlike Fortran, which uses call-by-reference parameter passing, the default BFS parameter passing convention is call-by-value. So, if the variable I in the above example is modified by function1, that change would not be propagated back to the caller. Other parameter-passing semantics are supported through the use of the key words IN, OUT, and INOUT. For example:
The effect is to pass the array A into subroutn2, and, when the method completes, to copy the array back to the caller. The maximum number of dimensions is not fixed, though the user should be aware that there must be adequate memory to copy parameters, and that large parameters require more communication time.
To this point, we have only demonstrated calls on stateless objects. As described in "Server-side programming" BFS also supports stateful objects -- objects that maintain state between method invocations. Unlike stateless invocations, which are performed on any object of a given class, stateful invocations must be performed on a specific, named object instance. In BFS, Legion objects are identified by integer OIDs (Object IDentifiers) that are only valid in the local address space. OIDs are obtained in two ways: as the result of object creation requests, and as the result of looking up object names in Legion context space.
This statement will cause the creation of a new object my_stateful_class. An OID that refers to the new object is stored into the integer variable OBJ. In addition to creating new objects, BFS programs can bind to existing objects. An OID for an existing object, which is named in Legion context space, can be obtained using the LOOKUP directive:
Unlike memory that is declared within a program, stateful objects can persist indefinitely beyond the lifetime of the program that creates them. To avoid creating "garbage" objects -- objects that are consuming system resources but are no longer needed by any programs -- BFS programs must clean up their stateful objects before terminating. Stateful objects may be deleted using the DESTROY directive.
Fortran code containing the pseudo-comment directives discussed in this section must first be preprocessed by the legion_bfs translator. Given a file with a ".f" suffix, the translator will produce a new Fortran source file with the suffix ".trans.f". This resulting file should be compiled by a standard Fortran compiler, and linked against the Legion libraries using the following flags:
In this section, we present a simple but complete program using stateful objects. Consider the Fortran code depicted in Figure 7. To make this code available in the form of a Legion object, we define an object interface for it using the IDL depicted in Figure 8 (below). Note that since the Fortran code in Figure 7 depends on state that is maintained between method invocations (i.e. the common block variable A), we use a stateful Legion object to wrap the code.
Once the IDL depicted in Figure 8 has been translated by legion_bfs, compiled by legion_mplc, and linked to the object code resulting from a Fortran compilation of the code in Figure 7, programs can create objects of the class "my_class" and invoke methods on them.
Figure 9 shows a simple BFS Fortran main program that demonstrates object creation, a remote subroutine call, a remote function call, and object deletion. Note that the integer variable OID is used as a local reference to the Legion object created in the program. Also note that in this simple example we use synchronous method invocation, since no parallelism was possible between the two method invocations. In Section 4.3 we consider the use of asynchronous methods and stateless objects, both of which have the potential to offer improved performance.
Both functions defined in this file are purely functional: they are free of side effects and do not rely on state set by previous methods when producing their results. Because of these features, this Fortran module can be wrapped in a stateless Legion object and thus benefit from improved performance through automatic load balancing. The BFS IDL required to wrap this code is depicted in Figure 11, below.
Figure 12 depicts a simple Fortran main program that uses the stateless class dprod_object. This example performs two dot product operations and computes the sum of their results. Note that the two dot product operations are completely independent of one another -- they operate on entirely disjoint data. Thus, we use ASYNCH methods to allow the functions to proceed in parallel. A further benefit of using ASYNCH methods comes from data dependence analysis performed by BFS: since the results of the dot product operations are needed to perform the sum operation, the results of these methods are forwarded directly to the object that will perform the sum operation. Direct forwarding of results, as afforded by ASYNCH methods, improves performance by reducing communication.
Here instead of sending results D1 and D2 first to the main program and then to the sum operation, the parameters skip the middle hop and go directly to the sum operation. This optimization is especially important when array parameters are used, as they consume the most communication resources. The output of the program depicted in Figure 12 is: