Abstract

Background

In the OSI Reference Model [OSI7498], each layer of the model is responsible for providing a specific set of services to the layers above and below it. For the transport layer, those responsibilities include:

client identification multiplexing multiple users of the transport layer and keeping their data streams separate

segmentation dividing an arbitrarily long message into packets which are of a convenient size for the underlying network, then reassembling them at the receiver into a duplicate of the original message

correctness implementing a transport checksum that is used to verify the correctness of data received, and rejecting data received with data errors

reliability assuring that the delivered data stream is sequential, with no lost data and no duplicate data

In the Internet world, the Transmission Control Protocol (TCP) [DARPA81a] provides these functions, and is typically used for data exchange that requires full reliability (like file transfer). In addition, the User Datagram Protocol (UDP) [DARPA80] is available for less demanding applications; it handles a single message of up to 64 KB on a "best-effort" (unacknowledged) basis and is used for data that does not require guaranteed delivery. Both TCP and UDP require the services of the Internet Protocol (IP) [DARPA81b] in the network layer below them to route messages to their destination. In the ISO world, the Transport Protocol class 4 (TP4) [IS08073] and the Connectionless Network Protocol (CLNP) [IS08473] provide similar services.

TCP, UDP, and IP are very important and very valuable protocols in the networking world; even though they are neither national nor international standards, they are defacto standards in the Internet environment. Having been designed many years ago, however, TCP/UDP and TP4/CLNP did not anticipate the many new types of applications that now require interconnection across LANs and MANs and WANs. Thus, when TCP/TP4 are applied to more demanding applications than file transfer or electronic mail, for example the interconnection of distributed applications, or support for real-time embedded systems, or transport of synchronized multimedia data streams, they show some deficiencies. For example:

1. TCP supports only peer-to-peer connections (that is, one transmitter talking to one receiver) so a communication from, say, one transmitter to ten receivers requires ten connections and ten independent transmissions. There is no concept of multicast.

2. Since there is no multicast, there is no concept of multicast group management, i.e., there is no ability to define a group of transmitters and receivers or for the application program to define the reliability requirements of the group members.

3. TCP provides little support for priority or latency control, both crucial for distributed real-time systems.

4. TCP supports only fully reliable transmission (i.e., it repeats missing data until it finally arrives at the destination, no matter how long it takes) which is too strong for some applications. UDP supports only a "best-effort" service (i.e., the transmitter never knows if the receiver got the data) which is too weak for some applications. Data reliability is defined by the protocol, not by the application.

5. To send a single message reliably (as with transactions), TCP requires the exchange of six packets (two to set up and acknowledge the connection, two to send and acknowledge the data, and two to close the connection). This slows the exchange of data for short-lived connections since connection setup and teardown are accomplished separately from data transfer.

6. When errors do occur, TCP recovers using a go-back-n algorithm that retransmits all data beginning at the point of loss. For networks with high data rates and/or long delays (e.g., coast-to-coast links, satellites), this leads to inefficient use of the network when data previously sent and correctly received is nevertheless resent, only to be discarded at the destination as duplicate data.

7. TCP, UDP, and TP4 all implement fixed policies defined by the protocol, not the application. For example, all connections are governed by flow control which restricts the amount of data the transmitter can send. While this is often correct, there are some "streaming" applications that would benefit from unrestricted flow, but that option is simply not available. Similarly, there are applications that can tolerate data errors, but neither TCP nor UDP will deliver corrupted data. The point here is that TCP and UDP embed policy with mechanism, and the two can not be untangled.

A New Approach

It is important to note that XTP is not perceived as a replacement for TCP or UDP or TP4; rather it is simply another transport protocol with its own unique functionality which is available as an option for the system designer. Just as TCP and UDP operate side-by-side above IP, XTP operates simultaneously with any number of other transport protocols, including TCP and UDP and TP4, and utilizes the services of the network layer below it, if any. XTP operates over IP (which makes it usable in any Internet environment), over CLNP (which covers the ISO networks), directly over the LLC or MAC of any LAN (which covers Ethernet, token ring, and FDDI), and directly over the adaptation layer of an Asynchronous Transfer Mode (ATM) network. The possible interplay of transport protocols, network protocols, and networks is shown in Figure 1:

XTP Functionality

Multicast

Figure 2(a) represents a traditional unicast, whereas Figure 2(b) shows the basic one-to-many approach of multicast. Figure 2(c) shows an N-to-M multicast. Groups that require more than a single transmitter can be synthesized by using N l-to-N multicast groups to form a N-to-N multicast capability as shown in Figure2(d):

Multicast Group Management (MGM)

Note how inappropriate it would be to have the protocol manage group membership via some predefined policy. Only the application can appreciate the side-effects of a change in group membership, so XTP reports group membership changes to the transmitting application and lets it decide what should be done.

Since group membership is under program control, one could define a multicast group in which, say, three particular members must receive all data reliably and all other members may eavesdrop as they wish. Given that definition, XTP MGM would report failures of any of the three essential nodes, but would ignore the joining and leaving of other non-essential listeners.

Also, we note that the reliability of group membership is orthogonal to the reliability of data transfer within a group. As discussed in the section on selectable error control, XTP supports multiple paradigms for error control, but these are independent of the group membership issues.

Priority

XTP puts a bit in the packet header that indicates whether or not each packet is participating in the overall priority scheme. If so, then an adjacent field indicates the relative priority of this packet compared to all other packets. XTP examines the priority bit and its associated numeric field and then processes that packet in accordance with its defined importance. One can say that, to a granularity of one packet's transmission time on the network, XTP is always operating on its most important packet. This is true not only in the end-systems, but also in all interior routers if they support the necessary mechanisms to detect and respond to XTP priority (see the section on the XTP-aware IP router for a discussion).

The XTP priority scheme is based on a static priority encoded in a 16-bit numeric field; the lower the numerical value, the higher the importance. By defining the priority system this way, putting a delivery timestamp in the priority field automatically implements an earliest-deadline-first delivery policy with no additional work on the part of the user.

Users generate These messages are XTP emits its messages of varying sorted using priority most important importance. queues. Message first.

Rate and Burst Control

When using TCP, a slow receiver's protection from a fast transmitter has to be synthesized by flow control, that is, the opening and closing of credit windows. This causes dynamic throughput fluctuation as transmission starts and stops in accordance with the demands of the flow control window. In contrast, rate control allows a steady stream of data to be emitted at a rate known to be acceptable to the receiver. Additionally, conventional flow control is only active end-to-end; that is, routers do not participate. Thus a receiver with spare capacity can open a large flow control window to invite rapid transmission, but this might only result in further congestion of some intermediate router. XTP's rate and burst control algorithms allow the routers to participate; thus a congested router could temporarily reduce the rate control parameter of its incoming connections until the period of congestion has passed.

Connection Management

In the special case of a transaction, XTP requires only two packets. Host A sends its first packet to B that opens the connection and transfers the data (the transaction request); host B replies by sending its data (the transaction response) and closing the connection.

Selectable Error Control XTP supports not one but three types of error control:

(a) a fully reliable mode like TCP and TP4, which would normally be used for applications such as file transfer;

(b) a UDP-like service, in which the receiver does not acknowledge transmission and thus the transmitter never knows whether the data was successfully delivered (i.e., a datagram); and (c) a special mode called fast negative acknowledgment that can improve error repair in certain situations. If the networking environment is a LAN, then out-of-sequence data delivery is much more likely to mean that the missing data is truly lost, as opposed to delayed (as it might be by taking an alternate path in a WAN). Using the fastnak option, a receiver that identifies outof-sequence delivery immediately (without waiting for any time-outs) sends a control packet that informs the transmitter about the missing data. The transmitter then immediately resends the missing data.

Another independent option is noerror mode which suspends the normal retransmission scheme. Correctly received data is properly sequenced, but gaps, if any, are not retransmitted. This mode is expected to find utility when carrying digital voice and video.

Not only does XTP provide a range of data reliability options, it does so in a single protocol. Each connection can support an independent choice of error control strategy, and the choice is entirely under the control of the application.

Selectable Flow Control

Selective Retransmission

Historically, retransmission has been regarded as either inappropriate or ineffective for jitter control in multimedia systems, and this may well be true if TCP-style retransmission is used. However, we have shown in [Dempsey94a,b,c,d] that retransmission, layered on top of an efficient retransmission mechanism like that in XTP, can be very effective for controlling packet loss in delay-sensitive streams.

Selective Acknowledgment

Maximum Transmission Unit (MTU) Detection

Out-of-band Data

Alignment

Traffic Descriptors

Performance

Running under pSOS+ on a pair of 133 MHz Pentium-based PCs connected by Rockwell FDDI interfaces, we see the following results:

Throughput: using reliable transport unicast and large (64KB) messages, these machines sustain a throughput in excess of 90 Mbits/sec.

Throughput: using unacknowledged transport multicast (for which there is no counterpart in TCP), the machines can sustain a throughput of 96 Mbits/sec.

Latency: the end-to-end delay for transmitting a message is very short; the elapsed time for moving a 16-byte message from user memory in the transmitter to user memory in the receiver is 220 Us.

In general, our experience has been that a skillful XTP implementation delivers to the application approximately 80-96% of the raw network bandwidth available at the level of the MAC device driver. Nevertheless, because it is difficult to separate the skill of the implementer from the inherent power of the protocol, XTP does not make the claim that it is faster than any particular protocol; the measured performance simply speaks for itself. XTP is a tool, just like TCP and UDP are tools, and it should be used wherever appropriate. In my opinion, XTP's major contribution is not so much its performance but its functionality. The ability to utilize many different communications paradigms, all in the same protocol, and all under the user's control, is a major advantage for the system designer.

Applications

Multicast. A government contractor needed to interconnect 22 CRAY supercomputers, all riding the belly of a C-130 aircraft, to form a flying signal intelligence center. Since this environment required sending identical data streams to multiple processors, they chose XTP so that they could utilize its multicast capabilities. Each multicast transmission consumes only a fraction of the network bandwidth that would otherwise have been required by N serial unicasts.

Gigabyte files. A commercial company needed to transfer gigabyte files routinely across a satellite-based network. They chose XTP to utilize its selective retransmission capability. When errors corrupt the satellite channel, XTP retransmits only the data that was lost; it does not restart the data stream from the point of loss.

High performance. A government agency needed a communications protocol that would support the real-time environment of weapons control. They chose XTP based on its low end-to-end latency characteristics. Being able to reliably send modest size messages with sub-millisecond latencies allowed them to write effective feedback control loops even in a packet-switched networking environment.

Image distribution. One company faced the problem of having to send very high resolution medical images over an FDDI-based hospital network. They needed to move multiple mammography images (40 MB each) to multiple work stations for simultaneous review by a staff of radiologists. They chose XTP for its combination of high throughput and multicast distribution.

Digital telephony. A government contractor wanted to operate a digital telephone/ intercom system over an existing FDDI network. They had to support 120 simultaneous bidirectional conversations (64 Kbits/s each), and each conversation had to have the capability of being sent to any combination of destinations. They chose XTP because its multicast feature would allow any number of receivers to join a multicast group (under the supervision and control of the transmitter), thus making it simple to implement the cross-communications requirement.

Video file server. A company needed the capability to send and receive "video mail" (synchronized audio/video) throughout their corporate network, in addition to their traditional text-based electronic mail. They chose XTP because it handled the multimedia requirements with ease. Personal computers were able to mount a remote disk drive (the file server) and retrieve a compressed multimedia stream in real time.

Priority support. A government research lab required that its packet-switched network be able to support digital multimedia streams in addition to file transfer and electronic mail. They used XTP in combination with our XTP-aware IP router (see following section) to achieve this goal. The IP router identifies XTP packets and handles them in accordance with the XTP priority option.

Real-time systems. A government contractor needed to design a ship-board network, based on Safenet, that would handle time-critical command and control messages, multimedia streams, background file transfer, and data distribution from one source to multiple destinations. They chose XTP because it provided latency control, message-level priorities, multimedia support, high overall performance, and multicast capabilities to handle the distribution of identical data streams.

Interoperability. Desiring to preserve the connectedness and interoperability associated with the classic protocols, one government contractor build a SAFENET network incorporating TCP and UDP and XTP, all operating over IP. Another vendor provided TP4 and XTP, both over CLNP. Yet another vendor, working with real-time data, put XTP directly on top of the MAC of FDDI.

Protocol Engine

XTP's Future

1. XTP is just as interoperable as TCP or TP4 because it can run over IP or CLNP;

2. XTP has all the functionality of TCP and UDP and TP4, plus more, all in a single protocol;

3. XTP offers significant new functionality (especially the transport layer multicast, multicast group management, and transport layer priorities).

References

[DARPA81a] Postel, J., ed., "Transmission Control ProtocolÑDARPA Internet Program Protocol Specification," RFC 793, USC/Information Sciences Institute, September 1981.

[DARPA81b] Postel, J., ed., "Internet Protocol-DARPA Intemet Program Protocol Specification," RFC 791, USC/Information Sciences Institute, September 1981.

[Dempsey90] Dempsey, B.J., Fenton, J.C., and Weaver, A.C., "The Multidriver: A Reliable Multicast Service for the Xpress Transfer Protocol," 15th Local Computer Networks Conference, Minneapolis, Minnesota, October 1-3,1990.

[Dempsey91] Dempsey, B.J., An Analysis of Multicast and Multicast Group Management, M.S. thesis, Department of Computer Science, University of Virginia, January 1991.

[Dempsey94a] Dempsey, B.J., Lucas, M.T., and Weaver, A.C., "An Empirical Study of Packet Voice Distribution over a Campus-Wide Network," l9th IEEE Local Computer Networks Conference, Minneapolis, Minnesota, October 1994.

[Dempsey94b] Dempsey, B.J., Lucas, M.T., and Weaver, A.C., "Design and Implementation of a High Quality Video Distribution System using XTP Reliable Multicast," Second International Workshop on Advanced Communications and Applications for High-Speed Networks, Heidelberg, Germany, September 1994.

[Dempsey94c] Dempsey, B.J., Retransmission-Based Error Control for Continuous Media Traffic in Packet-Switched Networks, Ph.D. dissertation, Computer Networks Laboratory, Department of Computer Science, University of Virginia, May 1994.

[Dempsey94d] Dempsey, B.J., Liebeherr, J., and Weaver, A.C., "A New Error Control Scheme for Packetized Voice over High-Speed Local Area Networks," 18th IEEE Local Computer Networks Conference, Minneapolis, Minnesota, September 1993.

[IS07498] International Organization for Standardization, "Information Processing Systems-Open Systems Interconnection-Basic Reference Model," International Standard 7498, October 1984.

[IS08073] International Organization for Standardization, "Information Processing Systems-Open Systems Interconnection-Transport Protocol Specifications," International Standard 8073, July 1986.

[IS08473] International Organization for Standardization, "Information Processing Systems-Open Systems Interconnection-Data Communications Protocol for Providing the ConnectionlessMode Network Service, International Standard 8473, March 1986.

[Mentat94] "Mentat XTP for Streams," Mentat, 1145 Galey Avenue, Suite 315, Los Angeles, California 90024 USA.

[Michel93a] Michel, J., Waterman, A., and Weaver, A., "Performance Evaluation of an Off-Host Communications Architecture," High Performance Communication Subsystems, Williamsburg, VA, September 1-3,1993.

[Michel93b] Michel, J., Waterman, A., and Weaver, A.C., "Performance Evaluation of an Off-Host Communications System," 18th Local Computer Networks Conference, Minneapolis, MN, September 19-22, 1993.

[MIL2204] Survivable Adaptable Fiber Optic Network, U.S. Department of Defense Military Standard MIL-STD-2204, October 1992.

[NetX95]"Xpress Transport Protocol version 4.0 Multiuser Application Programming Interface," Network Xpress Inc., 1111 Rose Hill Drive, Suite 9, Charlottesville, Virginia 22903 USA.

[Strayer92]Strayer, W.T., Dempsey, B.J., and Weaver, A.C., XTP: The Xpress Transfer Protocol, Addison-Wesley, 1992.

[Strayer94]Strayer, W.T., Gray, S., and Cline, R.E., Jr., "An Object-Oriented Implementation of the Xpress Transfer Protocol," Proceedings of the Second International Workshop on AdvancedCommunications and Applications for High-Speed Networks,

(IWACA'94), Heidelberg, Germany, September 26-28, 1994.

[XTP95] Strayer, W.T., ed., "Xpress Transport Protocol 4.0 Specification," XTP Forum Inc., 1394 Greenworth Place, Santa Barbara, CA 93108 USA.