WO1997020267A1 - Method and system for modeling and presenting integrated media - Google Patents

Abstract

A method and system for modeling interactive animation and other integrated media includes support for a declarative modeling language and a system for presenting media represented in a declarative language model. The modeling language enables authors to represent natural or modeled media in a compact model, and it allows for the explicit description of time varying behavior as well as reactive behavior, which occurs in response to discrete events. The system compiles or interprets the model, optimizes it, and controls the presentation of media represented in it.

Description

METHOD AND SYSTEM FOR MODELING AND PRESENTING INTEGRATED MEDIA

TECHNICAL FIELD The invention relates to modeling and presentation of interactive animation and integrated media such as sound, images, or graphics.

BACKGROUND

With the increasing interest in multimedia applications, there is a demand to expand the capacity of computers to support video, audio, two-dimensional (2D) and three-dimensional (3D) graphics, and animation. Presently, it is not unusual for a moderately priced personal computer to support video, audio, and 2D graphics. A wide variety of special add-on boards have been created to expand the capability of the PC. For example, a graphics accelerator can be connected to a PC to boost the PC's ability to support 3D graphics rendering. A sound card can be added to provide high quality, stereo output. A variety of input devices have also been developed to enhance the manner in which users interact with the computer system. For example, a variety of gestural input devices have been developed such as joysticks, data gloves, and head trackers, to name a few.

While computers are now capable of providing output using a variety of media types, the difficulty is integrating media types to produce more realistic and interactive effects. The central problem is the creation of applications that effectively integrate different media types including both natural and modeled media. Natural media refers to recorded or live media such as audio or video data streams. Modeled media refers to media that is generated from computer based models such as 2D and 3D graphics and animation.

A number of critical design issues arise in developing application and system software to gracefully present integrated media to the user. The difficulty stems from the fact that input and output is traditionally handled in separate subsystems. For instance, audio, video, geometry rendering, and gestural input is typically handled using disparate hardware devices as well as separate low level programming interfaces. These disparate subsystems present difficult obstacles to implementing applications using integrated media types. Important design issues include regulation of computer resources, synchronization of media streams, and reducing latency from user input to output.

Regulation generally refers to the management of system resources in an efficient manner when performing a task or a series of tasks. An example will help illustrate the concept. Consider an application where a 3D animation is to be displayed and an accompanying audio file is to be played along with the animation. In producing the desired effect, the computer uses its available resources such as its local memory, its processor, a display controller, and a sound card. At some level, whether it is at the application program level or at the system level, the computer should be programmed to use its resources as efficiently as possible to achieve the highest quality output Regulation is particularly important where resources are scarce and where available resources are changing In a multi-tasking environment, for example, processor availability varies with the number of concurrently executing tasks In these cases, the computer must utilize scarce resources more efficiently so that the user does not perceive a significant decline in the quality of the output Regulation is also important to achieve graceful presentation across a variety of platforms Systems supporting interactive animation and integrated media should adapt to different platforms to exploit additional computing power where available and to maintain quality even when such resources are not available Synchronization refers to how the computer coordinates the timing of different media streams Synchronization problems arise when attempting to integrate different media types because each media type is handled independently at the system software and hardware levels At the hardware level for example, each media type is typically controlled by a separate peripheral with its own clock At the software level, each media type has its own programming interface, which typically does not integrate media types Thus, it is very difficult to synchronize different media streams To the user, the lack of synchronization results in strange effects such as an animation stopping a few seconds before the audio intended to accompany it also stops

In the context of multimedia and interactive animation, latency refers to the delay between an input from the user and the output of media streams in response to that input To produce realistic and truly interactive output, the computer must process inputs as quickly as possible and create the desired output in a smooth and continuous fashion For example in a game where a user attempts to shoot an enemy spaceship, the user should experience a nearly simultaneous and continuous response from the display and audio speakers so that no delay is perceived between pressing the trigger and seeing and hearing a response Regulation, synchronization and latency issues impact the user, the application developer, and the system developer From the user's perspective, the failure of the computer to adequately address these issues results in poorly integrated output

From the application developer's perspective, there is a lack of programming interfaces or modeling languages that assist him or her in developing applications with integrated media types The application developer usually has to deal with a number of disparate media subsystems each controlling a separate media type

As a result, the application developer has to write code that addresses regulation, synchronization and latency This is difficult or even impossible for programmers to achieve Application programs that attempt to perform regulation and synchronization tend to be long and complex The difficulty in creating applications with integrated media also precludes many authors and artists from creating realistic animation and graphics because they do not have the programming expertise At the system level, system developers are confronted with the problem of providing an apphcation interface to each of the disparate media types such as sound, 3D graphics rendering video, etc The system level software includes a variety of device drivers that communicate with disparate hardware devices, each supporting a separate media type Preferably, the system should abstract the details of the underlying hardware and provide high level functions so that the programmer does not have to write code to control specific operation of the hardware

While progress has been made in application programming interfaces for different media types, the problem of successfully integrating different media types at the system level is still an elusive goal Present software architectures and programming techniques do not adequately solve the problems of integrating media types Advances have been made in developing software development tools to create interactive animation However, most interactive animation is created using imperative programming techniques, which have a number of limitations

In imperative programmmg, the programmer sets forth a sequence of operations telling the computer how to carry out a desired task In the context of animation, this can include specifying how the computer is to generate each frame of animation Obviously, this can be an onerous burden on the developer because it requires him or her to specify a significant amount of detail regarding how each frame is to be rendered

As alluded to above, the graceful presentation of media on the computer is left to the application developer due to the current lack of media support When issues such as regulation, synchronization, and low latency must be controlled in an application program written in an imperative language, the application tends to be very large and complex

Another drawback to an imperative programming approach is that it makes analysis and optimization of application code difficult Since the application provides more explicit detail as to how to create a desired effect, the system level software has less flexibility to determine more efficient ways to present the output using available resources of the computer Many existing programming environments for interactive graphics fail to provide effective and easy to use tools to represent integrated media types One significant limitation is the failure to support a continuous time model for media types Time is critical in interactive animation, and more generally in integrated media applications because media is inherently time varying Instead of representing continuous behavior, an imperative program must explicitly handle generation of discrete output such as discrete frames of animation

Another significant drawback is the failure to provide an integrated media development environment that results in a compact representation of media types A compact representation is a programming or modeling format that enables a user to express sophisticated, realistic, and interactive media without writing a long, complex piece of code Because of the characteristics outlined above, imperative programming tools do not produce compact representations Currently, efforts are underway to simplify the design issues at the application level and to provide developers and non-programmers the tools necessary to create interactive and realistic affects One proposed solution is to use a declarative modeling approach rather than an imperative programming approach In the declarative approach, the developer or author uses programmmg or modeling constructs to describe what behavior he or she wants to create This is in contrast to imperative programming where an experienced programmer gives a specific sequence of instructions describing how the computer must produce a desired output Aspects of a declarative approach for interactive media are descπbed in Elliott, C , Schechter, G , Yeung, R , AbiEzzi, S , "TBAG A High Level Framework for Interactive, Animated 3D Graphics Applications" Siggraph '94 Conference Proceedings, at 421-434 (Aug 1994), and Schechter, G , Elliott, C , Yeung, R , AbiEzzi, S ,

"Functional 3D Graphics in C++ - with an Object-Oriented, Multiple Dispatching Implementation," Proceedings of the Fourth Workshop on Eurographics Object-Oriented Graphics (EOOG '94)

TBAG proposes a solution which would enable the developer of an interactive animation to represent a continuous, and time varying behavior usmg abstract data types and explicit functions of time The declarative model discussed in connection with TBAG is a significant improvement over other alternative approaches, but it has limitations One of the limitations of the TBAG approach is that it does not support descriptions of media behaviors in terms of events

As noted above, there is an increasing demand for an application development environment that enables programmers and even non-programmers to create integrated media applications This demand is especially acute for applications designed for the Internet For years, the Internet was limited to electronic mail and file transfer services Now, the Internet is rapidly becoming a vast information and communication resource for millions of users With the development of graphical browsers and the World Wide Web, commercial exploitation is fueling tremendous growth Now, there is an increasing demand to support a variety of media types and to improve interactivity Images and simple two-dimensional graphics are common-place on the Internet However, support for animation, video and audio is still in its infancy

The Virtual Reality Modeling Language or "VRML" is one format that is currently being pursued Despite its name, VRML 1 0 only supports lifeless 3D models As such, there is a need for a much more powerful language for supporting interactive animation and for integrating a variety of media types including video, audio, images, etc

There are a number of important design issues that must be addressed to properly support interactive animation as well as other forms of media in Intemet applications The representation of media must be compact The creation of interactive animation is more powerful and easier to use if it can be expressed in a compact form A compact representation also consumes less memory and is easier to transfer over the Intemet

Security is also a major concem for Intemet applications The representation of media, therefore, must not be in a form that compromises the security of computers accessing the representation of interactive animation or other integrated media from the Intemet If an interactive animation is descπbed using an imperative program, the security of the computer receiving an executable version of the program can be adversely impacted

The representation of media and interactive animation must be platform independent In other words, the support for integrated media must be available to users on a wide variety of hardware platforms

As introduced above, the graceful presentation of media is critical This is especially true for Intemet applications where the limitation of a relatively low bandwidth data path still remains The issues of regulation, synchronization, and low latency are even more difficult for Intemet applications

SUMMARY OF THE INVENTION

The invention provides an improved method and system for modeling and presenting integrated media, and in particular, interactive animation One embodiment of the invention includes a declarative modeling language and computer-implemented system for presenting media described in declarative models Compact models representing integrated media or interactive animation can easily be created using a text editor or other authoring tool

The declarative models can express media and animation in terms of continuous time and discrete events In one embodiment, behaviors can be constructed from static data types such as geometry, sound, images, and transforms to name a few An author of a model can express behaviors in terms of time or functions of continuous time In addition, the author can describe how behaviors change in response to events Thus, models can include continuous time varying behavior and behavior that changes in response to discrete events

To present the media defined in a declarative model, the system maps the declarative model into an imperative set of commands for generating media In one implementation, a declarative modeling system parses the declarative model and creates static values and behaviors The system renders static values repeatedly to create animation To support reactivity, the system monitors events, and creates new behaviors in response to these events

The approach summarized here provides a number of advantages First, the declarative modeling language is easy to use, yet can be used to create complex models of natural media, modeled media, or a combination of both The declarative representation is compact, secure and platform independent, which makes it especially beneficial for Intemet applications

The declarative models are easy to analyze and optimize relative to imperative programming techniques The support for events enables the author of a model to express behavior over all time, not just between events This feature can lead to a number of optimizations because the model can be analyzed and events can be anticipated ahead of time This enables the system to take steps to reduce latency by performing tasks in anticipation of certain events that will occur in the future In addition, the system can perform temporal analysis on the model to determine how it changes over time Based on this analysis, unnecessary computation can be avoided

A higher quality media presentation can be achieved because regulation is handled at the system level rather than in the application The declarative model describes what to present, rather than how to present it As such, the system can determine how to regulate the use of computer resources to provide a graceful presentation, even under adverse conditions

Further advantages and features will become apparent with reference to the following detailed description and accompanying drawings

BRIEF DESCRIPTION OF THE DRAWINGS

Fig 1 is a block diagram of a computer system which serves as an operating environment for an implementation of the invention

Fig 2 is a diagram depicting a portion of the Internet

Fig 3 is a data flow diagram illustrating an example Reactive Behavioral Modeling Language (RBML) model of interactive animation

Fig 4 is a state diagram illustrating an example of reactivity m RBML

Fig 5 is a state diagram illustrating an example of cyclic reactivity

Fig 6 is a state diagram illustrating an example of hierarchical reactivity

Fig 7 is a functional block diagram illustrating the architecture of an embodiment of the invention

Fig 8A is a functional block diagram depicting an RBML engine in one embodiment of the invention

Fig 8B is a functional block diagram illustrating elevation in the RBML engine of Fig 8A Fig 9 is a flow diagram illustrating the interpreter the RBML engine

DETAILED DESCRIPTION OF THE INVENTION

Figure 1 is a block diagram illustrating a computer system 20, which serves as an operating environment for an embodiment of the invention Computer system 20 includes as its basic elements a computer 22, input device 24 and output device 26

Computer 22 generally includes at least one high speed processing unit (CPU) 28 and a memory system 30 that communicate through a bus structure 32 CPU 28 includes an arithmetic logic unit (ALU) 33 for performing computations, registers 34 for temporary storage of data and instmctions and a control unit 36 for controlling the operation of computer system 20 in response to instructions from a computer program such as an application or an operating system The CPU 28 may be any of a number of commercially available processors To name a few, the CPU can be a Pentium or Pentium Pro processor from Intel Corporation, a microprocessor from the MIPS family from Silicon Graphics, Inc , or the PowerPC from Motorola

Memory system 30 generally includes high-speed main memory 38 in the form of a medium such as random access memory (RAM) and read only memory (ROM) semiconductor devices and secondary storage 40 in the form of a medium such as floppy disks, hard disks, tape, CD-ROM, etc and other devices that use electrical, magnetic, optical or other recording material Main memory 38 stores programs such as a computer's operating system and currently running application programs Main memory 38 also includes video display memory for displaying images through a display device

Input device 24 and output device 26 are typically peπpheral devices connected by bus stmcture 32 to computer 22 Input device 24 may be a keyboard, modem, pointing device, pen, head tracker, data glove, or other device for providing input data to the computer Output device 26 may be a display device, modem, printer, sound device or other device for providing output data from the computer Output device may also include a graphics accelerator board or other graphics rendering device that operates in conjunction with the display to generate display images from two dimensional or three dimensional geometry

In accordance with the practices of persons skilled in the art of computer programming, embodiments of the invention are descπbed below with reference to acts and symbolic representations of operations that are performed by computer system 20, unless indicated otherwise Such acts and operations are sometimes referred to as being computer-executed It will be appreciated that the acts and symbolically represented operations include the manipulation by the CPU 28 of electrical signals representing data bits which causes a resulting transformation or reduction of the electrical signal representation, and the maintenance of data bits at memory locations in memory system 30 to thereby reconfigure or otherwise alter the computer system's operation, as well as other processing of signals The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, or optical properties corresponding to the data bits

One advantage of the invention is its suitability for applications on the Intemet Thus, before describing an embodiments of the invention in more detail, we begin with a brief overview of the Intemet Fig 2 is a diagram illustrating portions of the Intemet As is well known, the Intemet is a world-wide network of computer systems This network includes a number of specialized servers called Network Access Points 52-56 designed to provide access to a high speed network 58 Each of the Network Access Points communicate with other servers 60, 62 through the telephone network 64 There are a variety of possible links between end-users and the Intemet For example, if an Internet server 60 is at a local site, users (66a-b) at the site may access the Intemet by communicating with the server 60 through a local area network 68 Users can also access a remote Intemet server 62 by establishing a remote link to the server over a telephone line 72 using a modem These and other widely used communication techniques are well known and require no further elaboration here The primary inter-machine protocol on the Intemet is called TCP/IP This is a well known ITU (International Telecommunications Union) standard OSI (Open Systems Interconnect) protocol The TCP (Transmission Control Protocol) part is level 3 of the OSI model and the IP (Intemet Protocol) part is level 2 This common protocol enables computers to communicate with other computers all around the world

A typical sequence for accessing the Intemet proceeds as follows The user's computer 70 establishes a connection with its Intemet host server 62 After establishing a connection with the host server 62, the user's computer 70 launches a browser, a program for searching for, retrieving and viewing information on the Intemet and World Wide Web To retrieve information from the Intemet, the browser typically sends the path to a target server (74, for example) to its host server 62 The host server 62 retrieves requested data from target server 74 and passes it along to the user's computer 70 Examples of this data include Hypertext files, image files, graphics etc

As noted in the background section, there is a specific need to improve the way interactive animation is represented and presented for use in Intemet applications The approach to modeling interactive animation and other integrated media described below does provide an improved way of representing interactive animation for Intemet applications The approach is not limited to supporting interactive animation on the Intemet, however Below, we begin by describing a declarative modeling approach Then, we describe an implementation in more detail and discuss the use of this implementation for Intemet apphcations One embodiment of the invention provides a declarative modeling approach to interactive animation This approach includes two logical components 1) a declarative modeling language for the modeling of media types including interactive animation, and 2) a computer-based system for presenting the media through one or more output devices and for supporting user interactivity through one or more input devices The first component comprises a tool for expressing one or more media types in a declarative model The second includes software for mapping a model into an imperative format that can be executed by a computer In addition, the second includes software, a computer, and computer-related hardware to receive user input and produce output

The declarative modeling approach is particularly well-suited for Intemet applications for a number of reasons The representation of interactive animation in declarative models addresses a number of key design issues enumerated above Specifically, it is easy to use even for non- programmers It is a more compact and is more secure than imperative programming approaches The declarative approach improves the presentation of media as well It facilitates optimization and analysis It also facilitates regulation at the system software level to help ensure a more graceful presentation of interactive animation and integrated media The declarative modeling language in one embodiment is called the Reactive Behavior

Modeling Language ("RBML") RBML is a declarative modeling language that enables one to define a model of integrated media or interactive animation that will be valid over a period of time In a typical environment supporting RBML, an application program represents media types using one or more RBML models For example, an application program can present an interactive animation that is valid throughout the execution of the application This execution typically spans one or more discrete events as described below The underlying system software then uses this model to create the animation autonomously from the application In one implementation, this system software includes an RBML engine for inteφreting the RBML code and controlling the presentation of media, and media presentation libraries for carrying out the presentation This part of the system is described in more detail below

RBML can be used to integrate a vaπety of media including audio, video, 2D and 3D graphics, animation, and gestural input An RBML model provides a description of media content such as 3D geometry, sound, images, etc Because continuous time values can be represented in RBML, an RBML model can define how the media content behaves over time It can also define how the content behaves with respect to input to the computer such as mouse input or a key press, or through GUI controls such as a slider One of the conceptual building blocks in RBML is referred to as a behavior A behavior in RBML can be thought of as a value that is continuously varying through time For example, the value of a digital thermometer changes over time A model representing this thermometer in RBML would treat this as a number-valued behavior Another example of a behavior is the velocity of an airplane This is a vector value — three numbers that represent the direction and speed As the airplane banks, the velocity vector changes in real time In RBML, all values are time-varying in this way Numerical constants are just time-varying values that happen to remain constant

Interactive animation has the following properties "animation" referring to the motion of graphical objects over time, "interactive tracking" referring to tracking conceptual continuous input devices such as joysticks, mice, Graphical User Interface (GUI) sliders, and head trackers, and "event response" referring to response to events occurring in the animation or directly from the application or end user RBML allows all of these properties to be modeled using the concept of behaviors

There are a number of advantages that follow from the declarative modeling approach in RBML First, the system software that maps the model into an imperative sequence of instmctions can perform analysis and optimization For example, the system software can analyze the intent of the programmer over time as a prelude to optimization Data types that are usually considered static such as geometry, a vector, or color can be encapsulated with time varying behavior, stored transmitted, and mtegrated at n time

The declarative model simplifies the writing of applications that combine media types and is specifically useful in modeling interactive animation When using RBML to model an interactive animation, the application program does not need a frame loop to generate each frame of the animation Moreover, the application no longer requires an event loop or a web of callbacks to respond to events In RBML, behaviors can be written to respond to events There two types of behaviors in RBML non-reactive and reactive Non-reactive behaviors do not respond to events, but can support animation Reactive behaviors are a superset of non-reactive behaviors, and respond to events We begin with a description of non-reactive behaviors, and later, describe reactive behaviors in more detail One of the building blocks of a behavior is a modeling type RBML includes a variety of static modeling types including 3D geometry, discrete and continuous images, sounds, a 3D transform, a 2D transform, a color, point in 3D space, a point in 2D space, etc An RBML model may also include references to extemal components Rather than constructing a model entirely from basic primitives, an author of a model can build on material created by others, even if it is another format There is an enormous amount of raw material available today, both commercially and freely on the Intemet, that can be used as a starting point for constructing interactive animations This material is in the form of files in many different formats representing geometry, images, video, sound, animation, motion paths, etc RBML works with these representations directly, rather than requiring authors to create raw material specifically for RBML, or even converting existing material into a new format

Consider an example of a bust comprised of (a) a VRML 1 0 model of a cube, and (b) a 3DS model of Shakespeare's head An author can includes these models in an RBML model by means of "import", and name the results for later use

cube = impor ( "cube.wrl" ) head = import ("Shakespeare .3ds")

In addition to the static values described above, another building block of behavior is time Behaviors can be time varying by expressing them in terms of time or functions of tune A special case of non-reactive behaviors are constant behaviors, which maintain the same value for all time Examples include the static geometry value cube, and the value representing the unit y- vector, yVector

In RBML more complex models can be created through a concept called composition Composition involves the nesting or combination of behaviors to create new ones Consider the example below

bust = let base = scale (1,3,1) (cube) top = translate (0, 3 , 0) (head) m base union top

This is a simple example of a model of a graphical object representing a bust The bust is the "union" or combination of base and top behaviors In this example, the base is a cube scaled in 3D by the factors (1 , 3, 1), corresponding to x, y, and z directions in the (x, y, z) coordinate system Top is a graphical object called "head," which is translated by a factor of 3 in the y direction (0, 3, 0)

Each part of the model is constructed by starting with simple behaviors and building on them to create a more complex model For instance, the static cube is scaled to create a another static geometry called base The static geometry, head , is translated to create a new geometry called top Finally, the bust is constructed by combing the base and top with the "union" function In this manner, more sophisticated models can be constructed from more elemental ones As shown in this example, RBML enables the user to apply a behavior to another behavior or to combine behaviors to create new, more sophisticated behavior In RBML, naming and composition are completely independent, so the author is free to choose how much and where to introduce names, based on individual style, and intended reuse Naming is useful for making descriptions understandable, reusable and easy to remember, but can lead to a cluttering effect When intermediate models are named for the purpose of using in one or a few models only, they can interfere with choices of intermediate names used to help describe other models While this cluttering is not a problem with very simple models, described and maintained by a single author, it can become a serious obstacle as complexity grows and separately authored models are combined to work together

The solution to name clutter is to explicitly limit the scope of a name's definition In the example of a bust above, the bust definition is unscoped, but the scope of the base and top definitions is limited with the let and in constructs The author is free to reuse these names in other models without conflict with the base and top definitions in this model

Another way to create effects using RBML is to parameterize a model or a behavior with another behavior RBML behaviors may be defined in terms of a variety of different parameter types, including another RBML model The following example builds on the example above

bust (basecolor, headAngle) = let base = baseColor (scale (1, 3 , 1) (cube) ) top = translate (0, 3 , 0) (rotate (yAxis,headAngle) (head) )

base union top Here, the model of the bust is modified by parameters "baseColor" and "headAngle", which are behaviors defined elsewhere in the model Within the model of the bust, "baseColor" is applied to the scaled cube, which sets the color of the base to the color, "baseColor " "headAngle" is a parameter to the rotate function, which is apphed to the head to cause it to rotate about the y

Parameters can be constructed from expressions to create more sophisticated models For example, a parameter such as an angle, a color value, or a position can be expressed as a function of time As another example, a parameter can be expressed in terms of a conceptually continuous input device such as mouse In RBML, time varying behaviors are created by parameterizing models with time

Consider the following example

rotate (yAxis, 2 * pi * time) (head)

In this example, the angle parameter is expressed as a function of time When used in the model of a bust above, this behavior causes the head to rotate about the y axis as time progresses By allowing the author to parameterize the model with time or functions of time, the author can easily create animation without having to worry about generating each frame of the animation Instead, the system software processes the model and generates frames of animation transparently to the author

Input devices can also be represented in RBML models Input devices such as a mouse, a head tracker and a Graphical User Interface slider are each represented as conceptually continuous devices The user can create animation by using values produced by these continuous device models as parameters The examples below will help illustrate this concept

let hue = xComponent (mousePosition) angle = (pi/4) *sm(time) in bus (colorHsl (hue, 0.5, 0.5) , angle)

In this example, the RBML model includes two sources of animation the angle behavior is expressed as a periodic function of time, and the colorHsl() behavior is expressed as a function of a conceptually continuous input device The behavior "hue" is defined as a function of the x component of the position of the mouse (mousePosition) When used as a parameter to colorHsl(), hue controls the intensity value of one of the color components As the x position of the mouse varies, the value of hue changes, and the color of the bust changes accordingly.

The angle behavior is a periodic function of time. It begins at local time zero as the value zero and then varies between ±π/4 radians (i.e., ±45 degrees). It repeats every 2π seconds, the period of the sine function. The constant t ime represents local time in seconds. When used as a parameter to bust, the angle causes the head to oscillate back and forth.

Fig. 3 illustrates a data flow diagram of the last example. As shown in this diagram, the behaviors in RBML can be represented as a continuous data flow. The inputs to the data flow are time and mousePosition and the constants pi/4, and 0.5. The sine function (90) is applied to time to create a periodic function, and the periodic function is multiplied by pi/4 (92) to produce the angle behavior (94). The mousePosition is assigned to xComponent (96), which is a parameter of colorHsl (98) along with two constants, 0.5 and 0.5. The behaviors, angle (94) and colorHsl() (98) are passed to the bust. Data continues to flow according to this data flow graph as time progresses. The data flow graph represents the relationships among the behaviors in the model until either the model behavior terminates or an event occurs that causes a change in the data flow graph. As another example of time varying behavior, consider the following example comprised of 3D polygons representing a vessel, sai lBoat, centered at the origin. The angle behavior described above can be used to rock the boat:

heel = Rotate (zAxis, angle)

sailBoatl = heel (sailBoat)

The call to Rotate returns a geometric transformation that has been specialized by the axis and angle. The effect of applying heel to the boat geometry is that the boat begins upright and then heels to one side and then the other, passing through upright approximately every second and a half with the characteristic sinusoidal velocity.

In RBML, the time used to parameterize behaviors is not clock time but a local time line that starts at zero. This allows a behavior to be time transformed, e.g., time-shifted to start at any global (clock) time, and time-scaled to change its duration. Animation effects like slow-in and slow- out can also be expressed as time transformations. Basing behaviors on local time also allows them to be defined in a modular fashion, with each behavior conceptually in its own time line. Behaviors can then be used as the building blocks for more complex behaviors.

In addition to continuous behavior, RBML also provides for reactive behavior. While time in RBML allows a user to describe behavior that varies continuously over time, the support for reactivity enables the user to describe behavior that varies in response to discrete events. An event is something that can trigger a discrete change in behavior. In RBML, an event produces, at the time of the event, some data associated with that event that can be used in a new behavior after the event This feature allows events such as state changes and mouse clicks to be integrated with time-varying values

The following types of events are supported in RBML system events, Boolean events, handler events, and alternative events System events are events that originate from the operating system For example, in the Windows Operating System from Microsoft Coφoration, system events include LeftMousePress, RightMousePress, KeyPress, etc These types of system events can be represented as RBML events in an RBML model

A Boolean event is an event that occurs in response to a transition of a Boolean condition from false to true The following are example Boolean behaviors

In this example of RBML code, an event occurs when the Boolean expression evaluates "true" In other words, if x is less than or equal to a minimum value (minx) and the derivative of x (dx) is less than zero, than an event occurs

Handler events enable the user to create more complex events from other events If e is an α event and f is a function with type α->β, then e=>f is a β event Intuitively, this means wait for event e, and then take the data produced by that event and n the function f on it to produce something with type β This event happens at the same time that e did, and produces the new piece of data Consider the example

keyPress >= capitalize

This example in RBML means wait for an event "keyPress" and n the function

"capitalize" on the data produced by the event

Alternative events enable the user to specify a number of events from which the system will select based on which event occurs first The system then selects which event occurs first and returns the data corresponding to that event If e and e' are α events, then e ] e^* is an α event that means, intuitively, choose whichever event, e or e', happens first, and retum the correspondmg data for it If e and e' happen at the same time, it is undefined which will be chosen Consider the example

leftButtonPres s ! x < = minx

The system will monitor these events If leftButtonPress occurs first, the data associated with it will be returned If on the other hand, x<- minx occurs first, the system w ill return the data associated with it

In RBML, the unt i l construct is used to form reactive behaviors For example, if b is an α behavior and e an α event, then

b unt i l e

is a reactive behavior with type a behavior Intuitively, this statement means the behavior is b until the event e happens, and then becomes the behavior specified by the event data Here is a simple example model that mstmcts the system to display the color red until the mouse button is pressed, and then instructs it to display the color green

red until leftMousePress => green

As another example, the following expression produces a geometric transformation equal to the identity transform until a buttonDown event After the event, the system displays an animated rotation of π/2 radians, or 90 degrees, per second around the Z axis

myXform = identityTransform until buttonDown => Rotate(zAxis,

This transformation may be applied to animated geometry to create a reactive geometry, for instance

myGeom = myXform(VRMLModel ( "gesso.vrml" ) )

Reactive behaviors can be illustrated with state diagrams Consider the following example, where "LBP" means leftButtonPress

red unt il

LBP => green until LBP => yellow

Fig 4 is a state diagram illustrating this simple example of reactivity The state diagram begins at state 'red' (1 10) In response, to the leftButtonPress, the state transitions to 'green' (1 12) Finally, in response to another leftButtonPress, the state transitions to 'yellow' (1 14)

RBML also supports cyclic behavior using events

cyclic = red until

LBP => green until LBP => cyclic

Fig 5 is a state diagram illustrating the cyclic nature of this example The system transitions from a display of red and green color (1 16, 1 18) in response to each press of the left mouse button While this is a simple example, it illustrates the power of the model to express changes in behavior at a high level The previous example can be embedded in a hierarchical model as shown below

cyclic until

RBP => yellow

With addition of this RBML code, the behavior is cyclic until a leftButtonPress event In response to this event, the system displays the color yellow Fig 6 is another state diagram illustrating this hierarchical reactivity As shown in the state diagram, the cyclic behavior of Fig 5 is nested in a state (122) The system transitions from this state (122) to a state with behavior "yellow" (124) when the user presses the right mouse button

RBML also supports parametric reactivity Consider the following example

score (curr) = curr until scored => score (curr + 1) I curr=max => curr

In this example, the value of score(curr) is curr until the scored event occurs In response to this event, the value of curr is incremented This behavior repeats in response to the scored event This example also includes an alternative event, which occurs when the value of curr equals the value 'max ' When curr equals max, an event occurs which prevents further increments to curr

Parametric reactivity enables one to describe related reactive behavior in a compact form Rather than describing a sequence of reactive behavior with separate expressions, for example, one can define all of these behaviors at once as shown in the example above

Above, we illustrated the behaviors in an example of model using a data flow graph The system software generates this data flow when instructed to execute an RBML model When events occur, the system modifies the behaviors in the data flow graph that are affected by the event Consider the example depicted in Fig 3, modified as follows

angle = (pi/4) *sιn(time) until leftButtonPress =>

In this example, the data flow graph ts as shown in Fig 3 until the user presses the left mouse button At that point, the input to the sine function changes from time to time scaled by 1/4 (see 102 in Fig 3) To carry out this change in behavior, the system modifies the data flow graph in response to the leftButtonPres s event such that the input to the sine function becomes a new behavior, time scaled by 1/4 This example highlights a powerful feature of events in RBML the RBML model specifies behavior across all time, including what happens when events occur This feature enables the system that mteφrets and processes the RBML model to improve regulation and to optimize performance

Reactivity in RBML improves regulation because the system can determine ahead of time all possible events and predict reactive behavior before it occurs This enables the system to make more efficient use of computer resources The system can reduce latency by predicting events before they occur and taking steps to prepare for them For example, if an event causes a texture to be mapped to the surface of an object in a scene, the system can begin retrieving the texture from memory before the event occurs If an event requires that a video or sound file be decompressed and loaded from disk before playback, the process of decompressing and loading the data can be initiated ahead of time and latency can be reduced As another example, the system can reduce latency in establishing a network connection by foreseeing that it may occur and establishing a connection ahead of time A variety of other optimizations are also made possible because of the support for events in

RBML One optimization involves the use of temporal analysis to eliminate overhead in, for example, generating images in a graphics scene or generating sounds Since behavior can be predicted, the system can determine ahead of time which geometry will form part of a scene and queue it for rendering Temporal analysis can be used to reduce unnecessary computations such as rendering geometry that is not potentially visible in a scene, or rendering sound data that is masked by other sounds

Another useful construct in RBML is a "snapshot " The snapshot construct may be used to make a (dynamic, time varying) behavior, static (l e , no longer varying with time) This is useful when one wishes to record the instantaneous value of a time varying behavior for some other use For example,

0 until snapshot (LeftMousePress, xComponent (mousePosition) )

is the static behavior 0 until the left mouse button is pressed, and then becomes the static value of the x coordinate of the mouse position at the time of the press The behavior

0 until Lef MousePress => xComponent (mousePosition)

is different m that it produces a behavior that continues to vary with time and track the x coordinate of the mouse after the press event occurs

In RBML, the user can specify when a behavior terminates using the end construct The end construct is of type α (l e , it is any type) and it, intuitively, means immediately terminate the behavior For example,

let b = time until LMP => end

is a behavior that varies with time until the left mouse press event and then terminates If a behavior is defined in terms of other behaviors and one of those behaviors terminates, then the behavior itself terminates For example, consider

let b ' = f ( b , 3 ) If behavior b terminates, then b' terminates at the same time The constmct done is a unit event that can be used to detect that a behavior has terminated, and act accordingly Consider the following example

let repeat (b) = b until done => repeat (b)

This function takes a behavior b and mns it until it terminates, and then starts it again To improve modularity, all behaviors are defined to begin at local time zero in RBML

Consider

and assume b is the main behavior of the RBML model When b is performed the system begins at, for example, global time t_globll This is by definition, time = 0 for b The event e occurs when it happens the first time after t_globl, (1 e , the first time after its local time 0) Suppose this is at time t„_mt (as specified in global time) Then at global time t_^,, b' is performed starting with its local

Accordmg to global tune, this behavior was b until t_evm, and then became b' As for b, its time went from local time 0 to local time (t_event - t_{g oh},_t) and then b' was started with its local time 0 (assuming that no other time transformations are in effect)

Here is a simple example Consider the following behavior

green until

blue until time = 1 => red

This behavior is green for 2 seconds, and then blue for 1 second, and then red The t imeTransf orm constmct may be used to change the inteφretation of local time The function has type

t imeTransf orm : α- * number - > a

and it uses the number to redefine how time local time is inteφreted within a behavior Consider the following

doubleSpeed = tιme*2 b = playVideo (video)

doubleVideo = timeTransform(b, doubleSpeed)

This will result in a behavior that plays the video at twice its natural speed The perspective of global time, each 1 second interval corresponds to 2 seconds of local time The effects of time transformation are cumulative For example,

timeTransform(doubleVideo, doubleSpeed)

ends up with a behaviors that plays at 4 times the original video speed To be well defined, the number, n, argument for the time transform will be required to satisfy two mles 1 ) Monotone, for all times t₀ and t„ if t₀<t, then (n at time t₀₎ ) < ( n at time t,), and 2) Non-negative, For all times t, t, at t is non-negative

Monotonicity is required to make event reaction sensible (event transitions can not be undone) Time always increase to prevent trying to define a behavior before local time 0 (e g , it may not make sense to sample a behavior like a video before local zero)

Certain behaviors, principally those defined by system or user input devices, may not be transformed in time in the same way that synthetic behaviors can be Such devices ignore (or invert) the user-defined time transformation when being sampled The time transformation is actually a combination of the active time transformation and the start time RBML also supports integrals and derivatives on continuous, time varying behavior With

RBML, the author of a model can express time varying behavior in the form of expressions with integrals or derivatives on values that vary continuously over time For example, a variety of data types supported in RBML including real numbers, points, vectors, and quatemions can be operated on by derivative or integral functions To support the integral function, the RBML engine performs a forward-differencing algorithm to make numerical integration efficient To support the derivative function, the RBML engine can either use symbolic differentiation or can compute a derivative numerically using its mathematical definition

The author of a model can use the constmcts described above, to create descriptive models of animation or other media A typical model is comprised of a text file written in RBML To execute the model, this text file, or a compiled version of the model, is passed to system level software responsible for analyzing the model and mapping it into an imperative form controlling its presentation Fig 7 is a block diagram illustrating the architecture of an RBML system from authoring time to n time The diagram is separated into three sections authoring level, application level and system level

The authoring level of the system includes tools for creating and optionally compiling RBML source code files An authoring tool 150 is a program that enables a user to create RBML source code 152 This can include a simple text editor for creating RBML models, or more sophisticated graphical authoring tools Whether the author creates the model with a text editor or a graphical authoring tool, the format of the RBML source 152 at this level is a ASCII text file The compiler 154 shown in this diagram is designed to compile RBML source code into a form for fast processing As an alternative, the RBML source code can be inteφreted at n time We elaborate further on the inteφretive approach below

The system level portion of the diagram includes the RBML engine 158 and media presentation libraries 162 The RBML engine 158 is responsible for reading and inteφreting (if necessary) the RBML code and controlling the execution of the RBML model During execution of the model, the RBML engine 158 controls the presentation of the media described in the RBML model and regulates computer resources including such functions as memory management, and control of media presentation under varying conditions Since the RBML engine controls media presentation for all models, regulation of the output can be centralized within it To maintain smooth, continuous output under varying loads or across different platforms, it is sometimes necessary to use graceful degradation techniques that reduce the overhead in presenting the media without unduly affecting the quality of the output Graceful degradation techniques that can be supported in the RBML engine include reducing the frame rate (straight forward given the continuous time model in RBML), reducing the spatial resolution, using geometric levels of detail, and using simpler audio filters To present media output, the RBML engine issues media commands 160 to media presentation libraries 162, which interface to the presentation hardware in the computer system The media presentation libraries include, for example, DirectDraw for image and video rendermg, DirectSound for audio rendering, and Reality Lab or Dιrect3D for geometry rendering Each of these libraries are media presentation Application Programming Interfaces (API) for a Windows Operating System environment from Microsoft Coφoration In the alternative, other media presentation libraries can also be used Other examples of media presentation libraries include OpenGL from Silicon Graphics, Inc , for graphics, XIL from Sun Microsystems, for video and still images Software interfaces for input devices such as the mouse or keyboard are typically provided with operating systems such as the Windows Operating Systems from Microsoft Coφ , Solans for UNIX systems from Sun Microsystems, and the Macintosh O S from Apple Computer Co The application level of the system can include one or more application programs

Application programs can communicate with the RBML engine 158 through an application programming interface This type of interface enables applications to access functionality provided by the RBML engine In one specific implementation, for example, applications communicate with the RBML engine by making calls to functions in a programming interface This interface includes functions to read, inteφret and display an RBML model

Another feature of this declarative modeling system is its support for integration of declarative models and imperative programs Inter-language integration enables application developers to write applications incoφorating both RBML models and non-RBML code For example, parts of an application can be written in an imperative programming language such as C or Visual Basic, while other parts can be implemented in an RBML model This feature allows the application developer to exploit the strengths of RBML for integrating time varying media and animation, and imperative programming for implementing a sequence of discrete operations In addition, integration allows developers to extend the functionality of RBML

Some examples will illustrate the concept of integration of RBML and non-RBML modules To enhance a user interface for a spreadsheet application, the developer may want to create a 3D model of a spreadsheet The animated interface is much easier to represent in RBML, while file manipulation functions are easier to implement in an imperative programming language The application, therefore is preferably implemented by integrating a user interface in an RBML model with the remainder of the application implemented using imperative code When a user interacts with the user interface such as clicking on a cell in the spreadsheet for example, an RBML event may occur that should be communicated to a non-RBML module This is an example where the RBML module transfers information to a non-RBML module Consider another example where a user interface control, implemented in Visual Basic, is used to initiate an animation described with an RBML model In this case, an event in the non-RBML module needs to be communicated to the RBML module Below we describe alternative approaches for integrating non-RBML and RBML modules in an application One method of inter-language integration involves conversion of an RBML event to a non-

RBML event One way to convert an RBML event to a non-RBML event is to invoke a callback function when an RBML event occurs and pass the event data as a callback argument During event monitoring, an event monitor invokes this callback function each time an RBML event occurs This integration method is illustrated in Fig 7 by the paths 168, 170 from the RBML modules to the non-RBML modules The RBML module communicates the event to the non-RBML module by invoking the callback function and passing the event data Thus, the inter-language communication is referred to as "events with data" as shown (168,170) Note that the RBML modules are shown as part of the RBML engine 158 to reflect that they are processed in it However, the RBML modules can be part of an application One specific use of this inter- language communication mechanism is convert a continuous behavior into a stream of discrete events with data To accomplish this, the developer creates an RBML event that occurs frequently and then combines this event with an RBML behavior At ntime, the event monitor in the RBML engine invokes a callback function when the RBML event is detected and passes the value of the behavior and the time with each event As events occur, the RBML engine generates a stream of discrete samples of the behavior along with time of each sample Since imperative programs typically operate on static values, this feature is one way to make static samples of time varying behavior available to an imperative module An imperative module can, for example, use the stream of static values to analyze an animation

Non-RBML events can be converted to RBML events using an RBML primitive event that can be constmcted from C or some other imperative language This primitive event can implemented as a C object At mntime, the RBML engine notifies the C object to start monitoring the event on the C side To trigger this event, a C module calls the C event object and passes it time and event data as arguments In response, the C object causes an event on the RBML side to occur The C event object provides a useful mechanism for enabling non-RBML modules written in C to trigger RBML events While this example specifically refers the C programming language, it applies to other imperative programming languages as well Another method for a non-RBML program to communicate with an RBML module is through behavior elevation Consider an example of inter-language communication between a C program and an RBML model Behavior elevation in this context refers to taking a C header file and generating a compiled RBML-callable module, together with an RBML header file The RBML header file is analogous to a C header file, but is refeued to in an RBML module via the RBML equivalent of the C/C++ #ιnclude statement In one embodiment, an elevator in the RBML engine elevates functions in C to analogous functions callable from RBML This elevation is descπbed further below

In addition to elevating non-RBML functions to behavior producing RBML functions, non- RBML functions can also be elevated into event producing functions As one possible example, assume the non-RBML function is one that either returns a value of some type α, or raises an exception, "noevent " This function can be elevated to an event in RBML by having it retum an event with data of type α To detect such an event, an event monitor in the RBML engine would invoke the original, non-RBML function on sampled arguments and then analyze the results If the function raises the "noevent" exception, no event has occurred If the function returns a value of x, for example, then an event has occurred with data having a value, x

Elevation enables developers to extend the functionality of RBML by creating non-RBML functions and then elevating them so that they operate on behavior instead of static values The original functions operate on static values to produce static values The elevated functions operate on behaviors and produce behaviors In some cases, however, there are drawbacks to the elevation approach One drawback to elevating imperative functions is that the RBML compiler cannot easily optimize the imperative functions To the RBML compiler, these functions are a black box, and as such, are difficult to analyze and optimize Elevation of imperative functions also tends to undermine the inherent security provided by a declarative model A declarative model is secure because it is not in machine executable form, but this is not necessarily true of imperative functions Finally, the elevated code is often not machine independent and therefore may not be readily usable across a variety of platforms Despite these issues, elevation is still a viable option to integrating non-RBML and RBML modules and extending the functionality of RBML

Fig 8A is a functional block diagram illustrating the RBML engine in more detail Fig 8A also illustrates an example of how the RBML engine interacts with an RBML viewer As shown, one possible source of an RBML model is from an application program, which in this case is an RBML viewer In this embodiment, the RBML viewer is implemented as an OCX package, and can reside in the IExplorer browser from Microsoft Coφoration as well as other OLE containers

In this example, the process of executing an RBML model begins when the viewer supplies an RBML model for evaluation (200) The inteφreter 202 reads the RBML model and creates reactive behaviors (204) and static values (206) To accomplish this, the inteφreter parses the RBML model and creates an abstract syntax tree These and other functions of the inteφreter are described further below in connection with Fig 9

After the inteφreter creates the abstract syntax tree, the RBML engine builds a behavior graph (204) The behavior graph is a functional graph comprised of the behaviors in the mode! The RBML engine may also identify static values in the abstract syntax tree and constmct these static values as shown (214) Identifying and processing static values at this pomt is an optimization based on the observation that parts of the abstract syntax tree represent static values, which are not time varying Since the static values do not vary with time, they do not require sampling Behaviors, on the other hand, are time varying and must be sampled to generate static values

In this implementation, the RBML engine has an application programming interface including functions such as read script, inteφret, display, etc In response to a call to the display function, the RBML engine starts the ma reactive behavior of the model (216) At this point, there are two logical threads, although the implementation may actually use fewer or more process threads These logical threads include the event monitoring and response thread (208), and the performance sampling thread (210)

The event monitoring and response thread monitors discrete events such as interrupt dπven events, mouse clicks or keypresses, and timer events (218) It also monitors predicted, polled, and hybrid events A predicted event is an event where the time of the event is known before it occurs For example, tιmer(5) (a five second time) will cause an event five seconds from its start time An example of a polled event is a Boolean behavior, which may become true upon testing A hybrid event is an event composed of event types, such as an event combining the "and" and | (alternative) operators

In response to some events, the event monitoring and response thread invokes handlers These handlers either cause existing behaviors to generate new performances or cause existing RBML handler code to create new behaviors The first type of response is reflected by the arrow back to block 206 where an existing behavior generates a new performance In this case, the RBML engine creates a new performance subgraph and adds this subgraph to the existing performance graph We explain the performance graph in more detail below The second type of response, where existing RBML handler code creates a new behavior, is illustrated by the arrow back to block 202 In this case, the inteφreter processes the RBML handler code and generates a new behavior or behaviors

The second logical thread is the performance sampling thread The RBML engine creates a performance graph from the behavior graph (210) The RBML engine creates a performance by taking a behavior and a local start time and mapping the behavior to a global time When an event occurs that initiates a reactive behavior, the RBML engine takes the start time of the event and creates a performance

The performance sampling thread supports continuous animation and interaction through the sampling of time from continuous input devices One of these input devices is the system clock, which controls the progression of time represented in an RBML model Other input devices relate to user interaction and include, for example, the position of the mouse, the position of a head tracker, or the position of a GUI slider

The RBML engine samples behaviors to create static values A performance of a type α yields a value of type α upon sampling As a replacement for the traditional frame loop, performance sampling takes conceptually continuous behavior and produces discrete samples The result of this sampling is the construction of static values such as a geometry, an image, sound, a transform, color, etc The RBML engine produces time varying output by applying static operations to static values over and over

Static values are primarily built from primitive values and composition as explained above Primitive values include geometry, images, sound, transforms, colors, points, etc In constructing static values, the RBML engine sometimes creates a static value graph A static value graph is a data structure setting forth the components of a static value and describing the relationship among the components The static value graph is especially useful in describing composite static values For example, consider the example of a bust above, which is the union of two pieces of geometry A static value graph for this example specifies how to aggregate the components of the bust duπng rendering

In this implementation, static values are implemented as objects using an object oriented programming approach Objects for the static value types have a rendering method, which the RBML engine invokes to render static values or composite static values To render a static value for a single primitive, for example, the RBML engine calls the render method for that primitive and the object renders itself

An object composed of two or more static values renders itself by making repeated calls to the appropriate rendering method to render the objects within the composite object By invoking the render method on a composite object, the RBML engine automatically triggers the rendering of other objects As an example, consider the model of the bust which includes two elements of type geometry combined with the "union" operator The RBML engine invokes the render method on the composite object representing the bust The "union" operator causes the render method to be invoked twice once to render the head the base, and a second time to render the base

The RBML engine controls the generation of output by making calls to the media libraries to render the static values In this implementation, the RBML engine includes a program entity called a rendering context that accumulates state information during rendering The rendering context can include a variety of state information used to render a static value or values such as a drawing context, a bounding box of geometry being rendered, color applied to a set of geometric objects, or an accumulation of geometric transforms to be applied to geometry When a static value of type geometry renders itself, it uses the state information in the rendering context In this implementation, the RBML engine creates separate rendering contexts for geometry (224), image/video (226), and audio (228) This is only one possible implementation, it is also possible to have a single rendering context for all static value types

While rendering the static values, the RBML engine makes calls to media APIs that assist in rendering and control the output of media on output devices In this implementation, there are three separate media APIs including a geometry API 230, an image/video API, and an audio API The geometry API controls the rendering of a 2D and 3D graphical objects The image/video API controls the rendering and display of images or a sequence of images The audio API controls the playback of one or more sound files As noted above, there are a number of commercially available media APIs The media APIs used m this implementation include the DirectDraw API for video and image rendering, the DirectSound API for audio rendering, and the Reality Lab or Dιrect3D API for geometry rendering The DirectX APIs are described in the DirectX SDK from Microsoft Coφoration

The RBML engine performs efficient memory management using a transient memory heap It is not possible to determine before rendering how much memory should be allocated to render each frame of animation As such, the RBML engine must allocate memory dynamically Unless memory is reclaimed, the RBML engine can n out of memory The RBML engine addresses this problem by dynamically allocating memory with a transient memory heap As the RBML engine renders objects for a frame, it adds to the transient memory heap and adjusts a pointer to free, allocated memory After a frame is generated, the RBML engine resets the pointer and reclaims the memory allocated in the transient heap for the previous frame

The transient heap is not used in all cases, however In some cases, some behaviors remain constant from frame to frame To reduce computational overhead, these behaviors need not be re¬ processed for every new frame Instead, the static values representing these behaviors can be saved and re-used In this case, the RBML engine allocates memory for constant values (values that do not change from frame to frame) using the system heap

The RBML engine can make calls to other applications running in the same or another computer To provide better security, this implementation of the RBML engine calls functions of other client applications that are registered with it (222) The example of the Intemet browser and the RBML viewer will help to illustrate this concept Consider an example where the RBML engine needs to execute a hyperlink to another Web page or has to fetch data from the Intemet In the first case, the RBML viewer registers a hyperlink function with the RBML engine The RBML engine can then make a call to the hyperlink function where necessary In the latter case, the browser registers a function to fetch Intemet data with the RBML engine Then, if the RBML engine needs to fetch data from the Intemet, it makes a call to the function for fetching the data in the browser application

With respect to Intemet applications, it is also important that the media representation not compromise the security of computers that access and present the media described in a model or program RBML models are inherently secure because they can be transmitted over the Intemet in the form of an ASCII text files For example, one common use of an RBML model is to embed an animation in a Web page Computers on the Intemet can retrieve and view the animation represented as an RBML model usmg the Intemet Browser to fetch the model from a remote server and the viewer to initiate presentation of the animation The RBML model, if an ASCII text form, cannot corrupt computers that access it because it does not include executable code that impacts computer resources such as instmctions to write to the computer's memory In contrast, however, an imperative program representing a similar animation can potentially impact security because it is likely to have executable code that manipulates memory Thus, RBML models are more secure than other proposed formats for representing integrated media and interactive animation that are based on imperative programming approaches Fig 8B is a functional block diagram of the RBML engine, including functional blocks illustrating elevation in this implementation As described above, elevation refers to taking functions that operate on static values and generating functions that operate on behavior valued parameters As shown in fig 8B, functions the operate on static values can be elevated to operate on performance or behavior valued parameters To elevate to a performance, the elevator takes the function declaration (254) and creates a new function (252) that has the same number of arguments but operates on performance valued parameters instead of static values Similarly, to elevate to a behavior, the elevator takes a function declaration and creates a new function (250) that has the same number of arguments but operates on behavior valued parameters In one specific implementation, the elevator creates a new function with performance or behavior valued parameters and wraps it with a performance or behavior object As shown in Fig 8B, the elevated behaviors can then be understood by the RBML inteφreter 202

Consider as an example static value declarations m a C header file The elevator in the RBML engine takes the header file and creates elevated versions of the functions listed in it For example, the elevator can elevate a C function that operates on a number and returns a string by producing a function that operates on a number of type behavior and produces a string of type behavior This new behavior valued function may be called from within an RBML model Fig 9 is a flow diagram illustrating the operation of the inteφreter in one implementation of the RBML engine The process begins by performing lexical analysis and parsing the RBML model (300) Next, the inteφreter constructs an abstract syntax tree (302) The inteφreter then performs typechecking and overload resolution on the syntax tree (304)

In the next step (306), the inteφreter performs optimizations on the syntax tree One form of optimization used in the inteφreter is dynamic constant folding In the creation of a frame of animation, the RBML engine traverses every node in the syntax tree and calls functions on the static values Since some of these static values often do not change from frame to frame, RBML engine can improve performance by caching results that do not change from frame to frame The process of identifying and caching non-changmg results is referred to as dynamic constant folding To support dynamic constant folding, time varying behavior values keep track of whether they are changing If a behavior is not changing, data about its lack of change is percolated up a functional graph until a non-constant behavior value is encountered Everything up to that non- constant value is constant, and can thus be evaluated once and have the static value cached in that node When the inteφreter asks for a value of a node that is non-changing, the cache is returned

When a behavior value transitions from being constant to being animate again, or when it changes to a different constant value, the cache is invalidated

Another optimization performed by the inteφreter is referred to as stmcture sharing In stmcture sharing, multiple instanced nodes are only evaluated once This form of optimization reduces overhead in evaluating a model because efforts expended in parts of the model can be re-used rather than repeated Since the RBML engine generates animation by processing values that are free of side-effects, the inteφreter can optimize the functional graph using stmcture sharing as well as other graph optimization techniques

After completing graph optimization, the RBML engine proceeds to evaluate the RBML model as set forth above (306) While Fig 9 depicts graph optimization as a step preceding evaluation, it should be understood that the RBML engine can perform additional optimization and analysis throughout the evaluation process

The reactive behavior model makes very careful use of state and supports operations that are side-effect free Unlike traditional approaches, an animation in this embodiment does not have to be achieved via side-effecting (modifying state) Rather, it can be achieved conceptually via computing new values from previous ones This value based semantics lends itself to referential transparency, which permits reducing models into much simpler and more efficient forms through program transformation techniques Examples of this include constant folding and stmcture sharing as described above

Furthermore, the continuous time model and the fact that reactive behaviors represent a complete description of how an entity behaves in relation to time and in reaction to events, lends itself to temporal analysis of the behavior for optimization One aspect of temporal analysis is exploiting temporal coherency, where it becomes possible to constmct a frame by using incremental operations based on the previous frame, rather than generating it from scratch

Another aspect is the prediction of events before they happen This lends itself to priming the system in order to achieve a low latency reactivity to the event Possible techniques for prediction include analysis of rates of change and derivative bounds, and interval arithmetic For example, when a bullet is approaching a wall, the system can predict the explosion, which allows pre¬ loading a corresponding texture into main memory Also there is the possibility of determining that an event is not going to happen for a certain period of time, and hence stop checking for it until that time For example, if a meteor is approaching the viewing window, and the system determines based on derivative bounds that it is not going to enter for at least five units of time, then the testing for the event will not need to happen until five units of time pass by

Automatic regulation is a necessity to insure the graceful presentation of common content on varying platforms This is particularly important for Web content and in light of the wide variations of computer platforms on the Intemet In t ly interactive and realistic animation the user should feel like a real time participant of the system Therefore, the presentation of an interactive animation is a task whose correctness (like synchronization and smoothness) places time-critical needs on the delivery system

The architecture of the declarative modeling system described above is well-suited for addressing regulation issues The RBML engine can deal with the fluctuations in load on a particular computer, and the differences in capacity between different computers and still deliver a smooth presentation of the content, albeit at varying levels of quality Graceful degradation techniques include reducing the frame rate (straight forward given our continuous time model), reducing the spatial resolution, using geometric levels of detail, and using simpler audio filters

In addition to regulation, the architecture is also especially well-suited for time management, temporal optimizations, and low latency reactivity to events The RBML approach can factor all these mechanisms into a single engine that resides at the client side, while the content is reduced to the essence of the interactive animation

All of the advantages of the RBML system also apply to shared spaces with multi-user participation The modeling approach in RBML for these shared spaces, with the temporal and event aspects and the disciplined use of state, makes it straight forward to instill the shared experience to different distributed clients with their own viewers and with appropπate regulation

While we have described the invention in detail with reference to specific embodiments, the concepts of the mvention can be implemented in a variety of ways without departing from the scope of the invention For example, RBML models may either be compiled or inteφreted RBML models can be compiled into machine dependent object code or optimized virtual machine code (machine independent object code) Alternatively, RBML models can be inteφreted The operation and stmcture of the RBML engine can vary without departing from the scope of the invention A declarative modeling system designed according to the invention can be implemented in a stand-alone computer system, or in a distributed computer environment such as a local area network or on the Intemet

In view of the many possible embodiments to which the principles of our invention may be put, we emphasize that the detailed embodiments described above are illustrative only and should not be taken as limiting the scope of our invention Rather, we claim as our invention all such embodiments as may come within the scope and spirit of the following claims and equivalents to these claims

Claims

We claim

1 A computer implemented system for modeling and presenting interactive media comprising a declarative model stored on computer readable medium including a description of media as a continuous function of time, and discrete events which alter the presentation of the media, a declarative modeling engine operable to read the declarative model and to create a functional data graph representing the model, operable to control the presentation of the media through evaluation of the functional data graph, and operable to monitor the discrete events and update the functional data graph in response to the discrete events

2 A computer implemented method for presenting integrated media compnsmg reading from memory a declarative model of interactive animation including a description of media as a function of time, and a discrete event which alters the presentation of the media, parsing the declarative model to create a functional data graph, evaluating the functional data graph to identify a first set of the media that requires rendering, monitoring for the discrete event, in response to detecting the event, modifying the functional data graph, evaluating the modified functional data graph to identify a second set of the media that requires rendering, and generating rendered media including displaying a sequence of frames of animation on a display device

3 The method of claim 2 including analyzing the functional data graph to identify time varying behavior that has not changed between a first and second frame of animation, and caching rendered media in memory that has not changed between the first and second frame

4 In a programmed computer system for presenting integrated media, a method for regulating the presentation of the mtegrated media comprising reading a declarative model including a description of media in terms of continuous time varying behavior, and discrete events which alter the presentation of the media, at a first system time, analyzing the declarative model to predict an event from the discrete events programmed to occur at a second, later system time, based on the predicted event, performing preparatory actions to achieve low latency reaction to the predicted event

5 The method of claim 4 wherein the performing step includes retπeving a sound file from memory

6 The method of claim 4 wherein the performing step includes retrieving an image or texture file from memory

7 The method of claim 4 wherein the performing step includes retrieving one or more frames of video from memory

8 The method of claim 4 wherein the performing step includes establishing a network connection

9 A computer readable medium on which is stored a declarative modeling language model for representing interactive animation, said model comprising declarative instmctions, which instruct a computer to perform the steps of creating a first set of static values from a continuous, time varying expression of behavior in the model, rendering the first set of static values to generate a first set of frames of animation for display, monitoring for discrete events specified in the model, in response to detecting one of the discrete events, creating a second set of static values, and rendering the second set of static values to generate a second set of frames of animation for display

10 In a method of transmittmg an animated display sequence over a network to a user, and displaying the sequence to the user, an improvement comprising transmitting a textual declarative model defining behavior of an object to be displayed durmg said sequence over the network, said model including a description of animation as a continuous function of time and at least one event affecting behavior of the object 1 1 In a programmed computer system for presenting integrated media, a method for regulating the presentation of the integrated media compnsing reading a declarative model including a description of media in terms of continuous, time varying behavior, and discrete events which alter the presentation of the media, analyzing the declarative model to determine when a first event will occur, deferring evaluation of the first event to a later system time based on the analyzing step to avoid unnecessary calculation, evaluating the first event at the later system time, and displaying on a display device media associated with the first event 12 The method of claim 1 1 wherein the first event includes an intersection between first and second geometric objects represented in memory of the programmed computer system and the evaluating of the first event includes determining whether the first and second geometric ob_ject intersect