WO2013009085A2

WO2013009085A2 - Implementation method of user interface and device using same method

Info

Publication number: WO2013009085A2
Application number: PCT/KR2012/005484
Authority: WO
Inventors: 임성용; 차지훈; 이인재; 박상현; 임영권
Original assignee: 한국전자통신연구원
Priority date: 2011-07-12
Filing date: 2012-07-11
Publication date: 2013-01-17
Also published as: WO2013009085A3

Abstract

Disclosed are an implementation method of a user interface and a device using the same method. The implementation method of a user interface comprises a step for receiving AUI(Advanced User Interaction) pattern information and a step for interpreting the inputted AUI pattern information based on a predefined interface. Therefore, a preset AUI pattern generated in a user interaction device is implemented in various applications by defining the implementation method of an interface between a user interaction device and a scene description.

Description

How user interfaces are implemented and devices using these methods

The present invention relates to a method of implementing a user interface and an apparatus using the method.

User interaction devices have evolved in recent years. In addition to devices for interacting with users such as mice, keyboards, touch pads, touch screens, and voice recognition, new types of user interaction devices such as multi-touch pads and motion sensing remote controllers have recently emerged. Doing.

Although multimedia technologies have been studied to provide application technology for using advanced user interaction devices, current user interaction standards are mostly used for basic interactions such as pointing or keying that are used in existing electronic products. The situation is focused on devices.

There is no user interaction standard for evolving new types of user interaction devices such as multi-touch pads, motion sensing remote controllers and the like. In addition, there is no user interaction standard that can be applied to both basic interaction devices such as pointing or keying and the evolved new type of user interaction devices.

An object of the present invention is a method for implementing an Advanced User Interaction Interface (AUI).

In addition, another object of the present invention relates to an apparatus for performing a method for implementing an advanced user interaction interface (AUI).

According to an aspect of the present invention, there is provided a method for implementing a user interface, the method comprising: receiving AUI (Advanced User Interaction) pattern information and interpreting AUI pattern information received based on a predefined interface; It may include a step. The predefined interface may be an interface defining at least one pattern information among Point, Line, Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture. The predefined interface may be an interface in which at least one of the pattern information is selectively used as pattern information of a widget manager as predetermined predefined pattern information. The predefined interface may be an interface in which at least one of the pattern information is selectively used in a W3C (World Wide Web Consortium) application as predetermined predefined pattern information.

The user interface implementation method according to an aspect of the present invention for achieving the above object of the present invention by analyzing the input information provided from the scene description based on the step of receiving input information from the scene description and a predefined interface And inputting to a data format converter. The predefined interface may be an interface defining at least one pattern information of Point, Linem Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture. The predefined interface may be an interface in which at least one of the pattern information is selectively used as pattern information of a widget manager as predetermined predefined pattern information. The predefined interface may be an interface in which at least one of the pattern information is selectively used in a W3C (World Wide Web Consortium) application as predetermined predefined pattern information.

A user interface device according to an aspect of the present invention for achieving the above object of the present invention is a data format conversion unit for generating AUI (Advanced User Interaction) pattern information and an AUI (Advanced User Interaction generated by the data format conversion unit) ) May include an interface unit for interpreting pattern information based on a predefined interface. The interface unit may be an interface unit that defines at least one pattern information of Point, Line, Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture. The interface unit may be an interface unit in which at least one of the pattern information is selectively used as pattern information of a widget manager as predetermined predefined pattern information. The interface unit may be an interface unit in which at least one of the pattern information is selectively used in a W3C (World Wide Web Consortium) application as predetermined predefined pattern information.

As described above, according to the method for implementing a user interface according to an embodiment of the present invention and an apparatus using the method, a predetermined AUI (Advanced) generated in a user interaction device is defined by defining a method for implementing an interface between the user interaction device and a scene description. User Interaction) pattern can be implemented in various applications.

1 is a conceptual diagram illustrating a high-level view between MPEG-U (ISO / IEC 2007) and MPEG-V (ISO / IEC 2005) according to an embodiment of the present invention.

2 is a flowchart illustrating a method of analyzing pattern information in MPEG-U Part 1 according to an embodiment of the present invention.

3 is a conceptual diagram illustrating an operation between a widget manager and a physical device according to an embodiment of the present invention.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of this invention is described concretely with reference to drawings. In describing the embodiments of the present specification, when it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present specification, the detailed description thereof will be omitted.

When a component is said to be “connected” or “connected” to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may exist in between. Should be. In addition, the description "include" a specific configuration in the present invention does not exclude a configuration other than the configuration, it means that additional configuration may be included in the scope of the technical spirit of the present invention or the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

In addition, the components shown in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is made of separate hardware or one software component unit. In other words, each component is included in each component for convenience of description, and at least two of the components may be combined into one component, or one component may be divided into a plurality of components to perform a function. Integrated and separate embodiments of the components are also included within the scope of the present invention without departing from the spirit of the invention.

In addition, some of the components may not be essential components for performing essential functions in the present invention, but may be optional components for improving performance. The present invention can be implemented including only the components essential for implementing the essentials of the present invention except for the components used for improving performance, and the structure including only the essential components except for the optional components used for improving performance. Also included in the scope of the present invention.

The ISO / IEC 2007 standard (MPEG-U) is a standard for the Information Technology-Rich Media User Interface, which includes three parts: Part 1: Widget, Part 2: Advanced user interaction (AUI) interfaced, Part 3: Conformance and reference software. Standardization is underway.

Widget is a tool that enables extended communication and can be defined as follows.

Widget: self-contained entity, with extensive communication capabilities, within a Rich Media User Interface; composed of a Manifest and associated Resources, including Scene Descriptions for the Full and Simplified Representations and Context Information.

An advanced user interface (AUI) is to provide a medium for transmitting and receiving information on an enhanced user interaction device (eg, a Motion Sensing remote controller, a multi-touch device, etc.).

Referring to FIG. 1, the MPEG-V part 5 105 is used as a description tool for information input to actual physical interaction devices 100 or output from physical interaction devices 100. The information may be used as a description tool for transmitting the information to the Data Format Convertor 120 or the Semantic Generator (UI Format Interpreter 110) to be described below.

The data format convertor 120 may perform data conversion on the physical interaction device data interpreted by the MPEG-V part 5 105 so as to be applied to a widget that is an MPEG-U part 1 125. have.

The Semantic Generator (UI Format Interpreter, 110) generates syntax elements to be applied to AUI, which is MPEG-U part 2 (115), to physical interaction device data interpreted by MPEG-V part 5 (105), or UI (User Interface) Format analysis can be performed.

The MPEG-U Part 2 115 may serve as an interface between the Semantic Generator (UI Format Interpreter 110) and the data format converter 120 or the Scene Description 130.

Syntax information or interpreted UI information generated by the Semantic Generator (UI Format Interpreter, 110) may be interpreted by the MPEG-U Part 2 115 and input to the Scene Description 130 or the data format converter 120. .

The MPEG-U part 1 125 defines a method of performing widget management (widget packging, communication and lifecycle management) based on the data converted by the data format conversion unit 120.

The scene description 130 may be defined as follows.

A scene description is a self-contained living entity composed of video, audio, 2D graphics objects, and animations. For example, a Scene Description can be a widget or a W3C application.

Referring to FIG. 1, data instances created in MPEG-U part 2 115 may be transmitted by a widget communication method according to MPEG-U part 1 125. According to an embodiment of the present invention, the Scene Description 130 may receive an event of the AUI pattern without information on the AUI device based on the input AUI pattern value.

A. URN of a predefined message interface

The following table shows Uniform Resource Names (URNs) for predefined message interfaces.

TABLE 1

<mw: interface type = "urn: mpeg: mpegu: schema: widgets: aui: 2011">

<!-

Detail description of interfaces to be transpoted

->

</ mw: interface>

Referring to Table 1, using urn “urn: mpeg: mpegu: schema: widget: aui: 2100”, a new URN for the message interface of AUI pattern information is defined and used for the widget manager.

Table 2 below shows the input interface.

TABLE 2

</ messageIn>

</ messageIn>

</ messageIn>

</ messageIn>

</ messageIn>

</ messageIn>

</ messageIn>

</ messageIn>

Table 2 defines various pattern information and information constituting the pattern information.

(1) Point may be defined as first pattern information. Point can represent a two-dimensional or three-dimensional geometric point in eucladian space. As information included in the Point, capturedTimeStamp, userId, x, y, z, and the like may be used.

The capturedTimeStamp represents time information at which the AUI pattern is recognized. In general, capturedTimeStamp may be expressed in millisecond units based on January 1, 1970, at 0: 0.

userId may indicate user information.

x, y, z may be input values used to indicate the location information of the point.

(2) Line can be defined as another message interface. Line is pattern information indicating a straight line connecting two points. In order to indicate the pattern information, the positional information on both ends of the straight line and optionally the time information, velocity and acceleration information at which the straight line started can be used.

As information for representing a line, capturedTimeStamp, userId, firstPositionX, firstPositionY, firstPositionZ, secondPositonX, secondPositonY, secondPositonZ, startingTimestamp, averageVelocity, and maxAcceleration may be used.

capturedTimeStamp and userId have the same definitions as in the aforementioned Point.

firstPositionX, firstPositionY, firstPositionZ can be the position of the first end point of the line, and secondPositonX, secondPositonY, secondPositonZ can be the position of the second end point.

startingTimestamp represents information on when the line started, averageVelocity represents average velocity information during line pattern generation, and maxAcceleration represents highest acceleration information during line pattern generation.

(3) Rect may be defined as another pattern information. Rect is a rectangular pattern constituting the four corner positions of the rectangle can be expressed based on two opposite corner positions or four corner positions.

As information for representing a Rect, capturedTimeStamp, userId, TopLeftPosition, BottomRightPosition, TopRightPosition, BottomLeftPosition, firstTimeStamp, secondTimeStamp, thirdTimeStamp, and fourthTimeStamp may be used.

TopLeftPosition gives information about the position of the top left corner of the rectangle, BottomRightPosition gives information about the position of the lower right corner of the rectangle, TopRightPosition gives information about the position of the top right corner of the rectangle, and BottomLeftPosition gives information about the bottom left position of the rectangle. Indicates.

firstTimeStamp is the time information when the first corner was created when creating a square pattern, secondTimeStamp is the time information when the second corner was created when creating a square pattern, thirdTimeStamp is the time information when the third corner was created when creating a square pattern, and fourthTimeStamp is When generating the square pattern, the fourth corner shows the generated time information.

(4) Arc may be defined as another pattern information. Arc represents an arc pattern, and may include position information of the start and end points of the arc, position information of the center of the circle, angular velocity, angular acceleration, and time information at which the arc pattern starts to be drawn.

Information for representing the arc may include capturedTimeStamp, userId, firstPositionX, firstPositionY firstPositionZ, secondPositionX, secondPositionY, secondPositionZ, centerPositionX, centerPositionY, centerPositionZ, startingTimeStamp, averageAngularVelocity.

firstPositionX, firstPositionY, and firstPositionZ are information indicating the starting position of an arc and are expressed based on two-dimensional or three-dimensional position information.

secondPositionX, secondPositionY, and secondPositionZ are information indicating the position of the end point of the arc and are expressed based on two-dimensional or three-dimensional position information.

centerPositionX, centerPositionY, centerPositionZ Information indicating a center point position of an arc is expressed based on two-dimensional or three-dimensional position information.

startingTimeStamp may be information indicating time information at which an arc pattern is generated, and averageAngularVelocity may indicate average angular velocity information during arc pattern formation.

(5) Circle may be defined as another pattern information. Circle represents a circle pattern and may be expressed based on center position information, radius information, and average angular acceleration information of the circle.

Information for representing a circle may include centerPositionX, centerPositionY, centerPositionZ, startingTimeStamp, and averageAngularVelocity.

centerPositionX, centerPositionY, and centerPositionZ are information for indicating the center position of the circle and may indicate the center position of the circle in a two-dimensional or three-dimensional plane.

startingTimeStamp may be information indicating time information at which the original pattern is generated, and averageAngularVelocity may represent average angular velocity information during the formation of the original pattern.

(6) SymbolicPattern may be defined as another pattern information. SymbolicPattern can recognize the user's motion information as a new symbol based on the size and position of the motion information.

CaptureTimeStamp, userId, PositionX, PositionY, PositionZ, size, and symbolType may be used as information for representing a SymbolicPattern.

PositionX, PositionY, and PositionZ are positional information on which SymbolicPattern is recognized, size may be used as information indicating size of SymbolicPattern, and symboltype as information indicating what kind of symbolic pattern is.

(7) TouchPattern may be defined as another pattern information. The TouchPattern may recognize the user's motion information as a new touch pattern based on the input duration, the number of times, the input movement direction, and the rotation direction of the motion information as the pattern information of the user's touch.

As information for representing the TouchPattern, capturedTimeStamp, userId, Position X, Position Y, Position Z, touchtype, and value may be used.

Position X, Position Y, and Position Z may indicate position information where a touch is generated, touchType may be used as information for information on the type of touch, and value may be used as information for containing additional information required according to the type of touch pattern.

(8) HandPosture may be defined as another pattern information. HandPosture describes the pose of the user's hand, and Posture represents the pose type of the hand.

As information for representing a HandPosture, capturedTimeStamp, userId, Position X, Position Y, Position Z, postureType, and chirality may be used.

Position X, Position Y, and Position Z may indicate position information on which the posture of the hand occurred, and postureType may indicate pose type information of the user's hand, and chirality may indicate whether the user's hand is left or right.

HandGesture may be defined as another pattern information. HandGesture represents information about the operation of the user's hand. CaptureTimeStamp, userId, gesturetype, and chirality may be used as information for representing a HandGesture.

Position X, Position Y, and Position Z may indicate position information on which a hand gesture occurred, gestureType may indicate motion type information of the user's hand, and chirality may indicate whether the user's hand is left hand or right hand.

The pattern information described above in Table 2 may optionally be used. That is, at least one pattern information of the above pattern information may be selectively used as the pattern information of the widget manager.

That is, the message interface of the AUI pattern described above in Table 2 may be optional, and only a part of the message interface of the AUI pattern may be used in actual implementation.

According to another embodiment of the present invention, the AUI pattern may be utilized in a W3C application. For example, the html page may also be implemented based on a user's motion based on a geometric pattern, a touch pattern, or a symbolic pattern.

In an embodiment of the present invention, an interface definition language (IDL) event definition for communicating with a W3C widget is disclosed.

B. Interface Definition Language (IDL) of AUI Patterns

Hereinafter, an embodiment of the present invention discloses event types, syntax elements, and syntax elements for implementing an interface for a W3C (World Wide Web Consortium) application as follows.

TABLE 3

interface MPEGAUIEvent: UIEvent {

typedef float fVectorType [3];

typedef sequence <fVectorType> fVectorListType;

typedef sequence <float> floatListType;

readonly attribute unsigned long long capturedTimeStamp;

readonly attribute string userId;

readonly attribute float averageVelocity;

readonly attribute float maxAcceleration;

readonly attribute string sType;

readonly attribute string chirality;

readonly attribute float fValue;

readonly attribute fVectorListType positions;

readonly attribute floatListType timeStamps;

};

TABLE 4

Referring to Table 4,

(1) Point can be defined as the first event type. Point can use capturedTimeStamp and Position as contextual information. capturedTimeStamp is information indicating the time at which the user's interaction was captured (capturedTimeStamp has the same meaning hereinafter, and thus description is omitted in other event types), and Position indicates 2D or 3D position information on screen coordinates.

(2) Line can be defined as another event type. As context information to define a Line, capturedTimeStamp, position, timeStamps, averageVelocity, and maxAcceleration can be used as syntax elements. position is 2D or 3D coordinate information of two points forming a line, timeStamps is starting time information on which a line is drawn, averagevelocity is average speed information while a line is drawn, and maxAccelation is maximum acceleration information while drawing a line.

(3) Rect can be defined as another event type. As contextual information for defining a Rect, capturedTimeStamp, position, and timeStamps can be used as syntax elements. position is 2D or 3D coordinate value information of a corner point forming a rectangle, and timeStamps means time information on which each corner is drawn.

(4) Arc can be defined as another event type. You can use capturedTimeStamp, position, timeStamps, averageVelocity, maxAccelation as contextual information to define Arc. position is the 2D or 3D coordinate information indicating the position where the arc was drawn and ended and the center of the virtual circle, timeStamps is the start time of the user's interaction, averageVelocity is the average acceleration while drawing the arc, and maxAcceleration is The maximum angular acceleration information during the arc drawing.

(5) Circle may be defined as another event type. As context information for defining a circle, capturedTimeStamp, position, timeStamps, averageVelocity, maxAccelaeration, and fvalue can be used. Position is the origin coordinate information constituting the circle, timeStamps is the time information to start drawing the circle, averageVelocity is the average angular velocity while drawing the circle, maxAcceleration is the average angular acceleration while drawing the circle, fValue is the radius information of the circle.

(6) TouchPattern may be defined as another event type. CaptureTimeStamp, position, sType, and fValue may be used as context information for defining the TouchPattern. position may be position information where a touch has occurred, sType may be a label indicating a type of touch, and fValue may be value information required in a touch pattern.

(7) SymbolicPattern may be defined as another event type. As contextual information for defining a SymbolicPattern, capturedtimeStamp, position, sType, and fValue may be used. position may be position information where a symbol is generated, sType may be label information specifying a type of a symbol, and fValue may be size information of a symbol.

(8) HandPosture may be defined as another event type. As context information for defining a HandPosture, capturedtimeStamp, position, sType, and fValue may be used. The position may be information on the location of the hand posture, the sType may be label information indicating the type of the hand posture, and the fValue may be size information of the hand posture.

Hereinafter, an embodiment of the present invention describes an interface implementation for generating scene description information based on data input from a data format changer, but on the contrary, implements an interface for receiving scene description information and transferring information to the data format changer. Also applies to and is included in the scope of the present invention.

Referring to FIG. 2, the converted data format is received from the data format changer (step S200).

As described above, operation information of the physical interaction apparatus interpreted through the MPEG-U Part 2 and the MPEG-V Part 5 may be input to the data format converter. The data format switcher may perform data format conversion in an information format that can be input to MPEG-U Part 1 with respect to the transmitted operation information of the physical interaction apparatus.

Information input to the data format changer is transmitted as a scene description via a predetermined interface (step S210).

According to an embodiment of the present invention, the predetermined interface may be MPEG-U Part 1. Information interpreted through MPEG-U Part 1 can be used in widget managers or W3C applications.

As shown in Table 2, the interface for the widget manager can be defined. Pattern information as shown in Table 2 and input information for expressing each pattern information may be generated. That is, some pattern information of Table 2 may be used as an interface for representing motion information in the widget manager.

In addition, as shown in Table 3, the IDL interface may be defined to transmit operation information to the W3C application based on the event information defined in Table 3.

Hereinafter, in the embodiment of the present invention, for convenience of description, it is assumed that the physical signal is directly input to the interface between the widget manager and the physical device, but the information input through the actual interface is additionally a signal generated from the physical device to the MPEG-V Part. 5 may be information generated through the interface and data format conversion unit.

Referring to FIG. 3, operation information (eg, information generated by the data format conversion unit 300) generated in the user interaction apparatus is input to the MPEG-U Part 1 interface 320. The input operation information is transmitted to the widget manager 340 to implement the operation of the widget. For example, when an operation indicating a specific part of the touch mat is performed, such motion information may be interpreted through the MPEG U Part 1 interface 320 to be displayed in the widget 340. That is, the operation information is input based on Table 2 defined in the interface, and the corresponding information is transmitted to the widget manager 340. The widget manager 340 controls the operation of the widget based on the operation information interpreted through the MPEG U Part 1 interface.

On the contrary, the MPEG-U Part 1 interface may receive information generated from the widget and transmit the received information to the data format conversion unit 300. That is, the reverse operation of the above-described operation may be performed to transmit information generated from the widget to the data format conversion unit 300.

As another embodiment, as described above, the same operation may be performed in the W3C application 340.

For example, operation information generated in the user interaction apparatus is input to the MPEG-U Part 1 interface 320. The input operation information is transmitted to the W3C application 340 to implement the operation of the widget.

Although described with reference to the embodiments above, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

Claims

Receiving advanced user interaction (AUI) pattern information; And
A method of implementing a user interface comprising interpreting AUI pattern information received based on a predefined interface.
The method of claim 1, wherein the predefined interface,
A method for implementing a user interface, the interface defining at least one pattern information among Point, Line, Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture.
The method of claim 1, wherein the predefined interface,
And at least one of the pattern information as predetermined predefined pattern information is an interface that is selectively used as pattern information of a widget manager.
The method of claim 1, wherein the predefined interface,
And at least one of the pattern information as predetermined predefined pattern information is an interface selectively used in a W3C application.
Receiving input information from a scene description; And
And interpreting the input information provided from the scene description based on a predefined interface and inputting the input information to a data format converter.
The method of claim 5, wherein the predefined interface,
A method for implementing a user interface, the interface defining at least one pattern information among Point, Linem Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture.
The method of claim 5, wherein the predefined interface,
And at least one of the pattern information as predetermined predefined pattern information is an interface that is selectively used as pattern information of a widget manager.
The method of claim 5, wherein the predefined interface,
And at least one of the pattern information as predetermined predefined pattern information is an interface selectively used in a W3C (World Wide Web Consortium) application.
A data format switching unit for generating AUI (Advanced User Interaction) pattern information; And
And an interface unit configured to interpret the advanced user interaction (AUI) pattern information generated by the data format conversion unit based on a predefined interface.
The method of claim 9, wherein the interface unit,
A user interface device which is an interface unit that defines at least one pattern information among Point, Line, Rect, Arc, Circle, SymbolicPattern, TouchPattern, and HandPosture.
The method of claim 9, wherein the interface unit,
And at least one of the pattern information as predetermined predefined pattern information is an interface unit selectively used as pattern information of a widget manager.
The method of claim 9, wherein the interface unit,
And at least one of the pattern information as predetermined predefined pattern information is an interface unit selectively used in a W3C (World Wide Web Consortium) application.