US RE42847 E1 Abstract The present invention relates to a method, apparatus and system for optimizing an expression tree (101,902,1102) for compositing an image. Such an expression tree (101,902, 1102) can comprise at least two nodes. Each node is either a graphical element (102,104) or image compositing operator ((103,104) and has a region of the image represented by the node (102,103,104). In the method, for at least one node in the tree, several steps are carried out. The region represented by the node (103,104) is compared to a region representation data structure, which is preferably a quadtree representation, corresponding to one or more regions represented by at least one other node. A determination is then made if the region represented by the node (102,103,104) is totally or partially obscured by the one or more regions. If the region represented by the node is at least partially or totally obscured, the expression tree (101,902,1102) is modified. Modifying the expression tree (101,902,1102) involves applying a clipping operator (58,59) to the node if the region represented by the node is partially obscured. If the node is totally obscured, either removing the node if the node is a graphical element (102, 104) or applying a predetermined set of node replacement rules in accordance with the image compositing operator if the node (103) is a image compositing operator.
Claims(32) 1. A method of optimising an expression tree, said expression tree for compositing an image and comprising at least three nodes, each said node of said tree being at least either a graphical element or a graphical operator, the method comprising, for at least one node in said tree, the steps of:
comparing a first region of said node to a second region derived from at least one other node anywhere in said expression tree;
determining if said first region is totally or partially obscured by said second region; and
modifying the expression tree if said first region is at least partially or totally obscured by said second region, to form an optimised expression tree in which an optimised part of said expression tree substantially represents unobscured portions of said first region,
wherein said steps are performed by means of a programmed computer.
2. The method as recited in
3. The method as recited in
removing the node; and
if the node has a parent node which has a graphical operator, selecting a node replacement rule from a predetermined set of node replacement rules in accordance with said graphical operator and applying said rule.
4. The method as recited in
if the parent node is an “over” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node;
if the parent node is an “over” graphicgraphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is an “in” graphical operator, removing the parent node and any subtrees branching off the parent node;
if the parent node is a “ratop” graphical operator and the current node is at a left branch of the parent node, removing the parent node and any subtrees branching off the parent node;
if the parent node is a “ratop” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is an “out” graphical operator and the current node is at a left branch of the parent node, removing the parent node and any subtrees branching off the parent node;
if the parent node is an “out” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is a “plusC” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node;
if the parent node is a “plusC” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is a “plusW” or an “Xor” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node; and
if the parent node is an “plusW” or an “Xor” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node.
5. The method as recited in any one of
6. The method as recited in
7. The method as recited in
8. A method of optimising an expression tree for compositing an image, said expression tree comprising a plurality of nodes, each said node being at least either a graphical element or a graphical operator, said method comprising the steps of:
traversing the expression tree node by node; and
determining at a current node if a first region of the image represented at said current node is obscured by a second region derived from at least one other node anywhere in said expression tree, and modifying said expression tree in the event that said first region of said current node is partially or totally obscured by said second region, to form an optimised expression tree in which an optimised part of said expression tree substantially represents unobscured portions of said first region,
wherein said steps are performed by means of a programmed computer.
9. The method as recited in claim ti, wherein said modityinigmodifying includes removing said current rode or replacing said current node with another node of the expression tree.
10. The method as recited in claim
78, wherein said modifying further includes clipping, or marking for clipping at a later time, said first region represented by said current node.11. A method of optimising an expression tree for compositing an image, said expression tree comprising a plurality of nodes, each said node comprising:
at least either a graphical element or a graphical operator and having a region of the image represented by said node, said method comprising the steps of:
traversing the expression tree node by node and at each current node comprising a graphical operator applying the sub-steps of:
(i) receiving a first region representation from a parent node;
(ii) passing to a first operand of said graphical operator a modified first region representation in accordance with a first predetermined modification rule for said operator;
(iii) returning to the graphical operator a second region representation of regions obscured by a sub-tree associated with the first operand;
(iv) passing to a second operand of said graphical operator a modified second region representation in accordance with a second predetermined modification rule for said operator;
(v) returning to the graphical operator a third region representation of regions obscured by a sub-tree associated with the second operand; and
(vi) determining, in accordance will a set rule for said graphical operator, a final region representation to be returned to the parent node to form an optimised expression tree in which said final region representation substantially represents an unobscured portion of the first region represented at the parent nodea region within which said current node is capable of obscuring other nodes in said expression tree,
wherein said steps are performed by means of a programmed computer.
12. The method as recited in
(a) where the graphic operator is an “over” or a “plusC” operator, the final region representation to be returned to the parent node is determined from a union of the second region representation and the third region representation;
(b) where the graphic operator is an “in” operator, the final region representation to be returned to the parent node is determined from an intersection of the second region representation and the third region representation;
(c) where the graphic operator is a “ratop” operator, the final region representation to be returned to the parent node is the second region representation;
(d) where the graphic operator is an “out” operator, the final region representation to be returned to the parent node is determined from a difference of the second region representation and a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node; and
(e) where the graphic operator is an “Xor” or a “plusW” operator the final region representation to be returned to the parent node is determined from a union of the second region representation less a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node and the third region representation less a region representation containing a bounding box of a node at a right subtree of the current node.
13. The method as recited in
passing substantially the first region representation as the modified first region representation in the event that the graphical operator is an “over”, “in”, “ratop”, “plusC”, “plusW”, “Xor”, or “out” (visit left operand first)” or alike operatorsoperator; and
if the graphical operator is an “out (visit right operand first)” operation, passing as the modified first region representation a union of the first region representation with the second region representation.
14. The method as recited in
passing substantially the first region representation as the modified second region representation in the event that the graphical operator is an “in”, “ratop”, “out”, “plusC”, “plusW”, or “Xor” or alike operatorsoperator; and
in the event that the graphical operator is an “over” operator passing as the modified second region representation union of the first region representation with the second region representation.
15. The method as recited in any one of
16. The method as recited in
a right operand of an “over” operator and the “over” operator node does not need to return an image representation to its parent node;
a left operand of an “in”, “plusC”, “plusW” or “Xor” operator and said operator node does not need to return an image representation to its parent node;
a right operand of an “in”, “plusC”, “plusW” or “Xor” operator and said operator node does not need to return an image representation to its parent node;
a left operand of an “out” or “ratop” operator and said operator node does not need to return an image representation to its parent node;
a right operand of a “ratop” operator;
a root of the expression tree;
an operand of an image warp, affine transformation or convolution operator; and
an operand of a colour transformation that does not preserve opaqueness or if said transformation node does not need to return an image representation to its parent node.
17. An apparatus for optimising an expression tree, said expression tree for compositing an image and comprising at least three nodes, each said node of said tree being at least either a graphical element or a graphical operator, the apparatus comprising:
means for, comparing a first region of said node to a second region derived from at least one other node anywhere in said expression tree;
means for determining if said first region is totally or partially obscured by said second region; and
means for modifying the expression tree in the event that said first region is at least partially or totally obscured by said second region, to form an optimized expression tree in which an optimized part of said expression tree substantially represents unobscured portions of said first region.
18. The apparatus as recited in
19. The apparatus as recited in
means for removing the node; and
means for selecting a node replacement rule from a predetermined set of node replacement rules in accordance with said graphical operator and applying said rule if the node has a parent node which has a graphical operator and the node is totally obscured.
20. The apparatus as recited in
if the parent node is an “over” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node;
if the parent node is an “over” graphicgraphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is an “in” graphical operator, removing the parent node and any subtrees branching off the parent node;
if the parent node is a “ratop” graphical operator and the current node is at a left branch of the parent node, removing the parent node and any subtrees branching off the parent node;
if the parent node is a “ratop” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is an “out” graphical operator and the current node is at a left branch of the parent node, removing the parent node and any subtrees branching off the parent node;
if the parent node is an “out” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is a “plusC” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node;
if the parent node is a “plusC” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node;
if the parent node is a “plusW” or an “Xor” graphical operator and the current node is at a left branch of the parent node, replacing the parent node with a right subtree of the parent node; and
if the parent node is a “plusW” or an “Xor” graphical operator and the current node is at a right branch of the parent node, replacing the parent node with a left subtree of the parent node.
21. The apparatus as recited in any one of
22. The apparatus as recited in
23. The apparatus as recited in
24. An apparatus for optimizing an expression tree for compositing an image, said expression tree comprising a plurality of nodes, each said node being at least either a graphical element or a graphical operator, said apparatus comprising:
means for traversing the expression tree node by node;
means for determining at a current node if a first region of the image represented at said current node is obscured by a second region derived from at least one other node anywhere in said expression tree; and
means for modifying said expression tree in the event that said first region of said current node is partially or totally obscured by said second region to form an optimized expression tree in which an optimized part of said expression tree substantially represents unobsuredunobscured portions of said first region.
25. The apparatus as recited in
26. The apparatus as recited in
27. An apparatus for optimizing an expression tree for compositing an image, said expression tree comprising a plurality of nodes, each said node comprising at least either a graphical element or a graphical operator and having a region of the image represented by said node, said apparatus comprising:
means for traversing the expression tree node by node, said traversing means for each current node comprising a graphical operator, further comprising:
means for receiving a first region representation from a parent node;
means for passing to a first operand of said graphical operator a modified first region representation in accordance with a first predetermined modification rule for said operator;
means for returning to the graphical operator a second region representation of regions obscured by a sub-tree associated with the first operand;
means for passing to a second operand of said graphical operator a modified second region representation in accordance with a second predetermined modification rule for said operator;
means for returning to the graphical operator a third region representation of regions obscured by a sub-tree associated with the second operand; and
means for determining, in accordance with a set rule for said graphical operator, a final region representation to be returned to the parent to form an optimized expression tree in which said final region representation substantially represents an unobscured portion of said first region represented at the parent nodea region within which said current node is capable of obscuring other nodes in said expression tree.
28. The apparatus as recited in
(a) where the graphic operator is an “over” or a “plusC” operator, the final region representation to be returned to the parent node is determined from a union of the second region representation and the third region representation;
(b) where the graphic operator is an “in” operator, the final region representation to be returned to the parent node is determined from an intersection of the second region representation and the third region representation;
(c) where the graphic operator is ana “ratop” operator, the final region representation to be returned to the parent node is the second region representation;
(d) where the graphic operator is an “out” operator, the final region representation to be returned to the parent node is determined from a difference of the second region representation and a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node; and
(e) where the graphic operator is an “Xor” or a “plusW” operator the final region representation to be returned to the parent node is determined from a union of the second region representation less a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node and the third region representation less a region representation containing a bounding box of a node at a right subtree of the current node.
29. The apparatus as recited in
passing substantially the first region representation as the modified first region representation in the event that the graphical operator is an “over”, “in”, “ratop”, “plusC”, “plusW”, “Xor”, or “out” (visit left operand first)” or alike operatorsoperator; and
if the graphical operator is an “out (visit right operand first)” operation, passing as the modified first region representation a union of the first region representation with the second region representation.
30. The apparatus as recited in
passing substantially the first region representation as the modified second region representation in the event that the graphical operator is an “in”, “ratop”, “out”, “plusC”, “plusW”, or “Xor” or alike operatorsoperator; and
in the event that the graphical operator is an “over” operator, passing as the modified second region representation a union of the first region representation with the second region representation.
31. The apparatus as recited in any one of
32. The apparatus as recited in
a right operand of an “over” operator and the “over” operator node does not need to return an image representation to its parent node;
a left operand of an “in”, “plusC”, “plusW” or “Xor” operator and said operator node does nor need to return an image representation to its parent node;
a right operand of an “in”, “plusC”, “plusW” or “Xor” operator and said operator node docs not need to return an image representation to its parent node;
a left operand of an “out” or “ratop” operator and said operator node does not need to return an image representation to its parent node;
a right operand of a “ratop” operator;
a root of the expression tree;
an operand of an image warp, affine transformation or convolution operator; and
an operand of a colour transformation that does not preserve opaqueness or if said transformation node does not need to return an imap representation to its parent node.
Description This application is a reissue of U.S. Pat. No. 6,191,797, which issued from application Ser. No. 08/861,063 filed May 21, 1997. The present invention relates to the creation of computer-generated images both in the form of still pictures and video imagery, and, in particular, relates to a process, apparatus, and system for creating an image made up by compositing multiple components. Computer generated images are typically made up of many differing components or graphical elements which are rendered and composited together to create a final image. In recent times, an “opacity channel” (also known as a “matte”, an “alpha channel”, or simply “opacity”) has been commonly used. The opacity channel contains information regarding the transparent nature of each element. The opacity channel is stored alongside each instance of a colour, so that, for example, a pixel-based image with opacity stores an opacity value as part of the representation of each pixel. An element without explicit opacity channel information is typically understood to be fully opaque within some defined bounds of the element, and assumed to be completely transparent outside those bounds. An expression tree offers a systematic means of representation for rendering objects or elements of an image. Expression trees comprise a plurality of nodes including leaf nodes, internal nodes and a root node. A leaf node, being the outer most node of an expression tree, has no descendent nodes and consists of one or more graphical elements. An internal node typically branches to left and right subtrees, wherein each subtree is itself an expression tree comprising at least one leaf node. The internal nodes of an expression tree are compositing operators, which treat the left and right subtrees as operands of the operator. The first node of the expression tree is commonly referred to as a root node. The root node of an expression tree represents the final image, and each node of the tree represents a portion of the final image. Although a graphical element may of itself be of a certain size, it need not be entirely visible in a final image, or only a portion of the element may have an effect on the final image. For example, assume an image of a certain size is to be displayed on a display. However, if the image is positioned so that only the top left corner of the image is displayed by the display device, the remainder of the image is not displayed. The final image as displayed on the display device thus comprises the visible portion of the image, and the invisible portion in such a case need not be rendered. Another way in which only a portion of an element may have an effect is when the portion is obscured by another element. For example, a final image to be displayed (or rendered) may comprise one or more opaque graphical elements, sonic of which obscure other graphical elements. Hence, the obscured elements have no effect on the final image. If an element, or parts of elements, that have no effect on true final image can be identified, those elements (or parts) need not be rendered, thereby saving considerable time and possibly memory. Problems arise with prior art methods, at least for images where overlaps occur, because these methods do not easily cope with transparent graphical objects, nor do they handle the full range of compositing operators. It is therefore desirable to at least ameliorate one or more of those problems. In accordance with one aspect of the present invention, there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising graphical elements and graphical operators, each node in the tree being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising, for at least one node in the tree: -
- comparing the region represented by the node to a region representation data structure corresponding to one or more regions represented by at least one other node;
- determining if the region represented by the node is totally or partially obscured by the one or more regions; and
- modifying the expression tree in the event that the region represented by the node is partially or totally obscured.
In accordance with another aspect of the present invention, there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising a node being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising the steps of: -
- traversing the expression tree node by node;
- determining at a current node if a region of the image represented at the node is obscured by regions represented by at least one other node, and modifying the expression tree in the event that the current node is partially or totally obscured.
In accordance with yet another aspect of the present invention there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising a node being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising the steps of: -
- traversing the expression tree node by node and at each current node comprising a graphical operator applying the sub-steps of:
- (i) receiving a first region representation from a parent node;
- (ii) passing to a first operand of the graphical operator a modified first region representation in accordance with a first predetermined modification rule for the operator;
- (iii) returning to the graphical operator a second region representation of regions obscured by a sub-tree associated with the first operand;
- (iv) passing to a second operant of the graphical operator a modified second region representation in accordance with a second predetermined modification rule for the operator;
- (v) returning to the graphical operator a third region representation of regions obscured by a sub-tree associated with the second operand; and
- (vi) determining, in accordance with a set rule for the graphical operator, a final region representation to be returned to the parent node.
- traversing the expression tree node by node and at each current node comprising a graphical operator applying the sub-steps of:
A preferred embodiment of the present invention is hereinafter described with reference to the accompanying drawings and the Appendix, in which: The Appendix contains pseudo-code routines suitable for computer implementation of the preferred embodiment. In the following description of the preferred embodiment, it is assumed that an image composition expression tree, as herein described, has been determined for an image to be rendered. Preferably, image region representations are hierarchical data structures suitable for representing a region or portion of an image and typically used in image processing. One such image region representation is known to those skilled in the art as “quadtrees”. Other forms of image region representations can serve the same purpose. For the sake of simplicity, an image region representation is hereinafter referred to as a quadtree. Typically, the creation of a quadtree representing a region of an image requires the sub-division of the region into a plurality of cells, each cell being a portion of the region, and each cell represented by a node of the quadtree. Hence, increasing the number of subdivisions of a region of an image correspondingly increases the number of nodes of the quadtree, thereby increasing the depth of the quadtree and the resolution of the region represented by the quadtree. Compositing operations include
Table 1 specifically shows various compositing methods for combining two different images together utilising different operators. Additional operators are also possible. The additional operators may be utilized to implement special effects. The “wrap around” nature of the “plusW” operator means that when, for example, the addition of Ac+Bc is greater than a maximum value of a color component, the value is “wrapped around” to start again with reference to the minimum value in the color space. Alternatively, the process of “clamping” utilized by “plusC” involves clamping the addition of, for example, Ac+Bc to the maximum value of a color component when the addition is greater than this component. In the preferred embodiment, an expression tree can contain a variety of node types including binary compositing operators, unary operators and primitives. Unary operators typically include colour transformations, image convolutions, affine transformations and image warping. Primitives typically include graphical elements like pixel-based images, spline-based paths, text, “all” (“all” is a graphical element which spans the size of the entire image being created), edge blends, boxes or the like. Table 2 lists a set of binary compositing operators and the action to be performed when those operators are treated when simplifying an expression tree.
At a node of an expression tree represented by an operator, typically a region representation, such as a quadtree, is passed to each operand during the process of simplifying the expression tree. At the node comparing the operator, an action is to be taken as to whether a sub-tree branching off the node is to vainisih (i e , branches need to he pruned) or a quadtree corresponding to the unobscured portions of graphical elements is to be returned from this node for possible further processing at other nodes. The following notation is used in Table 2:
In the last two columns of Table 2, typical replacement rules are specified, where “L” means replace a current node with the left sub-tree branching off the current node, “R” means replace a current node with the right sub-tree branching off the current node, and “V” means that the current node vanishes. A node that is to “vanish” implies that the region of the image represented by the node is obscured by other graphical elements. Hence, the node has no effect on the final image. If both operands vanish, the current node also vanishes. Reverse operators can be substituted for the operators described in Table 2. The “over” operator, described as “A over B” implies graphical element “A” is over graphical element “B”. For example, the can he substituted by a reverse operator of the “over” operator, typically denoted as “rover” (reverse over), so that “B rover A” results in a composite of graphical element “A” and “B” equivalent to “A over B”. As an illustrative example, consider the (first) operator in the first row of the “Operator” column of Table 2 (i.e., the “over” operator). At a current node of an expression tree represented by an “over” operator, a parent node passes a quadtree q The quadtree q If the region represented by the left operand is found to be completely obscured by the quadtree q The treatment of unary operators when simplifying an expression tree depends on the type of operation: -
- (a) In colour transformation, the quadtree q
_{0 }is passed down to the operand of the colour transformation operator. If the transformation preserves opaqueness (i.e. opaque pixels remain opaque after transformation), the quadtree returned from the operand is returned by the unary operator. In other words, the operand obscures that which the result of the colour transformation obscures. If the transformation does not preserve opaqueness, the unary operator returns an empty quadtree, because the region that the unary operation obscures cannot be determined. If the operand vanishes, the unary operator vanishes, unless invisibility (zero opacity) is not preserved. If invisibility is not preserved, the sub-tree having the unary operator as its root is replaced by an appropriate “all” graphical element. - (b) Affine transformations and image warps do not preserve geometry between the quadtree and the primitives. If the unary operator is obscured by quadtree q
_{0}, it vanishes. Otherwise, traversal is restarted at the operand of the affine transformation or image warp operator, passing an empty quadtree as the obscuring region. An empty quadtree is returned by the operator unless the quadtree returned by its operand can be easily transformed. - (c) Image convolution: If the unary operator is obscured by quadtree q
_{0}, it vanishes. Otherwise, traversal is restarted at the operand of the image convolution. passing an empty quadtree as the obscuring region. An empty quadtree is returned by such an operator because the blurring induced by the operator makes it difficult to use any quadtree returned by its operand. However, if the image convolution operator does not alter opacity, the quadtree returned by the operator's operand can be in turn returned by the operator of the image convolution to its parent node.
- (a) In colour transformation, the quadtree q
In the preferred embodiment, an image composition expression tree (hereinafter “expression tree”) of an image to be rendered is traversed, preferably in a depth-first fashion. Each node of the expression tree receives from its parent node a region representation of one or more areas of the image. The region representation is compared to the region represented at the node to determine if the region represented by that node is obscured. A node in which the region represented by the node is totally obscured is removed from the expression tree with an appropriate simplification of tile expression tree, as hereinafter described. In the event that the region represented by the node is only partially obscured, a clipping operator is applied to the region represented by the node to clip the region of the image represented at the node to discard the obscured portions of the image. For example, if the region represented by a node is totally obscured by one or more regions represented by other nodes of the expression tree, the node is removed from the expression tree in such a way that a graphical operation or a graphical element at the node need not be executed or rendered, whichever the case may be. If a node is partly obscured by one or more regions represented by other nodes in the expression tree, a clipping operator is applied to the node in such a way that, when executing a compositing operator, substantially unobscured regions of the image represented at the node are in the resultant composite of the region of the node. When an image is composited and subsequently rendered from an expression tree comprising nodes clipped by a clipping operator, substantially those portions of the graphical elements that are unobscured by other graphical elements of the image are reproduced or rendered. Applying a clipping operator to a node can, in its simplest form, result in the cropping of the graphical elements represented at the descendent nodes to substantially those portions of the graphical elements that are unobscured. However, applying a clipping operator to a node is not limited thereto. Applying a clipping operator to a node of an expression tree having a compositing operation at that node can result in a different combination of compositing operators, having an effect on the node as if the region represented is cropped to its unobscured portion. The process of compositing an expression tree Alternatively, as shown in If a region of the image represented by a node has been determined to be unobscured or only partially obscured, the node passes the region representation that the node received from a parent node, to each of its descendant nodes in turn. The same process occurs at each descendant node with the net effect that each descendant node passes back to its parent node either an image representation of tae areas of the image obscured by the region represented at the descendant node, or an indication that the descendant node is totally obscured. After the descendants of a node have been processed, the region representations returned from the descendants are utilized to derive a region representation of the regions of the image that are obscured by the node. This result is returned to the node's parent. In the preferred embodiment, the traversal of the expression tree to simplify the tree is initiated at the root of the tree in a “depth-first fashion”, known to those skilled in the art. Preferably, when traversing an expression tree in a depth-first fashion, the path leading down the left branch, at any node, is given priority and this path down the tree to a descendent node is taken first. When no further left ranch paths are available at a current node, processing returns to the previous node and a path heading down a right branch of this node is taken. An expression tree is traversed in this manner until all nodes of the expression tree have been visited. In step In decision block The predetermined set of node replacement rules (not shown in -
- if the parent node is an “over” graphical operator and the current node is at a left branch of the parent node, replace the parent node with a right subtree of the parent node;
- if the parent node is an “over” graphic operator and the current node is at a right branch of the parent node, replace the parent node with a left subtree of the parent node;
- if the parent node is an “in” graphical operator, remove the parent node and any subtree branching off the parent node;
- if the parent node is a “ratop” graphical operator and the current node is at a left branch of the parent node, removing the parent node and any subtree branching off the parent node;
- if the parent node is a “ratop” graphical operator and the current node is at a right branch of the parent node, replace the parent node with a left subtree of the parent node;
- if the parent node is an “out” graphical operator and the current node is at a left branch of the parent node, remove the parent node and any subtree branching off the parent node.
- if the parent node is an “out” graphical operator and the current node is at a right branch of the parent node, replace the parent node with a left subtree of the parent node;
- if the parent node is a “plusC” graphical operator and the current node is at a left branch of the parent node, replace the parent node with a right subtree of the parent node;
- if the parent node is an “plusC” graphical operator and the current node is at a right branch of the parent node, replace the parent node with a left subtree of the parent node;
- if the parent node is a “plusW” or an “Xor” graphical operator and the current node is at a left branch of the parent node, replace the parent node with a right subtree of the parent node; and
- if the parent node is an “plusW” or an “Xor” graphical operator and the current node is at a right branch of the parent node, replace the parent node with a left subtree of the parent node.
In step In step -
- (A) where the graphic operator is an “over” or a “plusC” operator, the final region representation is determined from a union of the second region representation and the third region representation;
- (B) where the graphic operator is an “in” operator, the final region representation is determined from an intersection of the second region representation and the third region representation;
- (C) where the graphic operator is an “ratop” operator, the final region representation is the second region representation;
- (D) where the graphic operator is an “out” operator, the final region representation is determined from a difference of the second region representation and a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node; and
- (E) where the graphic operator is an “Xor” or a “plusW” operator, the final region representation is determined from a union of the second region representation less a region representation comprising at least a region represented by a bounding box of a node at a right subtree of the current node and the third region representation less a region representation containing a bounding box of a node at a right subtree of the current node.
The first predetermined modification rule preferably is to pass substantially tile first region representation as the modified first region representation if the graphical operator is an “over”, “in”, “ratop”, “plusC”, “plusW”, “Xor”, “out” (visit left operand first)” or a like operator. If the graphical operator is an “out (visit right operand first)” operation, it involves passing as the modified first region representation a union of the first region representation with the second region representation. Further, the second predetermined modification rule is to pass substantially the first region representation as the modified second region representation if the graphical operator is an “in”, “ratop”, “out”, “plusC”, “plusW”, “Xor” or like operators. If the graphical operator is an “over” operator, it involves passing as the modified second region representation a union of the first region representation with the second region representation. Preferably, the image representation is not created at a node, or returned to a parent node of the node, unless the image representation is subsequently utilised, or if the node is the right operand of an “over” operator and the “over” operator node does not need to return an image representation to its parent node. Likewise, the image representation is not created at a node or returned if the node is the left or the right operand of an “in”, “plusC”, “plusW” or “Xor” operator and the operator node does not need to return an image representation to its parent node. Still further, the image representation may not be created at a node or returned to the parent node if the node is the left operand of an “out” or “ratop” operator and does not need to return an image representation to its parent node. The image representation may not be created at a node or returned if the node is the right operand of a “ratop” operator, the root of the expression tree, the operand of an image warp, affine transformation or convolution operator, the operand of a colour transformation that does not preserve opaqueness, or if the transformation node does not need to return an image representation to its parent node. Further aspects of the preferred embodiment are set forth in detail in the Appendix forming part of the description. In particular, the Appendix contains pseudocode listings for implementing the method according to the preferred embodiment. In this connection, the preferred embodiment is preferably implemented as computer software, capable of being stored on recording media, that can be carried out as a process executing on a computing device, such as a general purpose computer. The embodiments of the invention can preferably be practiced using a conventional general-purpose computer, such as the one shown in The computer The video interface The first node Following receipt of the empty quadtree at the first node Comparing a quadtree with a graphical element (eg, the sub-image Sub-image The union of an empty quadtree with the first quadtree representation of the sub-image The first left quadtree is forwarded to the second node The second node A quadtree substantially equivalent to the first left quadtree representing the sub-image The text is not obscured by the quadtree (the image region represented by the quadtree) passed down from the second node The second node At the first node Another example of simplifying (optimising) an expression tree is now described with reference to The following steps S -
- S
**0**: An empty quadtree q_{0 }is created representing the empty foreground region**39**not obscuring the entire image**40**. This empty quadtree q_{0 }is passed to a first node**51**(or root node) of the expression tree**50**. - S
**1**: The first node**51**of the expression tree**50**is an “over” operator. The process receives the empty quadtree q_{0 }passed to the first node**51**from the previous step S**0**and compares the region of the image represented by the quadtree q_{0 }with a region represented by the first node**51**to determine if the region represented by the first node**51**is obscured. Since q_{0 }is an empty quadtree and cannot obscure the region represented by the first node**51**, the process continues on to the descendant nodes. Firstly, the quadtree q_{0 }is passed down the left branch of the node**51**to a second node**52**. - S
**2**: The second node**52**, being in this example an “in” operator, receives the empty quadtree q_{0}. The quadtree q_{0 }is compared with a region represented by the second node**52**to determine if this region is obscured by the quadtree q_{0}. The region of the second node**52**is not obscured by the quadtree q_{0 }since the quadtree q_{0 }is empty. The process continues in a depth-first fashion and passes the quadtree q_{0 }to the left branch of the second node**52**. - S
**3**: A third node**53**is a leaf node representing the sub-image**42**. This third node**53**receives the quadtree q_{0 }passed down from the S**2**step and compares the region of this node**53**with the region represented by the quadtree q_{0 }to determine if the region represented by node**53**is obscured by the quadtree q_{0}. In this example, the quadtree q_{0 }is empty and therefore the node**53**is not obscured. However, the image “A” is a graphical element that can potentially obscure other graphical elements. Hence, a quadtree q_{1 }that represents the region obscured by the image is created, and passed back to the second node**52**since no further left branches are available at the third node**53**. - S
**4**: The second node**52**receives back from the third node**53**the quadtree q_{1 }and as the second node**52**is an “in” operator, the quadtree q_{1 }is stored in memory as the obscuring region of the left operand of the “in” operator. The obscuring region of the left operand of an operator is denoted herein as q_{L}. Thus, in this example, q_{L}=q_{1}. The action of the process, in accordance with Table 2. is to pass down to a right descendant node**54**of the node**52**the quadtree received at the second node**52**passed down from its parent node**51**. In this example, the quadtree q_{0 }passed to the second node**52**from the first node**51**(parent node) is sent down the right branch to the right descendent node**54**. - S
**5**: The right descendent node**54**(fifth node) is again a leaf node and has represented therein the region indicated as the circle**44**. The quadtree q_{0 }has been passed down from the second node**52**following step S**4**, and is compared with the region of the image occupied by the circle**44**to determine if the region represented by the quadtree q_{0 }is obscured by the circle**44**. Again, the quadtree q_{0 }is empty, and the node**54**is therefore not obscured. However, the circle**44**is a graphical element (object) with the potential to obscure graphical elements (objects) which may lie beneath. Hence, a quadtree q_{2 }is created representing the region of the image occupied by the circle. The quadtree q_{2 }is passed back to the second node**52**since no further branches are available at this node**54**. - S
**6**: The second node**52**receives the quadtree q_{2 }passed back from its right descendent node**54**, and the quadtree q_{2 }is stored as the obscuring region of the right operand of the “in” operator (i.e., q_{R}=q_{2}). The process proceeds in accordance with the action set out in Table 2 for the “in” operator. It passes back to its parent node (ie, the first node**51**) the intersection of the regions represented by its two operands (ie, the sub-image**42**with the region of the circle**44**). The intersection results in the region represented by the portion of sub-image**42**that coincides with the circle**44**(ie, the quadtree q_{2}). In this example, this intersection q_{L}∩q_{R}=q_{1}∩q_{2}=q_{2 }represents the region in which node**52**can obscure other graphical elements. - S
**7**: The first node**51**receives the quadtree q_{2 }passed back from the second node**52**. The quadtree q_{2 }is stored as the obscuring region of the left operand of the “over” operator (q_{L}=q_{2}). In accordance with Table 2, the action to be performed when descending a right branch of a node having an “over” operator is to pass down the right branch a union (ie, q_{0}∪q_{L}=q_{0}∩q_{2}=q_{2}) of the quadtree q_{0 }and the quadtree q_{L }passed back from the second node**52**. The result of this union (q_{0}∪q_{L}) is a quadtree substantially identical with q_{2}. Hence, the result of this union (the quadtree q_{2}) is passed down the right branch to a fifth node**55**also representing an “over” operator. - S
**8**: The region represented by the quadtree q_{2 }passed to the fifth node**55**is compared with the region represented at the fifth node**55**to determine if the region of the node**55**is obscured by the quadtree q_{2 }(region of). The region of the image represented at the fifth node**55**is not obscured by the region of the quadtree q_{2}. The quadtree q_{2 }is passed down to the left branch descendent of the fifth node**55**. - S
**9**: The left descendent of the fifth node**55**is a leaf node**56**representing the region of the image ofFIG. 6 illustrating the text**43**. The leaf node**56**receives the quadtree q_{2 }passed down from the fifth node**55**and is compared to the region represented at the leaf node**56**(typically, the region of the image ofFIG. 6 occupied by the text**43**is a bounding box comprising text) to determine if the region represented by quadtree q_{2 }obscures the region represented at leaf node**56**. The region represented by the quadtree q_{2 }(the region occupied by circle**44**) partly obscures text**43**. Hence, the text**43**is clipped or tagged for clipping at a later stage. The text**43**is clipped by applying a clipping operator, wherein the clipping operation constructs a “clip” path from the quadtree q_{2 }and clips or cuts the text**43**to this path.
- S
At this point, typically, a new quadtree representing the region of the image occupied by the text is created and returned to the fifth node -
- S
**10**: The fifth node**55**receives the empty quadtree q_{3 }passed back by the previous step S**9**. This quadtree q_{3 }is stored as the obscuring region of the left operand of the “over” operator at the fifth node**55**(q_{L}=q_{3}). Again, in accordance with Table 2, the action to be performed when descending a right branch of a node having an “over” operator is to pass down to the right branch a union of the quadtree q_{2 }passed to the node**55**by the parent node**51**with the quadtree q_{3 }associated with the left operand (q_{L}∪q_{2}=q_{3}∪q_{2}=q_{2}). The union of the quadtree q_{3 }with the quadtree q_{2 }results in a quadtree equivalent to quadtree q_{2}, since quadtree q_{3 }is the empty quadtree described in step S**9**. Therefore, quadtree q_{2 }is passed down the right branch of the expression tree to a right leaf node**57**of parent node (fifth node)**55**. - S
**11**: The right leaf node**57**is represented by the graphical element page “D”**41**representing the background page inFIG. 6 . The quadtree q_{2 }passed down to the right leaf node**57**by the fifth node**55**is compared with the region of page “D”**41**to determine if the region represented by the quadtree q_{2 }obscures the region represented by page “D”**41**. The result of this comparison is that the region represented by quadtree q_{2 }(circle**44**) partly or in total obscures page “D”**41**.
- S
The graphical element page “D” is therefore either tagged so as to be clipped to the boundary of the circle -
- S
**12**: The process returns to the fifth node**55**, where no further quadtrees need to be created. - S
**13**: The process returns to the first node**51**, where no further quadtrees need to be created. - S
**14**: The process ends having optimised the expression tree**50**ofFIG. 7 to provide the expression tree**60**ofFIG. 8 . The diamond shape symbols**58**and**59**shown inFIG. 8 indicate that the text**43**and the page “D”**41**are to be clipped (or have been clipped whichever the case may be), respectively.
- S
The foregoing examples of quadtree representations, described with reference to -
- (a) it is preferable to do the little that covers most cases than to attempt perfect results; and
- (b) at a node, at least initially, it is not known whether or riot obscuration actually occurs in an image, so it is preferable to avoid expensive tests having benefits that are uncertain. These principles apply in the following ways.
Firstly, increasing the depth (ie, the number of nodes and branches) of a quadtree increases the quadtree resolution and the ability to detect obscuration. However, beyond a predetermined resolution, the computational cost of creating and combining quadtrees increases exponentially, exceeding the savings in performance gained by attempting to eliminate from an expression tree the diminishing areas represented by the increased quadtree depth. Secondly, it is computationally expensive to treat every opaque primitive as a potential obscurer (a graphical element likely to obscure other graphical element of an image). The smaller a primitive is the less likely it is to obscure another primitive. Hence, the creation of quadtrees is preferably limited to potential obscurers that are of a predetermined size or greater. Typically, primitives that are too costly to convert to a quadtree are not considered because they cannot guarantee a good return on the investment. Thus, a “good obscurer” preferably has the following features: -
- (a) fully opaque;
- (b) larger than a predetermined size (and thus likely to obscure other primitives of an image);
- (c) simple to convert to a quadtree very quickly (for example, choose only graphical objects comprising a single simple convex outline).
Thirdly, testing for obscuration (ie, determining whether a first graphical element obscures one or more graphical elements of an image) can be performed by representing the region covered by the first graphical element as a quadtree and testing if one or more cells of the region represented at the nodes of the quadtree obscure regions covered by the one or more graphical elements of the image. Typically, the one or more regions are also represented by quadtrees, and the cells of quadtrees are compared. However, representing an arbitrary region of an image as a quadtree representation, to a predetermined resolution, may prove very computationally expensive though entirely possible. Hence, a bounding box of a region represented at a node of an expression tree is preferably constructed. Whether the node is a graphical element or an operator, the region represented at the expression tree node is well defined. While the bounding box at a node of an expression tree may not exactly represent the region covered by the node, the enhancement in computational performance typically out-weighs the detriment in performance by failing to detect obscurities. Testing for obscured graphical elements by comparing their respective bounding box is preferred over comparing a quadtree representation of the regions of the image occupied by the graphical elements. This may result in some obscured graphical elements, below a predetermined size, being missed and considered not obscured. However, selecting a simple test for determining whether graphical elements are obscured by other graphical elements of the image is preferable over computationally expensive tests that in most common cases do not justify the return on the investment. The following is an example of a pseudo-code call to a routine “test” which compares the bounding box at a node with a quadtree cell (hereinafter “cell”).
This function (routine) is invoked with:
Quadtrees are created and discarded continuously. A very simple and fast scheme to manage computer memory is preferred with low memory allocation activity (eg, allocating largish blocks of memory, say, 1000 quadtree cells at a time). Cells are allocated to these blocks, treated as write-once-read-only, and not deallocated until the end of the entire expression tree traversal. This approach allows cells to be shared amongst quadtrees, and considerably reduces copying when performing quadtree set operations. Preferably, as a performance issue, if a region representation (quadtrees) need not be created, no region representation is generated at a node. For example, a parent node may request from a descendent node a quadtree of the region which the descendent node and its descendent node may obscure. Typically, if a region representation is never to be utilized in subsequent computation, the region representation preferably does not need to be created. The aforementioned process for optimising an expression tree is described using recursion for convenience. Implementation of the process is also possible using a non-recursive process utilising back-pointers. This is both to reduce function-call overhead, and to handle very large trees that in practise are rarely balanced. The foregoing describes only a small number of embodiments of the present invention and modifications, obvious to those skilled in the art in view of the foregoing description, can be made thereto without departing from the scope and spirit of the present invention. The following function tests node for obscuration against quadtree q
Patent Citations
Non-Patent Citations
Classifications
Legal Events
Rotate |