Patent Application 17369438 - SYSTEMS AND METHODS FOR MODELLING INTERACTIONS - Rejection
Appearance
Patent Application 17369438 - SYSTEMS AND METHODS FOR MODELLING INTERACTIONS
Title: SYSTEMS AND METHODS FOR MODELLING INTERACTIONS OF PHYSICAL ASSETS WITHIN A WORKSPACE
Application Information
- Invention Title: SYSTEMS AND METHODS FOR MODELLING INTERACTIONS OF PHYSICAL ASSETS WITHIN A WORKSPACE
- Application Number: 17369438
- Submission Date: 2025-05-15T00:00:00.000Z
- Effective Filing Date: 2021-07-07T00:00:00.000Z
- Filing Date: 2021-07-07T00:00:00.000Z
- National Class: 703
- National Sub-Class: 002000
- Examiner Employee Number: 93438
- Art Unit: 2187
- Tech Center: 2100
Rejection Summary
- 102 Rejections: 0
- 103 Rejections: 3
Cited Patents
The following patents were cited in the rejection:
- US 0200863đ
- US 0178257đ
- US 0094981đ
- US 0137681đ
Office Action Text
DETAILED ACTION Claims 1 â 20 have been presented for examination. Claims 1 and 11 are currently amended. This office action is in response to submission of the amendments on 02/03/2025. Response to Claims Rejections â 35 USC 103 Applicantâs arguments with respect to the amendments have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Langseth, J. (US 2013/0178257) (see Claims Rejections â 35 USC 103 for the detailed mapping). Applicantâs arguments with respect to He over feature (ii) have been fully considered. However, the Office does not consider them to be persuasive. Applicant argues: âWith respect to feature (ii), the Office action alleges that He discloses manipulating the asset representation in accordance with an interaction request. Applicant respectfully disagrees. In particular, in reading the claim as a whole, an interaction results in a physical response of the physical asset and the workspace. The annotation mechanism of He clearly does not result in a physical response of the physical asset in the workspace. Further, with respect to Applicant's claims 8 and 18, it is clear that an annotation is an entirely independent feature of the presently claimed system and method, rather than being a type of interaction request as claimed in claim 1. Neither Hu nor Hossain disclose or teach manipulation of an asset representation in accordance with a selected interaction request, and hence do not cure the deficiencies of He with respect to feature (ii).â (emphasis added) Applicant argues that the claim as a whole requires that the interaction results in a physical response of the physical asset (see emphasis in Applicantâs remarks). The requirements for broadest reasonable interpretation are discussed in MPEP 2111. Examiner notes that the claim explicitly recites âcomputing a physical response of the physical asset ⌠as a result of the interaction; and outputting the computed physical responseâ. Therefore, the claim as whole merely requires that an interaction results in a computed physical response of a physical asset, in contrast to the interaction directly resulting in a physical response of the physical asset as argued. Applicant further appears to argue that He does not teach âthe interaction request specifying an interaction ⌠manipulating the asset representation in accordance with the interaction request; ⌠computing a physical response of the physical asset ⌠as a result of the interactionâ, and argues that Hossain (Thesis) does not teach the âmanipulatingâ. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Specifically, the âmanipulatingâ is taught by Langseth (257), and the âcomputing a physical responseâ is taught by Hossain (Thesis). Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: Determining the scope and contents of the prior art. Ascertaining the differences between the prior art and the claims at issue. Resolving the level of ordinary skill in the pertinent art. Considering objective evidence present in the application indicating obviousness or nonobviousness. Claims 1 â 8 and 11 - 18 are rejected under 35 U.S.C. 103 as being unpatentable over He et al âA Mobile Solution for Augmenting a Manufacturing Environment with User-Generated Annotationsâ (hereinafter âHeâ) in view of Hu, X. âSHARED VIRTUAL WORLD TECHNOLOGIESâ (henceforth âHu (Thesis)â), and further in view of Langseth, J. (US 2013/0178257) (henceforth âLangseth (257)â), and further in view of Hossain, A. âVirtual Reality Simulation Modeling for Tele-Surgery and Tele-Haptic Class of Applicationsâ (henceforth âHossain (Thesis)â). He and Hu (Thesis) and Langseth (257) and Hossain (Thesis) are analogous art because they solve the same problem of modeling interactions of physical assets. With regard to claim 1, He teaches a method comprising: obtaining a workspace definition representing a workspace and an asset definition representing a physical asset located within the workspace; (He Page 6, Bottom and Figure 4 a tree data structure is available comprising the environment and industrial assets as nodes PNG media_image1.png 456 807 media_image1.png Greyscale ) generating a model by: (i) defining a root node for the model, (He Page 6, Bottom and Figure 4 the data structure is comprises of root node and child nodes) the root node designating a navigational region of the model based on the workspace definition, and (He Page 3, Bottom and Figure 4 a manufacturing environment is captured which is able to be navigated (a navigational regional)) (ii) defining at least one child node descending from the root node, the child node designating an asset representation based on the asset definition; (He Page 6 Bottom and Figure 4 the child nodes represent physical elements in the environment) He does not appear to explicitly disclose in response to a connection request from a client device: defining an avatar for the client device as a further child node descending from the root node; locating the avatar within the model in the navigational region; and presenting the model with the avatar to the client device. However Hu (Thesis) teaches: in response to a connection request from a client device: (Hu (Thesis) Figure 3.2 the system is implemented as a client-server architecture using HTTP connections and a Service and a Connector (in response to a connection requestion from a client device) PNG media_image2.png 243 392 media_image2.png Greyscale ) defining an avatar for the client device as a further child node descending from the root node; (Hu (Thesis) Figure 2.1 and Page 62 the avatar node can be defined in a VRML scene graph PNG media_image3.png 280 419 media_image3.png Greyscale , and Page 12 the root node contains the scene information) locating the avatar within the model in the navigational region; and (Hu (Thesis) Table 5.1 the avatar desirably changes position within the scene) presenting the model with the avatar to the client device; (Hu (Thesis) Figure 5.2 the movement of the avatar is on a client browser (presenting to the client device) PNG media_image4.png 288 380 media_image4.png Greyscale ) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets disclosed by He with the methods of including an avatar in an environment disclosed by Hu (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to dynamically generate avatars for users (Hu (Thesis) Page 5). He in view of Hu (Thesis) does not appear to explicitly disclose: providing, for presentation at the client device, interaction request options for potential interactions with the asset representation; in response to selection of an interaction request from the interaction request options from the client device, the interaction request specifying an interaction of the asset representation relative to the workspace, manipulating the asset representation in accordance with the interaction request. However, Langseth (257) teaches: providing, for presentation at a client device, interaction request options for potential interactions with an asset representation (Langseth (257) Paragraph 38 interacting with a virtual object is by a visual user interface on a user device (providing, for presenting at a client device) âIn one implementation, FIG. 5 illustrates an exemplary user interface 500 that may be shown on the mobile device in response to the mobile device having a suitable proximity to a particular virtual object 560 and the user selecting the actions option in the virtual object menu 520 to collect, view, move, and/or otherwise interact with the virtual object 560â, and Paragraph 39 a kick or movement action is desirably defined with a specified location (interaction request options) âAccordingly, the action option 590 may enable the user to specify a certain location on the map shown in the augmented reality area 540 and kick or otherwise move the virtual object 560 to the specified locationâ) in response to selection of an interaction request from the interaction request options from the client device, the interaction request specifying an interaction of the asset representation relative to a workspace, manipulating the asset representation in accordance with the interaction request; (Langseth (257) Paragraph 39 specifying the location in the context of an action option (specifying of interaction relative to a workspace) implements the kick/move in the virtual world (manipulating the asset in accordance with) âAccordingly, the action option 590 may enable the user to specify a certain location on the map shown in the augmented reality area 540 and kick or otherwise move the virtual object 560 to the specified locationâ) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis) with the methods of specifying an action on a virtual object using a mobile device disclosed by Langseth (257). One of ordinary skill in the art would have been motivated to make this modification in order to desirably perform actions on a virtual object (Langseth (257) Paragraph 38). He in view of Hu (Thesis), and further in view of Langseth (257) does not appear to explicitly disclose: in response to a navigation request from the client device, the navigation request specifying a target location, navigating the avatar within the navigational region to the target location; in response to manipulating the asset representation, computing a physical response of the physical asset and the workspace as a result of the interaction; and outputting the computed physical response. However Hossain (Thesis) teaches: in response to a navigation request from a client device, the navigation request specifying a target location, navigating an avatar within the navigational region to the target location; (Hossain (Thesis) Page 24 â 25 a user avatar desirably moves through a virtual space âVirtual reality allows people to walk inside the planned building and get a solid idea of what the building will look like before the construction decision is made ⌠the designer can even modify the building interactively when he sees something he wishes to alter as he moves through the model. Being able to walk inside the virtual representation of the building is an extremely powerful and effective customer presentation toolâ) in response to manipulating an asset representation, computing a physical response of the physical asset and the workspace as a result of an interaction; and outputting the computed physical response. (Hossain (Thesis) Figure 3.2 and Page 61 haptic response is outputted from interaction with virtual object âThe calculation of the movement of the virtual box, knife, cotton, or scalpel with which the PHANTOM is in contact requires both a collision detection system to determine the contact surface on the virtual object and a kinematic model for movement of the virtual objects based on the PHANTOM's motionâ PNG media_image5.png 223 366 media_image5.png Greyscale ) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis), and further in view of Langseth (257) with the methods of simulating mixed reality interactions disclosed by Hossain (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to better simulated real world interactions (Hossain (Thesis) Page 12). With regard to claim 11, it recites substantially similar limitations as claim 1 which is taught by He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis). Claim 11 further recites a server comprising: a memory; a processor interconnected with the memory, the processor configured to. Hu (Thesis) teaches: a server comprising: a memory; a processor interconnected with the memory, the processor configured to. (Hu (Thesis) Page 5 HTTP servers, Page 46 servers have threads accessing memory area, and Page 27 avatar limited by CPU speed) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets disclosed by He with the methods of including an avatar in an environment disclosed by Hu (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to dynamically generate avatars for users (Hu (Thesis) Page 5). With regard to claim 2 and 12, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches: the workspace definition comprises one of: a computer-aided design model of the workspace or a generated model built based on captured data representing the workspace. (He Figure 1 a digital factory model is created (a computer-aided design model of the workspace) PNG media_image6.png 361 537 media_image6.png Greyscale ) With regard to claim 3 and 13, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches: wherein the asset definition comprises one of: a computer-aided design model of the physical asset or a generated model built based on captured data representing the physical asset. (He Figure 1 a digital factory model is created (a computer-aided design model of the workspace) PNG media_image6.png 361 537 media_image6.png Greyscale ) With regard to claim 4 and 14, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches: in response to a second connection, request from a second client device: (Hu (Thesis) Page 4 the system is used with multiple users (request from a second client device) âWe have developed a multiuser virtual world system prototype with up to 10 avatars. It can be scaled to a larger scale virtual reality system if some network problems are resolvedâ) defining a second avatar for the second client device as a second further child node descending from the root node; locating the second avatar within the model in the navigational region; and presenting the model with the avatar and the second avatar to the second client device. (Hu (Thesis) Page 4 additional avatars can be realized with the same effects) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets disclosed by He with the methods of including an avatar in an environment disclosed by Hu (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to dynamically generate avatars for users (Hu (Thesis) Page 5). With regard to claim 5 and 15, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 4 and 14, and further teaches: updating the model presented to the client device to include the second avatar. (Hu (Thesis) Page 4 additional avatars can be realized with the same effects) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets disclosed by He with the methods of including an avatar in an environment disclosed by Hu (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to dynamically generate avatars for users (Hu (Thesis) Page 5). With regard to claim 6 and 16, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches wherein the interaction comprises one or more of: a transformation of the asset representation, a modification of the asset representation, and a process executable by the asset representation. (Hossain (Thesis) Page 61 haptic response is outputted based on movement of a virtual object (a modification of the asset representation) âThe calculation of the movement of the virtual box, knife, cotton, or scalpel with which the PHANTOM is in contact requires both a collision detection system to determine the contact surface on the virtual object and a kinematic model for movement of the virtual objects based on the PHANTOM's motionâ) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis), and further in view of Langseth (257) with the methods of simulating mixed reality interactions disclosed by Hossain (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to better simulated real world interactions (Hossain (Thesis) Page 12). With regard to claim 7 and 17, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches: wherein the physical response is computed based on one or more of: a size and/or dimension of the physical asset, and stresses and/or forces of the physical asset and the workspace. (Hossain (Thesis) Page 61 haptic response based on kinematic model is outputted based on movement of a virtual object (forces of the physical asset and the workspace) âThe calculation of the movement of the virtual box, knife, cotton, or scalpel with which the PHANTOM is in contact requires both a collision detection system to determine the contact surface on the virtual object and a kinematic model for movement of the virtual objects based on the PHANTOM's motionâ) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis), and further in view of Langseth (257) with the methods of simulating mixed reality interactions disclosed by Hossain (Thesis). One of ordinary skill in the art would have been motivated to make this modification in order to better simulated real world interactions (Hossain (Thesis) Page 12). With regard to claim 8 and 18, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and further teaches: receiving an annotation and a target annotation location; (He Figure 5 and 7 annotations are associated with an asset and have a specific location (receiving target annotation location)) when the target annotation location is associated with the asset representation, defining an annotation node descending from the child node designating the asset representation; and when the target annotation location not associated with the asset representation, defining the annotation node descending from the root node. (He Figure 4 annotation for assets have their own node under the asset) Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Hu (Thesis), and further in view of Hossain (Thesis), and further in view of Chang et al. (US 2018/0137681) (henceforth âChang (681)â). He and Hu (Thesis) and Langseth (257) and Hossain (Thesis) and Chang (681) are analogous art because they solve the same problem of modeling the interactions of physical assets. With regard to claim 9 and 19, He in view of Hu (Thesis), and further in view of Langseth (257) and further in view of Hossain (Thesis) teaches all the elements of the parent claim 8 and 18, and does not appear to explicitly disclose: when the avatar navigates within a threshold proximity of the target annotation location, presenting the annotation at the client device. However, Chang (881) teaches: when the avatar navigates within a threshold proximity of the target annotation location, presenting the annotation at the client device. (Chang (681) Paragraph 47 text is positioned within a threshold proximity of an object in a virtual reality environment (within a threshold proximity) âThe text elements that are positioned within a threshold proximity of an image is selected as a text annotation for the virtual reality module 140. The threshold proximity value is a programmable value, configurable by the author device 101.â, and Paragraph 20 a text annotation is automatically displayed (presenting the annotation) when a user navigates (when the avatar navigates) to defined positions, where the using a threshold proximity around target annotation locations would have predictable results âThe text is added as an annotation layer on top of the images so that, when the user navigates to defined positions while in the virtual reality environment, the corresponding text providing narrative context for the images appears as a pop up layer.â) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) with the method of displaying annotation text disclosed by Chang (681). One of ordinary skill in the art would have been motivated to make this modification in order to desirably display annotation to objects in a virtual world to a viewer (Chang (681) Abstract). Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis), and further in view of Kamat et al. (US 2014/0200863) (henceforth âKamat (863)â). He and Hu (Thesis) and Langseth (257) and Hossain (Thesis) and Kamat (863) are analogous art because they solve the same problem of modeling interactions of physical assets. With regard to claim 10 and 20, He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) teaches all the elements of the parent claim 1 and 11, and does not appear to explicitly disclose: defining a layer node as an intermediary node between the root node and the child node; and when the layer node is toggled on, displaying the asset representation in the model; and when the layer node is toggled off, hiding the asset representation in the model. However, Kamat (863) teaches: defining a layer node as an intermediary node between the root node and the child node; and when the layer node is toggled on, displaying the asset representation in the model; and when the layer node is toggled off, hiding the asset representation in the model. (Kamat (863) Paragraph 47 a parent node of child nodes (a layer node as an intermediary node), can be used to hide or show the child nodes (displaying the asset representation, or hiding the asset representation) âIn essence, scene graphs form another layer of abstraction between the graphics hardware and the application. The scene graph data structure consists of nodes of differing nature. Nodes can be used to store the polygon geometry of 3D objects, apply translation and rotation to objects, serve as parent nodes for a group of child nodes, hide and show other nodes, as well as other uses. The node at the top of the scene graph is customarily referred to as the root node. All of the constituent elements of the scene are child nodes of the root node.â) It would have been obvious for one of ordinary skill in the art before the filing date of the claimed invention to have combined the methods of modeling a manufacturing environment comprising assets and avatars disclosed by He in view of Hu (Thesis), and further in view of Langseth (257), and further in view of Hossain (Thesis) with the method of displaying or hiding groups of nodes disclosed by Kamat (863). One of ordinary skill in the art would have been motivated to make this modification in order to desirably display modify groups of nodes (Kamat (863) Paragraph 47). Examiner General Comments With regard to the prior art rejection(s), any cited portion of the relied upon reference(s), either to specific areas or as direct language, is intended to be interpreted in the context of the reference(s) as a whole, as would be understood by one of ordinary skill in the art. Therefore the lack of a citation to other portions which inform the interpretation of the cited portions, is in no way intended to exclude said other portions. Any direct language, as shown with quotation marks, is intended solely to further point out the teachings provided to one of ordinary skill in the art, and is in no way intended to limit the relied upon teachings to only the quoted portions existing in a vacuum. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Bradski et al. (US 2019/0094981) teaches a proxy input for a user to use gestures to move virtual objects. Baker et al. (US 11204678) teaches moving a virtual object using object variations in a detail card, and a virtual object moves with the surrounding objects as if the user is touching it. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALFRED H. WECHSELBERGER whose telephone number is (571)272-8988. The examiner can normally be reached M - F, 10am to 6pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examinerâs supervisor, Emerson Puente can be reached at 571-272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ALFRED H. WECHSELBERGER/ExaminerArt Unit 2187 /EMERSON C PUENTE/Supervisory Patent Examiner, Art Unit 2187