Patent Application 18092868 - GENERATING TRAINING DATA USING REAL-WORLD SCENE

Title: GENERATING TRAINING DATA USING REAL-WORLD SCENE DATA AUGMENTED WITH SIMULATED SCENE DATA
Application Information

Invention Title: GENERATING TRAINING DATA USING REAL-WORLD SCENE DATA AUGMENTED WITH SIMULATED SCENE DATA
Application Number: 18092868
Submission Date: 2025-05-15T00:00:00.000Z
Effective Filing Date: 2023-01-03T00:00:00.000Z
Filing Date: 2023-01-03T00:00:00.000Z
National Class: 345
National Sub-Class: 632000
Examiner Employee Number: 99823
Art Unit: 2612
Tech Center: 2600
Rejection Summary

102 Rejections: 0
103 Rejections: 3
Cited Patents

No patents were cited in this rejection.
Office Action Text


    DETAILED ACTION

Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .

Status of claims:
Claims 1-20 are pending
Claims 1, 8, and 15 are amended

Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.



	Claim(s) 1-4, 8-11, and 15-18 is/are rejected under 35 U.S.C. 103 as being anticipated by Govardhanam et al. (WO 2022265664) in view of Hong et al. (CN 202080004653).

Regarding claim 1,
Govardhanam teaches:
A computer-implemented method comprising (Govardhanam: Fig 5; [0008]: FIG. 5 illustrates an example processor-based system with which some aspects of the subject technology can be implemented):
generating, based on real-world autonomous vehicle (AV) scene data captured by sensors of an AV during a real-world scenario, a simulation of the real-world scenario (Govardhanam: [0016]: By way of example, the training data can correspond with data collected by various AV sensors (either real or simulated)), wherein the real- world AV scene data represents a real-world environment of the AV during the real-world scenario (Govardhanam [0049] The simulation platform 456 can replicate a variety of driving environments and/or reproduce real-world scenarios from data captured by the AV 402, including rendering geospatial information and road infrastructure (e.g., streets, lanes, crosswalks, traffic lights, stop signs, etc.) obtained from the map management system platform 462;);
adding a first object to the simulation of the real-world scenario (Govardhanam: [0029-0030]: In step 306, process 300 includes inserting a virtual object into a first location within the driving scene. In some aspects, the virtual object can be based on the object data. For example, the virtual object may correspond with a modified version of an object corresponding with the object data…the insertion process can take consideration of several factors including a determination of one or more locations within the recorded driving scene where the virtual object is to be inserted.);
generating synthetic AV scene data based on the simulation of the real-world scenario, including the first object (Govardhanam: [0028]: In other aspects, the selected object may be entirely synthetic, e.g., comprising object data that was generated using a synthetic process, such as by simulating the object using a 3D simulation platform; In other aspects, training data can include synthetic data collected or originated from a generated three-dimensional (3D) environment, e.g., a simulated environment.), wherein the synthetic AV scene data represents a simulated environment of the AV that includes the first object and corresponds to the real-world environment (Govardhanam [0015] training data can include synthetic data collected or originated from a generated three-dimensional (3D) environment, e.g., a simulated environment. In such instances, sensor data can also include synthetic data, for example, corresponding with data collected from a synthetic 3D environment using simulated (synthetic) sensors. [0016] As such, the training data can include various objects, such as other active traffic participants, including but not limited to: vehicles or motorists, bicyclists, and/or pedestrians, etc. In some aspects, the training data can include other objects that are detected by vehicle sensors (either real or simulated)); 
resulting in augmented real-world AV scene data that represents an augmented-reality environment of the AV, including the first object (Govardhanam [0014] Aspects of the disclosed technology address the foregoing limitations of ML- based AV perception training by providing solutions for augmenting training data to improve classification accuracy for rare event and/or rare contexts. [0015] training data can include synthetic data collected or originated from a generated three-dimensional (3D) environment, e.g., a simulated environment. In such instances, sensor data can also include synthetic data, for example, corresponding with data collected from a synthetic 3D environment using simulated (synthetic) sensors. [0016] As such, the training data can include various objects, such as other active traffic participants, including but not limited to: vehicles or motorists, bicyclists, and/or pedestrians, etc. In some aspects, the training data can include other objects that are detected by vehicle sensors (either real or simulated)).

Govardhanam fails to teach:
And augmenting the real-world AV scene data with a portion of the synthetic AV scene data that describes the first object by replacing a portion of the real-world AV scene data with the portion of the synthetic AV scene data,

Hong teaches:
and augmenting the real-world AV scene data with a portion of the synthetic AV scene data that describes the first object by replacing a portion of the real-world AV scene data with the portion of the synthetic AV scene data (Hong [Pg 4 Par 8] According to the solution of the embodiment of the invention, the target test case indicates the first scene element, using the first scene element to replace the scene element in the candidate virtual scene, avoiding each test again loading all scene elements in the target virtual scene, reducing the time cost of the test;), 

Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Govardhanam with Hong. Replacing a portion of the real world scene with part of the synthetic scene, as in Hong, would benefit the Govardhanam teachings by allowing for a way to augment the scenes together. Additionally, this is the application of a known technique, Replacing a portion of the real world scene with part of the synthetic scene, to yield predictable results.

Regarding claim 2,
Govardhanam and Hong teach:
The computer-implemented method of claim 1, further comprising (as seen above): 
training a machine learning model based on the augmented real-world AV scene data (Govardhanam: [0046]: In this example, the data center 450 includes a data management platform 452, an Artificial Intelligence/Machine Learning (AI/ML) platform 454).

Regarding claim 3,
Govardhanam and Hong teach:
The computer-implemented method of claim 1 (as seen above), 
wherein augmenting the real- world AV scene data with the portion of the synthetic AV scene data that describes the first object comprises (as seen above): 
identifying a first synthetic data point in the portion of the synthetic AV scene data, the first synthetic data point being associated with a first synthetic distance value indicating a distance of the first synthetic data point from a position of the AV within the simulation of the real-world scenario (Govardhanam: [0023]: However, in the example of FIG. 2B, object 204 is provided at multiple different locations, and at different distances from AV 202);
identifying a first real-world data point in the real-world AV scene data that corresponds to the first synthetic data point (Govardhanam: [0020]: For both LiDAR and camera sensors, pixel intensity variations can also depend on various factors, including but not limited to: atmospheric conditions, daylight conditions, object reflectivity properties, object distance properties), 
the first real-world data point being associated with a first real-world distance value indicating a distance of the first real- world data point from a position of the AV within the real-world scenario (Govardhanam: [0024]; The modification of the road data necessary to insert object 204 into the scene (e.g., either scene 200 or 201), is performed in a manner that takes consideration of the object distance from the AV 202); 
and modifying the real-world AV scene data based on a comparison of the first synthetic distance value to the first real-world distance value (Govardhanam: [0024]; The modification of the road data necessary to insert object 204 into the scene (e.g., either scene 200 or 201), is performed in a manner that takes consideration of the object distance from the AV 202).

Regarding claim 4,
Govardhanam and Hong teach:
The computer-implemented method of claim 3 (as seen above), 
wherein modifying the real- world AV scene data based on the comparison of the first synthetic distance value to the first real-world distance value comprises (as seen above): 
replacing the first real-world data point with the first synthetic data point based on determining that the first synthetic distance value is less than the first real-world distance value (Govardhanam: [0024]: The modification of the road data necessary to insert object 204 into the scene (e.g., either scene 200 or 201), is performed in a manner that takes consideration of the object distance from the AV 202; [0020]: The process of augmenting training data to perform object insertion can require the modification of the sensor data in a manner that comports with the physical constraints of the corresponding sensor modality.).

Regarding claims 8-11, Govardhanam and Hong teach the system limitations of these claims as discussed above with respect to claims 1-4, including  one or more computer processors (Govardhanam: [0054]: FIG. 5 illustrates an example processor-based system with which some aspects of the subject technology can be implemented; [0054]: In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described;) and one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations (Govardhanam: [0054]: In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described; [0056]: Example system 500 includes at least one processing unit (CPU or processor) 510 and connection 505 that couples various system components including system memory 515, such as read-only memory (ROM) 520 and random-access memory (RAM) 525 to processor 510).

Regarding claim 15-18, Govardhanam and Hong teach the limitations of these claims as discussed above with respect to claims 1-4, including a non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of one or more computing devices, cause the one or more computing devices to perform operations (Govardhanam: [0054]: In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described; [0056]: Example system 500 includes at least one processing unit (CPU or processor) 510 and connection 505 that couples various system components including system memory 515, such as read-only memory (ROM) 520 and random-access memory (RAM) 525 to processor 510).


Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being un patentable over Govardhanam et al. (WO 2022265664) in view of Hong et al. (CN 202080004653).

Regarding claim 7,
Govardhanam and Hong teach: 
The computer-implemented method of claim 1 (as seen above),

Govardhanam does not teach:
	adding a second object to the simulation of the real-world scenario; and augmenting the real-world AV scene data with a portion of the synthetic AV scene data that describes the second object.

While Govardhanam does not explicitly teach a specific second object and augmenting a real-world AV scene data with a portion of the synthetic AV scene data that describes the second object, this is the mere duplication of parts and associated acts in Govardhanam of the specific first object and augmenting a real-world AV scene data with a portion of the synthetic AV scene data that describes the first object as shown above in the rejection of claim 1, which has no patentable significance unless a new and unexpected result is produced. See MPEP 2144.04(VI)(B)).

Before the effective filing date of the claimed invention, it would have been
obvious to a person having ordinary skill in the art to use a combination of parts in Govardhanam. Using a second object and augmenting a real-world AV scene data with a portion of the synthetic AV scene data that describes the second object, would benefit the Govardhanam teachings by allowing for the ability to add multiple objects in the simulation of the real-world scenario, which would create more realism in the simulation. 

Regarding claim 14, Govardhanam teaches the system limitations of this claim as discussed above with respect to claim 7.

Claim(s) 5, 12 and 19 is/are rejected under 35 U.S.C. 103 as being un patentable over Govardhanam et al. (WO 2022265664) and Hong et al. (CN 202080004653) in view of Heimberger et al. (WO 2015090843).
	
Regarding claim 5 
Govardhanam and Hong teach:
	The computer-implemented method of claim 3 (as seen above), 
wherein modifying the real- world AV scene data based on the comparison of the first synthetic distance value to the first real-world distance value comprises (as seen above): 

Govardhanam fails to teach:
maintaining the first real-world data point based on determining that the first synthetic distance value is less than the first real-world distance value.

Heimberger teaches:
	maintaining the first real-world data point based on determining that the first synthetic distance value is less than the first real-world distance value (Heimberger: [Page 8 Paragraph 5]: The evaluation device 5 now checks whether the measured distance 17 is less than or greater than a distance of the potential marking 7 "from the motor vehicle 1).

Before the effective filing date of the claimed invention, it would have been
obvious to a person having ordinary skill in the art to combine the teachings of
Heimberger with Govardhanam. Evaluating whether the distance is less than or greater than a set distance value, as in Heimberger, would benefit the Govardhanam teachings by allowing for the ability to maintain data points based on whether or not the distance is less than or greater than a specific threshold, which would enhance accuracy. Additionally, this is the application of a known technique, measuring the distance and evaluating if the distance is less than another distance, to yield predictable results.

Regarding claim 12, Govardhanam in view of Heimberger teaches the system limitations of this claim as discussed above with respect to claim 5.

Regarding claim 19, Govardhanam in view of Heimberger teaches the limitations of this claim as discussed above with respect to claim 5.

Response to Arguments
Applicant's arguments filed 3-17-2025 have been fully considered but they are not persuasive.
In regards to the applicant’s arguments with respect to the amendments to  claims 1, 8 and 15 that state that the cited prior art reference Govardhanam failed to teach the limitations of claims 1, 8, and 15, these arguments are not persuasive in view of the 35 U.S.C 103 rejections of claims 1, 8 and 15 provided in the above office action.
The applicant argues that Govardhanam does not describe the generation of synthetic scene data that "represents a simulated environment of the AV that includes the first object and corresponds to the real-world environment." At most, Govardhanam generates a representation of the object itself that does not include a representation of a simulated environment. Even if, arguendo, inserting the object into the driving scene in Govardhanam results in synthetic AV scene data (which Applicant does not concede), Govardhanam does not describe the additional step of "augmenting the real-world AV scene data with a portion of the synthetic AV scene data that describes the first object by replacing a portion of the real-world AV scene data with the portion of the synthetic AV scene data," as also recited in independent Claims 1, 8, and 15. However, this additional step of replacing the real world scene with synthetic AV data is covered by Govardhanam because the synthetic object has already been inserted into the real world driving scene (Govardhanam: [0029-0030]).

Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENIS VASILIY MINKO whose telephone number is (571)270-5226. The examiner can normally be reached Monday-Thursday 8:30-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.





/DENIS VASILIY MINKO/Examiner, Art Unit 2612                                                                                                                                                                                                        
/Said Broome/Supervisory Patent Examiner, Art Unit 2612