Jump to content

Patent Application 18166918 - MODEL DESIGN AND EXECUTION FOR EMPTY SHELF - Rejection

From WikiPatents

Patent Application 18166918 - MODEL DESIGN AND EXECUTION FOR EMPTY SHELF

Title: MODEL DESIGN AND EXECUTION FOR EMPTY SHELF DETECTION

Application Information

  • Invention Title: MODEL DESIGN AND EXECUTION FOR EMPTY SHELF DETECTION
  • Application Number: 18166918
  • Submission Date: 2025-05-15T00:00:00.000Z
  • Effective Filing Date: 2023-02-09T00:00:00.000Z
  • Filing Date: 2023-02-09T00:00:00.000Z
  • National Class: 382
  • National Sub-Class: 103000
  • Examiner Employee Number: 99323
  • Art Unit: 2662
  • Tech Center: 2600

Rejection Summary

  • 102 Rejections: 0
  • 103 Rejections: 4

Cited Patents

The following patents were cited in the rejection:

Office Action Text


    DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .

Priority
Acknowledgement is made of Applicant’s claim of priority from U.S. Provisional Application No. 63/308,387, filed February 9, 2022.

Information Disclosure Statement
The information disclosure statement (“IDS”) filed on February 9, 2023 was reviewed and the listed references were noted.

Claim Objections
Claims 2 and 10 are objected to because of the following informalities: “by either adjust…” in claim 2 should read “by either adjusting…” and “sending the images corresponding to the plurality product displays…” in claim 10 should read “sending the images corresponding to the plurality of product displays…” Appropriate correction is required.

Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. 

The following is a quotation of pre-AIA  35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.

The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art.  The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, is invoked. 
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph:
(A)	the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function; 
(B)	the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and 
(C)	the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function. 
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function. 
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function. 
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.

This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.  Such claim limitation(s) is/are: “data cleaning pipeline stage”, “data annotation pipeline stage”, “model training pipeline stage”, and “inference optimization pipeline stage” in claims 6 and 18-19.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph, applicant may:  (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA  35 U.S.C. 112, sixth paragraph.

Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b)  CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.


The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.


Claims 6-7, 16 and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA  35 U.S.C. 112, the applicant), regards as the invention.

Claims 6 and 18 recite the limitation "the model deployment platform".  There is insufficient antecedent basis for this limitation in the claim. For examination purposes, these claims will be interpreted as if “a model deployment platform” were stated.

Claims 7 and 16 recite the limitation “the type of aisle”. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, these claims will be interpreted as if “a type of aisle” were stated.

Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.


Claims 1-5 and 7-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite a system, method, and non-transitory computer-readable medium for empty shelf detection. Consider method claim 10:

Step 1:
	With regard to Step 1, the instant claim is directed to a method or a process; and therefore, the claim is directed to one of the statutory categories of invention.

Step 2A, Prong One:
	With regard to 2A, Prong One, the limitations “sending the images corresponding to the plurality of product displays to an in-store computing system in realtime”, “implementing a machine learning model to analyze the image and annotate the images with indications of an empty space”, “determining a quantity of empty space at the plurality of product displays corresponding to the images” and “adding a product to the product display at a location corresponding to the empty space” as drafted, recite an abstract idea, such as a process that, under its broadest reasonable interpretation, covers performance of the limitations manually and in the mind of a person. That is, a user or person skilled in the art may manually examine images of a product display in realtime, analyze the images and annotate them with indications of an empty space, determine a number of empty spaces in the image, and add a product to a display at a location corresponding to an empty space. This is the concept that falls under the grouping of abstract ideas mental processes, i.e., a concept performed in the human mind, evaluation, judgement, and/or opinion of the user.

Step 2A, Prong Two:
	The 2019 PEG defines the phrase “integration into a practical application” to require an additional step or a combination of additional steps in the claim to apply, rely on, or use the judicial exception. In the instant case, the additional step of “detecting, by a plurality of cameras each arranged at a retail location, an image corresponding to a corresponding plurality of product displays” is considered to be extra-solution activity of gathering information. In addition, with respect to the system claim of claim 1, the mere recitation of a generic processor, memory, storage medium, or machine learning model to perform/store programming instructions of the recited/identified abstract idea does not integrate the identified abstract idea into a practical application. Accordingly, the above-mentioned additional elements/limitations do not integrate the abstract idea into a practical application; and therefore, the independent claims recite an abstract idea.

Step 2B:
	Because the claims fail under Step 2A, the claims are further evaluated under Step 2B. The claims herein do not include additional elements that are sufficient to amount to significantly more than the judicial exception, because as discussed above with respect to integration of the abstract idea into practical application, the additional elements/limitations to perform the recited steps, amount to no more than insignificant extra-solution activity. Mere instructions to apply an exception using a generic component cannot provide an inventive concept. Therefore, independent claims 1 and 10 are not patent eligible. In addition, claims 2-5, 7-9 and 11-17 of the instant application provide limitations that both individually or in combination do not integrate the identified abstract idea into a practical application or provide significantly more than the identified abstract idea.
	
	Independent claim 19 and dependent claims 6 and 18 recite significantly more than the recited abstract idea. The limitation of “an inference optimization pipeline stage that is executable on the one or more in-store computing systems to perform one or more quantization or pruning operations on the trained model” overcomes the abstract idea because the step cannot be performed mentally or manually by a person skilled in the art. Thus, claims 6 and 18-20 overcome the 35 USC 101 abstract idea rejection.

Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.


The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.

This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.

Claims 1, 3-5, 7-8, 10 and 12-17 are rejected under 35 U.S.C. 103 as being unpatentable over Zou et al. (US 11,842,321 B1, filed March 17, 2021) in view of Adato et al. (US 2019/0213546 A1).

Regarding claim 1, Zou teaches a system for real-time, on-site empty shelf detection, the system comprising:
a plurality of cameras configured to capture corresponding images at a plurality of product displays at a retail environment (Zou, Col. 11, lines 62-67, multiple cameras are positioned throughout the store and oriented to capture and provide images of the various fixtures and the product instances supported or held by the fixtures);
an in-store computing system configured to receive the images from the plurality of cameras, the in-store computing system comprising (Zou, Col. 12, lines 15-23, the cameras are configured to capture still images of the fixtures and to provide the still images to one or more computer systems for processing. The computer systems may use the images for performing tasks related to inventory, checkout, payroll, time scheduling, and/or other aspects of store management):
a memory storing a machine learning model for empty space detection (Zou, Col. 14, lines 4-15, the classifier may comprise a convolutional neural network (CNN). Col. 39, lines 60-66, the memory provides storage of computer-readable instructions, data structures, program modules, and other data for the operation of the servers); and
a processor configured to implement the machine learning model to analyze the images and annotate the images with indications of empty space therein (Zou, Col. 39, lines 28-38, the servers may include one or more hardware processors configured to execute one or more stored instructions. Col. 32 line 61 – Col. 33 line 3, defining a fourth bounding box, representing, an empty space, between two bounding boxes. The planogram data may be updated to indicate that the bounding box associated with the corresponding 3D coordinates is an empty space on the shelf or other fixture).
Although Zou teaches annotating an empty space on the shelf with a bounding box (Zou, Col. 32 line 61 – Col. 33 line 3), Zou does not explicitly teach “wherein the machine learning model is configured to determine a quantity of empty space at the plurality of different product displays corresponding to the images from the plurality of cameras”. However, in an analogous field of endeavor, Adato teaches the size of a vacant space may be determined and quantified by any suitable technique. A vacant space may be characterized by a linear measurement, a unit of area, or a unit of volume (e.g., cubic centimeters, cubic inches, etc.) (Adato, Para. [0593]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Zou with the teachings of Adato by including determining from the bounding box the volume of the vacant space. One having ordinary skill in the art would have been motivated to combine these references, because doing so would allow for automatically analyzing images of products displayed in retail stores for providing one or more functions associated with the products, as recognized by Adato. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.

Regarding claim 3, Zou in view of Adato teaches the system of claim 1, and further teach wherein annotating the image with the indication of an empty space comprises annotating the image with a flat face representing a front of an empty shelf section (Zou, Fig. 29, empty space 2910, Col. 29, lines 36-65, the product-volume detection component may generate a new bounding box corresponding to the empty space. The created bounding box may be defined by coordinates corresponding to side faces that touch the side faces of the first and second bounding boxes, a bottom and front face that corresponds to the aligned bottom and front faces of the first and second bounding boxes).

Regarding claim 4, Zou in view of Adato teaches the system of claim 3, and further teaches wherein the quantity of empty space at the product display is a volume of a cuboid region on the product display behind the flat face (Adato, Para. [0593], the size of a vacant space may be determined and quantified by any suitable technique. A vacant space may be characterized by a linear measurement, a unit of area, or a unit of volume (e.g., cubic centimeters, cubic inches, etc.)).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 4 and are incorporated herein by reference.  Thus, the system recited in Claim 4 is met by Zou in view of Adato.

Regarding claim 5, Zou in view of Adato teaches the system of claim 3, and further teaches wherein annotating the image with the indication of an empty space further comprises annotating the image with a flat face representing a back end of the empty shelf section (Zou, Fig. 29, empty space 2910, Col. 29, lines 36-65, the product-volume detection component may generate a new bounding box corresponding to the empty space. The created bounding box may be defined by coordinates corresponding to side faces that touch the side faces of the first and second bounding boxes, a bottom and front face that corresponds to the aligned bottom and front faces of the first and second bounding boxes).

Regarding claim 7, Zou in view of Adato teaches the system of claim 1, and further teaches wherein determining the type of aisle corresponding to the at least one image comprises determining a type of product located at the aisle (Adato, Para. [0585], system may access a database to access data indicating where products of a certain type, category, brand, etc. are located in a retail store and use that information to determine where an area represented in an image, for example, is located within a retail store. The area may be identified as an aisle within a store, as a certain shelf within an aisle, etc.).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 7 and are incorporated herein by reference.  Thus, the system recited in Claim 7 is met by Zou in view of Adato.

Regarding claim 8, Zou in view of Adato teaches the system of claim 7, and further teaches wherein the type of product located at the type of aisle is a product that is stacked on a shelf (Adato, Para. [0585], the area of the retail store may be identified by analyzing one or more images of the retail store to identify in the one or more images regions corresponding to the desired area, by analyzing a store map to identify in the one or more images regions corresponding to the desired area, and so forth. For example, the area may be identified based on the products detected in images depicting a store shelf).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 8 and are incorporated herein by reference.  Thus, the system recited in Claim 8 is met by Zou in view of Adato.

Regarding claim 10, Zou teaches a method for empty shelf detection, the method comprising:
detecting, by a plurality of cameras each arranged at a retail location, an image corresponding to a corresponding plurality of product displays (Zou, Col. 11, lines 62-67, multiple cameras are positioned throughout the store and oriented to capture and provide images of the various fixtures and the product instances supported or held by the fixtures);
sending the images corresponding to the plurality product displays to an in-store computing system (Zou, Col. 12, lines 15-23, the cameras are configured to capture still images of the fixtures and to provide the still images to one or more computer systems for processing. The computer systems may use the images for performing tasks related to inventory, checkout, payroll, time scheduling, and/or other aspects of store management);
implementing a machine learning model to analyze the image and annotate the images with indications of an empty space (Zou, Col. 32 line 61 – Col. 33 line 3, defining a fourth bounding box, representing, an empty space, between two bounding boxes. The planogram data may be updated to indicate that the bounding box associated with the corresponding 3D coordinates is an empty space on the shelf or other fixture).
Although Zou teaches annotating an empty space on the shelf with a bounding box (Zou, Col. 32 line 61 – Col. 33 line 3), Zou does not explicitly teach receiving the images in “realtime”, “determining a quantity of empty space at the plurality of product displays corresponding to the images” and “adding a product to the product display at a location corresponding to the empty space”. However, in an analogous field of endeavor, Adato teaches a processing device that receives images in real-time (Adato, Para. [0678]), teaches the size of a vacant space may be determined and quantified by any suitable technique. A vacant space may be characterized by a linear measurement, a unit of area, or a unit of volume (e.g., cubic centimeters, cubic inches, etc.) (Adato, Para. [0593]), and teaches a restocking event may be detected when a part of a shelf is determined to be empty, when an amount of products on a part of a shelf is determined to be below a threshold associated with the part of the shelf and/or with the product type of the products (for example, according to a planogram), and so forth (Adato, Para. [0657]).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 10 and are incorporated herein by reference.  Thus, the method recited in Claim 10 is met by Zou in view of Adato.

Regarding claim 12, Zou in view of Adato teaches the method of claim 10, and further teach wherein annotating the image with the indication of an empty space comprises annotating the image with a flat face representing a front of an empty shelf section (Zou, Fig. 29, empty space 2910, Col. 29, lines 36-65, the product-volume detection component may generate a new bounding box corresponding to the empty space. The created bounding box may be defined by coordinates corresponding to side faces that touch the side faces of the first and second bounding boxes, a bottom and front face that corresponds to the aligned bottom and front faces of the first and second bounding boxes).

Regarding claim 13, Zou in view of Adato teaches the method of claim 12, and further teach wherein the quantity of empty space at the product display is a volume of a cuboid region on the product display behind the flat face (Adato, Para. [0593], the size of a vacant space may be determined and quantified by any suitable technique. A vacant space may be characterized by a linear measurement, a unit of area, or a unit of volume (e.g., cubic centimeters, cubic inches, etc.)).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 13 and are incorporated herein by reference.  Thus, the method recited in Claim 13 is met by Zou in view of Adato.

Regarding claim 14, Zou in view of Adato teaches the method of claim 12, and further teaches wherein annotating the image with the indication of an empty space further comprises annotating the image with a flat face representing a back end of the empty shelf section (Zou, Fig. 29, empty space 2910, Col. 29, lines 36-65, the product-volume detection component may generate a new bounding box corresponding to the empty space. The created bounding box may be defined by coordinates corresponding to side faces that touch the side faces of the first and second bounding boxes, a bottom and front face that corresponds to the aligned bottom and front faces of the first and second bounding boxes).

Regarding claim 15, Zou in view of Adato teaches the method of claim 14, and further teaches wherein the quantity of empty space at the product display is a volume of a cuboid region between the flat face representing the front of the empty shelf section and the flat face representing the back end of the empty shelf section (Adato, Para. [0593], the size of a vacant space may be determined and quantified by any suitable technique. A vacant space may be characterized by a linear measurement, a unit of area, or a unit of volume (e.g., cubic centimeters, cubic inches, etc.)).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 15 and are incorporated herein by reference.  Thus, the method recited in Claim 15 is met by Zou in view of Adato.

Regarding claim 16, Zou in view of Adato teaches the method of claim 10, and further teaches wherein determining the type of aisle corresponding to the at least one image comprises determining a type of product located at the aisle (Adato, Para. [0585], system may access a database to access data indicating where products of a certain type, category, brand, etc. are located in a retail store and use that information to determine where an area represented in an image, for example, is located within a retail store. The area may be identified as an aisle within a store, as a certain shelf within an aisle, etc.).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 16 and are incorporated herein by reference.  Thus, the method recited in Claim 16 is met by Zou in view of Adato.

Regarding claim 17, Zou in view of Adato teaches the method of claim 16, and further teaches wherein the type of product located at the type of aisle is a product that is stacked on a shelf (Adato, Para. [0585], the area of the retail store may be identified by analyzing one or more images of the retail store to identify in the one or more images regions corresponding to the desired area, by analyzing a store map to identify in the one or more images regions corresponding to the desired area, and so forth. For example, the area may be identified based on the products detected in images depicting a store shelf).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 17 and are incorporated herein by reference.  Thus, the method recited in Claim 17 is met by Zou in view of Adato.

Claims 2 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Zou et al. (US 11,842,321 B1, filed March 17, 2021) in view of Adato et al. (US 2019/0213546 A1), as applied to claims 1, 3-5, 7-8, 10 and 12-17 above, and further in view of Kumar (US 11,468,400 B1).

Regarding claim 2, Zou in view of Adato teaches the system of claim 1, as described above.
Although Zou in view of Adato teaches a classifier that comprises a convolutional neural network (Zou, Col. 14, lines 4-15), they do not explicitly teach “wherein the processor is configured to conduct a drift analysis by either adjust a frequency of imaging by the at least one camera or by causing the machine learning model to be updated upon detecting a predetermined threshold of drift”. However, in an analogous field of endeavor, Kumar teaches determining whether accumulated drift is greater than a drift threshold and if the accumulated drift is greater than the drift threshold, the stable buffer is reset (Kumar, Col. 13, lines 48-64).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Zou in view of Adato with the teachings of Kumar by including conducting a drift analysis by updating the machine learning model (i.e., resetting stable buffer) when drift exceeds a threshold. One having ordinary skill in the art would have been motivated to combine the proposed references because doing so would allow for preventing accuracy degradation of the model. Thus, the claimed invention would have obvious to one having ordinary skill in the art before the effective filing date.

Regarding claim 11, Zou in view of Adato teaches the method of claim 10, as described above.
Although Zou in view of Adato teaches a classifier that comprises a convolutional neural network (Zou, Col. 14, lines 4-15), they do not explicitly teach “further comprising conducting a drift analysis by either adjusting a frequency of imaging by the at least one camera or by causing the machine learning model to be updated upon detecting a predetermined threshold of drift”. However, in an analogous field of endeavor, Kumar teaches determining whether accumulated drift is greater than a drift threshold and if the accumulated drift is greater than the drift threshold, the stable buffer is reset (Kumar, Col. 13, lines 48-64).
The proposed combination as well as the motivation for combining the Zou, Adato, and Kumar references presented in the rejection of Claim 2, apply to Claim 11 and are incorporated herein by reference.  Thus, the method recited in Claim 11 is met by Zou in view of Adato further in view of Kumar.

Claims 6 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zou et al. (US 11,842,321 B1, filed March 17, 2021) in view of Adato et al. (US 2019/0213546 A1), as applied to claims 1, 3-5, 7-8, 10 and 12-17 above, and further in view of Yi et al. (US 10,943,096 B2) and Nitzan et al. (US 2023/0267335 A1, with priority to U.S. Provisional Application No. 63/151,839, filed February 22, 2021, which teaches the subject matter used herein).

Regarding claim 6, Zou in view of Adato teaches the system of claim 1, further comprising an image modeling system remote from the retail environment (Zou, Col. 21, lines 36-39, a remote system may receive and store image date received from the cameras and may analyze the image data), the image modeling system communicatively coupled to the inference server (Zou, Col. 21, lines 24-35, the environment communicable couped to a system (i.e., remote system) comprising one or more servers via one or more networks) and comprising a model development pipeline that includes:
a data annotation pipeline stage that is executable on the one or more in-store computing systems to receive annotations of the  (Zou, Col. 17, lines 19-27, classifiers may be trained on manually annotated images to identify different items (i.e., empty locations));
a model training pipeline stage that is executable on the one or more in-store computing systems to form a trained model usable to identify empty shelf regions, the trained model being based on the(Zou, Col. 16 line 64 – Col. 17 line 7, the classifier can be trained using supervised learning, based on training images that have been manually annotated to show image segments corresponding to product instances or product lanes); and
wherein the model deployment platform is configured to:
receive, in a realtime data stream, one or more shelf camera images from cameras installed at the retail location (Adato, Para. [0678], processing device may continuously analyze the plurality of images and/or continuously receive real-time images); and
generate an output data stream indicative of shelf and product availability information based on the trained model generated via the model development pipeline (Zou, Col. 35, lines 22-32, the output data comprises information about the event. For example, where the event comprises an item being removed from an inventory location, the output data may comprise an item identifier indicative of the particular item that was removed from the inventory location and a user identifier of a user that removed the item. Output data may also include planogram data, such as coordinates of product volumes within the facility).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 6 and are incorporated herein by reference.
Although Zou in view of Adato teaches providing images of the various fixtures and product instances (Zou, Col. 11, lines 62-67), they do not explicitly teach “a data cleaning pipeline stage that is executable on the one or more in-store computing systems to create a filtered data set of image samples of a retail shelf, the image samples meeting predefined quality criteria”. However, in an analogous field of endeavor, Yi teaches performing a data cleaning operation to identify a subset of images in the group of images which are “noisy” or “dirty”, e.g., being incorrectly-labeled and/or having very poor image quality. The disclosed data cleaning technique for raw training dataset includes an iterative operation which repeats a common data cleaning procedure for each and every group of identically-labeled face image (Yi, Col. 15, lines 6-26) which outputs high-quality training dataset comprising groups of cleaned and balanced images (Yi, Col. 25, lines 20-30).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Zou in view of Adato with the teachings of Yi by including a data cleaning pipeline to clean the shelf images to create a filtered data set that meets quality criteria. One having ordinary skill in the art would have been motivated to combine these references, because doing so would allow for generating a clean and balanced training dataset, as recognized by Yi.
Although Zou in view of Adato further in view of Yi teaches a classifier trained using supervised learning (Zou, Col. 16 line 64 – Col. 17 line 7), they do not explicitly teach “an inference optimization pipeline stage that is executable on the one or more in-store computing systems to perform one or more quantization or pruning operations on the trained model”. However, in an analogous field of endeavor, Nitzan teaches optimizing the model for inference, where possible optimizations include pruning and quantization (Nitzan, Para. [0085]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Zou in view of Adato further in view of Yi with the teachings of Nitzan by including performing an inference optimization such as quantization or pruning. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for an optimized machine learning model, as recognized by Nitzan. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.

Regarding claim 18, Zou in view of Adato teaches the method of claim 10 further comprising:
obtaining the machine learning model from an image modeling system remote from the retail environment (Zou, Col. 21, lines 36-39, a remote system may receive and store image date received from the cameras and may analyze the image data), the image modeling system communicatively coupled to the inference server (Zou, Col. 21, lines 24-35, the environment communicable couped to a system (i.e., remote system) comprising one or more servers via one or more networks) and comprising a model development pipeline that includes:
a data annotation pipeline stage that is executable on the one or more in-store computing systems to receive annotations of the filtered data set of image samples identifying one or more empty locations (Zou, Col. 17, lines 19-27, classifiers may be trained on manually annotated images to identify different items (i.e., empty locations));
a model training pipeline stage that is executable on the one or more in-store computing systems to form a trained model usable to identify empty shelf regions, the trained model being based on the filtered data set of image samples and associated annotations (Zou, Col. 16 line 64 – Col. 17 line 7, the classifier can be trained using supervised learning, based on training images that have been manually annotated to show image segments corresponding to product instances or product lanes);
wherein the model deployment platform is configured to:
receive, in a realtime data stream, one or more shelf camera images from cameras installed at the retail location (Adato, Para. [0678], processing device may continuously analyze the plurality of images and/or continuously receive real-time images); and
generate an output data stream indicative of shelf and product availability information based on the trained model generated via the model development pipeline (Zou, Col. 35, lines 22-32, the output data comprises information about the event. For example, where the event comprises an item being removed from an inventory location, the output data may comprise an item identifier indicative of the particular item that was removed from the inventory location and a user identifier of a user that removed the item. Output data may also include planogram data, such as coordinates of product volumes within the facility). 
Although Zou in view of Adato teaches providing images of the various fixtures and product instances (Zou, Col. 11, lines 62-67), they do not explicitly teach “a data cleaning pipeline stage that is executable on the one or more in-store computing systems to create a filtered data set of image samples of a retail shelf, the image samples meeting predefined quality criteria”. However, in an analogous field of endeavor, Yi teaches performing a data cleaning operation to identify a subset of images in the group of images which are “noisy” or “dirty”, e.g., being incorrectly-labeled and/or having very poor image quality. The disclosed data cleaning technique for raw training dataset includes an iterative operation which repeats a common data cleaning procedure for each and every group of identically-labeled face image (Yi, Col. 15, lines 6-26) which outputs high-quality training dataset comprising groups of cleaned and balanced images (Yi, Col. 25, lines 20-30).
The proposed combination as well as the motivation for combining the Zou, Adato, and Yi references presented in the rejection of Claim 6, apply to Claim 18 and are incorporated herein by reference.
Although Zou in view of Adato further in view of Yi teaches a classifier trained using supervised learning (Zou, Col. 16 line 64 – Col. 17 line 7), they do not explicitly teach “an inference optimization pipeline stage that is executable on the one or more in-store computing systems to perform one or more quantization or pruning operations on the trained model”. However, in an analogous field of endeavor, Nitzan teaches optimizing the model for inference, where possible optimizations include pruning and quantization (Nitzan, Para. [0085]).
The proposed combination as well as the motivation for combining the Zou, Adato, Yi and Nitzan references presented in the rejection of Claim 6, apply to Claim 18 and are incorporated herein by reference.  Thus, the method recited in Claim 18 is met by Zou in view of Adato further in view of Yi and Nitzan.

Regarding claim 19, Zou teaches a real time empty shelf detection system comprising:
one or more in-store computing systems at a retail location, the one or more in-store computing systems implementing a model development pipeline and a model deployment platform (Zou, Col. 12, lines 15-23, the cameras are configured to capture still images of the fixtures and to provide the still images to one or more computer systems for processing. The computer systems may use the images for performing tasks related to inventory, checkout, payroll, time scheduling, and/or other aspects of store management);
wherein the model development pipeline includes:
a data annotation pipeline stage that is executable on the one or more in-store computing systems to receive annotations of the filtered data set of image samples identifying one or more empty locations (Zou, Col. 17, lines 19-27, classifiers may be trained on manually annotated images to identify different items (i.e., empty locations));
a model training pipeline stage that is executable on the one or more in-store computing systems to form a trained model usable to identify empty shelf regions, the trained model being based on the filtered data set of image samples and associated annotations (Zou, Col. 16 line 64 – Col. 17 line 7, the classifier can be trained using supervised learning, based on training images that have been manually annotated to show image segments corresponding to product instances or product lanes); and
wherein the model deployment platform is configured to:
generate an output data stream indicative of shelf and product availability information based on the trained model generated via the model development pipeline (Zou, Col. 35, lines 22-32, the output data comprises information about the event. For example, where the event comprises an item being removed from an inventory location, the output data may comprise an item identifier indicative of the particular item that was removed from the inventory location and a user identifier of a user that removed the item. Output data may also include planogram data, such as coordinates of product volumes within the facility).
Although Zou teaches providing images of the various fixtures and product instances (Zou, Col. 11, lines 62-67), Zou does not explicitly teach the model deployment platform is configured to “receive, in a realtime data stream, one or more shelf camera images from cameras installed at the retail location”. However, in an analogous field of endeavor, Adato teaches continuously analyzing the plurality of images and/or continuously receive real-time images (Adato, Para. [0678]).
The proposed combination as well as the motivation for combining the Zou and Adato references presented in the rejection of Claim 1, apply to Claim 19 and are incorporated herein by reference.
Although Zou in view of Adato teaches providing images of the various fixtures and product instances (Zou, Col. 11, lines 62-67), they do not explicitly teach “a data cleaning pipeline stage that is executable on the one or more in-store computing systems to create a filtered data set of image samples of a retail shelf, the image samples meeting predefined quality criteria”. However, in an analogous field of endeavor, Yi teaches performing a data cleaning operation to identify a subset of images in the group of images which are “noisy” or “dirty”, e.g., being incorrectly-labeled and/or having very poor image quality. The disclosed data cleaning technique for raw training dataset includes an iterative operation which repeats a common data cleaning procedure for each and every group of identically-labeled face image (Yi, Col. 15, lines 6-26) which outputs high-quality training dataset comprising groups of cleaned and balanced images (Yi, Col. 25, lines 20-30).
The proposed combination as well as the motivation for combining the Zou, Adato, and Yi references presented in the rejection of Claim 6, apply to Claim 19 and are incorporated herein by reference.
Although Zou in view of Adato further in view of Yi teaches a classifier trained using supervised learning (Zou, Col. 16 line 64 – Col. 17 line 7), they do not explicitly teach “an inference optimization pipeline stage that is executable on the one or more in-store computing systems to perform one or more quantization or pruning operations on the trained model”. However, in an analogous field of endeavor, Nitzan teaches optimizing the model for inference, where possible optimizations include pruning and quantization (Nitzan, Para. [0085]).
The proposed combination as well as the motivation for combining the Zou, Adato, Yi and Nitzan references presented in the rejection of Claim 6, apply to Claim 19 and are incorporated herein by reference.  Thus, the system recited in Claim 19 is met by Zou in view of Adato further in view of Yi and Nitzan.

Regarding claim 20, Zou in view of Adato further in view of Yi and Nitzan teaches the real time empty shelf detection system of claim 19, and further teaches wherein generating an output data stream indicative of shelf and product availability information based on the trained model generated via the model development pipeline comprises annotating the one or more shelf camera images from the cameras installed at the retail location with a flat face corresponding to the empty shelf regions therein (Zou, Fig. 29, empty space 2910, Col. 29, lines 36-65, the product-volume detection component may generate a new bounding box corresponding to the empty space. The created bounding box may be defined by coordinates corresponding to side faces that touch the side faces of the first and second bounding boxes, a bottom and front face that corresponds to the aligned bottom and front faces of the first and second bounding boxes).

Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Zou et al. (US 11,842,321 B1, filed March 17, 2021) in view of Adato et al. (US 2019/0213546 A1), as applied to claims 1, 3-5, 7-8, 10 and 12-17 above, and further in view of Shah et al. (US 2014/0299663 A1).

Regarding claim 9, Zou in view of Adato teaches the system of claim 7, as described above.
Although Zou in view of Adato teaches determining a type of aisle based on the type of product on the shelf in that aisle (Adato, Para. [0585]), they do not explicitly teach “wherein the type of product located at the type of aisle is a product that is arranged on a hanger”. However, in an analogous field of endeavor, Shah teaches identifying different types of products on the display hanger and determining a quantity of each of the different types of products (Shah, Para. [0043]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Zou in view of Adato with the teachings of Shah by determining a type of aisle based on the type of products on display hangers. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for monitoring inventory in a store, as recognized by Shah. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.

Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emma Rose Goebel whose telephone number is (703)756-5582. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.



/Emma Rose Goebel/Examiner, Art Unit 2662                                                                                                                                                                                                        /AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662                                                                                                                                                                                                        


    
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
        
            
    


Cookies help us deliver our services. By using our services, you agree to our use of cookies.