AOP Community Resources

Evaluating References

Evaluating References

A comprehensive search strategy will always result in a high number of references, but the biggest challenge is to determine which references are worth submitting to the Study.  Your search results can be divided into three groups: Not Relevant, Somewhat Relevant, and Very Relevant.  This may sound over-simplified, but it can be a very effective way to organize all of the results that come from your broad and narrow searches.

This process is critical to your chances of winning, as well as the overall quality of your research in the long-run.  Each Researcher has a limited number of submission opportunities in each Study, and you need to use them wisely.  Reviewing your references allows you to identify those that give you the best chance of winning, while also removing any that will clearly receive low rankings.


Before you evaluate the relevance of your references, you need to determine whether they are actually eligible to submit.  There are two primary pieces to review for eligibility:

  • LDR – All of your submissions need to have a relevant date earlier than the LDR.
  • Known Prior Art – For each of the references that you consider submitting, remember to confirm that they are not on the Known Prior Art list.  These references are already known, either through past searches or on the Study patent, and provide no extra value to the Study.

It only takes a few seconds to ensure that your reference is eligible.  However, if the reference is not eligible, it will be automatically declined from the Study.  This wastes one of your valuable submission slots and will negatively impact your long-term quality scores.


The final review of your references is meant to determine whether they are truly relevant to the Study description.  But, what does “relevant” mean?  A reference is considered “relevant” if it closely matches the Study technology.  This means that it matches almost all of the elements listed in the patent claims or in the Research Requirements.

Remember when you were selecting keywords to drive your search?  These keywords were based around the elements in the Study.  This is your starting point for determining relevance.  The easiest way to sort your references is based on how many of these elements are found in each reference.  They can be organized into three categories:

  • Not Relevant – Only cover the background of the technology and none of the detailed elements
  • Somewhat Relevant – Include a few elements, but not enough to clearly match the Study
  • Very Relevant – Include several elements in a structure that closely describes the Study

Obviously, the most relevant references are the most valuable to submit because they give you the best chance of winning.  However, the other references can also have unique factors that make them worth submitting.  This can include references that utilize unique synonyms from a different industry, or a good representation of the technology in a non-English language.  While these may be risky due to the lack of elements, they may be worth the gamble.

In addition to the number of elements, it is important to consider how the elements connect.  A document with a lot of keywords is not necessarily relevant.  The elements in your reference need to work together in describing the technology in the same way as the Study description and patent.