Key Topics for Today’s NGS Assay Validation, Part 1: Accuracy Determination

Ensuring compliance with the individual requirements of different regulatory bodies can be daunting when designing an efficient NGS assay validation or verification plan. Validation and approval guidelines continue to evolve as comprehensive multi-marker NGS tests grow, requiring more samples and variant-type-specific data.

Rapid advances in NGS platform throughput enable more multiplexed tests in challenging sample types including liquid biopsies. While complex biomarkers such as TMB or HRD, comprehensive genomic profiling, and other genome-scale tests are becoming routine in clinical diagnostics, they require more nuanced validation than established single-SNP-based testing for solid tumor and germline diagnostics. 

This level of change led us to create a blog series on three fundamental topics as a supplement to our popular e-book, Next-Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory. If you have validated an NGS LDT in the past few years, you are probably already familiar with this free guide. It remains a great resource and a starting point for planning an assay validation.

This article focuses on accuracy determination, and the next two articles will cover the limit of detection (LOD) and, finally, precision validation.

Specificity and sensitivity

In the context of NGS, accuracy indicates how well the test correctly calls a mutant variant (True Positive) or wild-type (True Negative) at a given genomic locus compared to a reference genome sequence. It is a combination of specificity (the percentage of True Positives) and sensitivity (the percentage of True Negatives). 

The error rate is then the number of false variant calls (False positives; FP + False negatives; FN) divided by the total number of valid calls. Non-calls and error calls are excluded from the calculation.

The sensitivity calculation for detecting false negatives (equivalent to the LOD) relies heavily on reference materials. We will discuss this later in the article, but here we are focusing on specificity.

To determine specificity in DNA sequencing, samples must have known variant-negative sites, but they may also contain known-positive variants, allowing full concordance to be assessed at the assay’s desired LOD, which could be as low as a 5% allele burden or even lower depending on the application. Therefore, false positives must not be detected. It is acceptable if the samples contain other clinically significant variants that are not being targeted. 

Regulatory agency guidelines

Different authoritative bodies provide varying details on variant types, sample size, and sources used to determine the error rate. See the table for a summary of the requirements outlined below. 

U.S. Food and Drug Administration (FDA)

• Does not state the required number of samples to be tested, only suggesting a minimum number needed to achieve statistical significance.
• Does not prescribe what variant types should be tested, other than that they should be clinically relevant.
• Does more directly address the sample type to use, indicating a preference for fresh clinical samples. If fresh clinical samples aren’t readily available, either cell lines or contrived reference materials are recommended (1).
• Recommends a minimum of 10-20 positive samples for each type of variant included in the assay’s claims with both the assay and an orthogonal method (2).
• States a preference for using patient sample controls over others, and that the controls need to be of the same type as those being tested.
• Sets a requirement to use only full-process contrived reference materials.
• Provides specific guidelines on the number of samples to be tested.
• Recommends using well-characterized real-world and in the case of rare variants, allows contrived reference samples for validation (3).
• Suggests using only samples of the same type as those to be tested (this restricts the use of contrived reference materials to full-process sample formats such as FFPE or plasma).
• Distinguishes between initial and full validation with different requirements.
• The initial validation requires 25 germline (4) or 50 somatic (5) patient samples, which must include a representation of all sample types to be tested and variant types in target areas (including CG-rich and homopolymer sequences).
• The number of samples required for full validation depends on the variant type being tested (see table). The updated 2024 guidelines require separate validation of SNVs and INDELS in different target regions (normal, CG-rich and homopolymer) as well as category of CNVs (single gene, moderate or large gains and losses).
• For analytical accuracy the error rate can be established using at least 2 well-characterized reference samples and must be below 2%.

Clinical and Laboratory Standards Institute (CLSI)

Association of Molecular Pathologists and the College of American Physicians (AMP-CAP)

New York State Department of Health (NYSDoH)

Each of these regulatory bodies states a preference for patient samples when obtainable. When it is acceptable to use contrived reference materials for accuracy determination, the Seraseq product line can be used to replace hard-to-source clinical samples harboring only a few variants.

SeraCare’s contrived materials feature highly multiplexed spiked-in variants in the well-characterized non-tumor cell line GM24385 and are suitable for use in accuracy studies for sensitivity. Likewise, wild-type GM24385 material is available in multiple formats for specificity analysis. Full-process FFPE and plasma Seraseq reference materials can also help meet the common requirement of including samples that match the intended test sample format in the validation study.

Limit of blank (LOB)

In NGS, the LOB is the highest apparent target concentration expected to be found in a negative sample. Related to LOD, the LOB is essential for accurately reporting results from highly sensitive liquid biopsy tests (6) but is also important in any application where a level of false-positive artifacts is expected, such as when analyzing FFPE samples.

CLSI recommends sequencing multiple well-known reference samples, such as cell lines, or using clinical samples known to be wild-type in the target genes or regions, which have been previously characterized using an orthogonal method. In both cases, these should be tested with the highest DNA input amount used for the test. Reference samples can be used during assay development, followed by clinical samples whenever possible during assay validation. They also recommend testing 30 variant-negative samples in duplicate, totaling at least 120 measurements. 

Achieving this with patient samples alone can be challenging, especially for liquid biopsy assays, where sample material is limited when testing high DNA inputs. It is also time-consuming to confirm the wild-type status of each clinical sample in the target regions before beginning validation, and doing so risks depleting the material. Reference materials can be produced at a large scale in any format to enable sufficient characterization of the batch with an orthogonal assay (if not multiple assays) and repeat measurements of the same batch to confirm the observations.

Looking for more information?

Our next article will address establishing the LOD of an NGS diagnostic assay. Subscribe to our blog so you don’t miss it.

Or download our e-book Next Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory.

References

1. Considerations for Design, Development, and Analytical Validation of Next Generation Sequencing (NGS) - Based In Vitro Diagnostics (IVDs) Intended to Aid in the Diagnosis of Suspected Germline Diseases
2. Clinical & Laboratory Standards Institute, Human Genetic and Genomic Testing Using Traditional and High-Throughput Nucleic Acid Sequencing Methods
3. Jennings LJ, et al. Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists
4. Next Generation Sequencing (NGS) guidelines for germline genetic variant detection
5. Next Generation Sequencing (NGS) guidelines for somatic genetic variant detection
6. Vincent D de Jager, et al. Reporting of molecular test results from cell-free DNA analyses: expert consensus recommendations from the 2023 European Liquid Biopsy Society ctDNA Workshop
7. Philip M Galbo, et.al. A Comprehensive Guide to Achieving New York State Clinical Laboratory Evaluation Program Approval for Next-Generation Sequencing Assays