Key Topics for Today’s NGS Assay Validation, Part 3: Precision Validation

More complex testing and the increasing need for required test continue to pose challenges for laboratories. As regulatory agencies create validation and approval guidelines to keep pace with rapid advances in NGS platforms, it can be demanding to ensure your lab complies with each regulator’s requirements.

Precision studies validation of reproducibility and repeatability, ensure the integrity of laboratory results by confirming accuracy and reliability. In this article, the third in a series focused on current topics central to NGC assay validation, we review the different requirements for precision studies and explore ways to meet the growing demand for testing samples.

 This blog series crucially updates and complements our popular e-book, Next-Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory, a useful free resource for planning assay validation, to provide the latest global regulatory recommendations. 

Reproducibility (or inter-run precision): Variation in repeated measurements of the same variable across different experiments. For NGS-based tests, it involves measuring the same variants under a range of conditions that capture the main sources of variability between sequencing runs. 

Repeatability (or intra-run precision): Variation in test results when using the same sample over a short period, while reducing other sources of variability.

Regulatory bodies provide varying although broadly aligned guidelines on variant types, sample size, and sample sources used for precision studies. See the table for a summary of the requirements outlined below. 

Reproducibility validation

New York State Department of Health (NYSDoH) publishes separate somatic and germline cancer guidelines. For somatic cancer, include at least 4-5 independent batches or runs, performed on different days by different technologists and sequencers, if possible (1). For SNVs and INDELs, each run should contain 4-5 positive samples at the lower limit of nucleic acid input and variant allele frequency of the assay being evaluated and 20 replicates each must be analyzed to achieve 95% reproducibility. Since different variant types are handled separately, reproducibility should be assessed for each variant type (see table). Structural variants (SVs) and copy number variants (CNVs) require a minimum of 3 positive samples, each with different fusion partners or copy gains/losses, in 3 separate runs on 3 different days. The guidelines for germline cancer testing specify fewer samples and are less detailed. (2)

ISO 21474: Include challenging samples at 2-3 times the assay LOD or near the clinical cutoff, and ensure rare alleles are tested for both reproducibility and repeatability.

Clinical and Laboratory Standards Institute (CLSI): Evaluate clinical samples from different specimens and tissue types and anticipated VAFs, including samples with low nucleic acid input. Test at least 10 positive samples for each variant type included in the assay’s claims, using both the assay and an orthogonal method. Contrived reference samples might be used for rare variants if the contrived sample type has been characterized as equivalent to a clinical specimen (3).

U.S. Food and Drug Administration (FDA): Although the FDA does not provide detailed validation recommendations, the 2018 guidance on NGS assays for germline diseases states that precision (reproducibility and repeatability) should be assessed for both variant and wild-type calls. Each metric should be reported separately for each condition, interrogated region, and variant type. It also advises conducting testing that considers major factors that could contribute to test variability, such as operators, days, reagent lots, instruments, lanes, test sites, and other operating conditions (4). 

Association of Molecular Pathologists and the College of American Physicians (AMP-CAP): Test at least 3 samples across all steps and over an extended period to include all instruments, testing personnel, and multiple lots of reagents. Replicate (within run) and repeat (between run) testing should be performed (5).

Best practice: Include one known negative sample, one positive contrived sample with all relevant variant types (such as a Seraseq reference material), and the number of real-world samples (preferably clinical samples or mixtures of clinical samples) as required by the guidelines being followed.

Repeatability validation

Intra-run precision studies have similar sample size requirements as inter-run precision studies. It is ideal to identify samples with enough nucleic acid to perform both experiments (although they could be combined). A key challenge in these studies is obtaining enough sample from a single clinical specimen to conduct testing with a sufficient number of replicates. It is frustrating and counterproductive to find a sample positive for a relevant variant but unable to be tested multiple times due to sample depletion.

Typically, 4-5 replicates are needed for each sample. If possible, performing the extra two batches with different technologists helps the lab ensure consistent performance across staff and gives team members a chance to demonstrate their proficiency with the assay.

Sourcing samples

The number of patient samples required can quickly surpass available supply, significantly raising sequencing costs. Therefore, the most effective approach might be to include samples with single-nucleotide variants (SNVs), indels, and, if necessary, SVs and CNVs near the assay’s detection limit. 

Since obtaining such clinical samples is challenging, a chimeric sample, such as a mixture of an SNV-containing sample and an indel-containing sample, can meet the requirements for both SNV and indel validation. Creating multiple sample mixtures further reduces the total number of tests needed, saving time and money. While some jurisdictions, like New York State, mandate the use of real-world samples, validations in this and other areas may be supplemented by contrived samples containing variants near the assay’s LOD (5).

Seraseq biosynthetic NGS reference materials include important mutations essential for precision studies and serve as a sustainable, plentiful source. This enables large-scale replicate testing and long-term quality monitoring.

Regulatory agency guidelines

Gain a clearer understanding of NGS assay validation

Read the first two articles in this series: accuracy determination and limit of detection.

Subscribe to our blog for more articles like this.

Download our e-book Next-Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory.

References

1. Next Generation Sequencing (NGS) guidelines for somatic genetic variant detection
2. Next Generation Sequencing (NGS) guidelines for germline genetic variant detection
3. Clinical & Laboratory Standards Institute, Human Genetic and Genomic Testing Using Traditional and High-Throughput Nucleic Acid Sequencing Methods
4. Considerations for Design, Development, and Analytical Validation of Next Generation Sequencing (NGS) - Based In Vitro Diagnostics (IVDs) Intended to Aid in the Diagnosis of Suspected Germline Diseases
5. Jennings LJ, et al. Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists. J Mol Diagn. 2017 May;19(3):341-365.