Evolving validation and approval guidelines require more samples and variant-type-specific data. Rapid advances in NGS platforms mean more complex tests are becoming commonplace. In this environment, ensuring compliance with the individual requirements of different regulatory bodies is a challenge.
To help address this challenge, we’ve created a blog series on three crucial topics as a critical update and supplement to our popular e-book, Next-Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory. You may already be familiar with this free guide. It’s a great resource and starting point for planning assay validation, with the blogs incorporating the latest global regulatory recommendations.
This is the second article of the series and focuses on the limit of detection (LOD). The first article covered accuracy determination, and the last one will address precision validation.
The limit of detection (LOD) of an NGS assay can be defined as the variant allele frequency (VAF) or copy number detected in 95% of samples or test runs. Usually, LOD refers to the lower LOD, which is the lower limit of allele frequency at which a variant would reliably be detected with a minimum nucleic acid input.
Because samples undergo multiple processing steps, including nucleic acid extraction (DNA/RNA), library preparation, sequencing, and a post-sequencing bioinformatics pipeline, not all DNA/RNA alleles present are reflected in the final reads used for variant calls. Efficacy and accuracy at each stage influence the LOD, making the LOD value meaningful only after other factors such as input amount, library preparation, bioinformatics pipeline settings, and minimum read depth have been validated.
Identify the minimum required nucleic acid input, tumor content input, sequencing coverage, and number of reads.
Set an acceptable range for each of these parameters that allows the test to provide expected results in 95% of test runs without a loss of accuracy (1). LOD studies can then be performed within the established assay framework.
Establish and document the LOD for each specimen type in the test’s indications for use.
Different specimen types and nucleic acid extraction methods may affect LOD, so studies should be performed for all sample types and nucleic acid purification methods used. Analytical sensitivity studies should also include tissue representative of different tumor samples and sample qualities, including challenging ones, to demonstrate equivalent performance across all nucleic acid extraction methods. For instance, cfDNA samples with varying tumor burdens, as well as FFPE samples across a range of tumor fractions and expected nucleic acid quality levels, should be included (2). Some of the samples need to have VAFs below the expected LOD to help establish a meaningful cutoff (3).
Conduct LOD studies using clinical or contrived samples.
The minimum number of clinical samples required for an LOD study is usually dictated by the governing regulatory authority. To achieve required low VAF or tumor fraction dilutions of positive samples in negative background are often needed. Contrived samples can help expand the range of variants tested and increase confidence when using orthogonally validated dilutions, but laboratories should demonstrate that the performance of contrived samples is equivalent to that of clinical samples. Recent research has shown that linearized plasmid controls perform with similar efficiency to formalin-fixed cell line genomic DNA. These controls can be used to assess or monitor LOD, as long as the genomic DNA and plasmids are fragmented to similar sizes.
Establish and document the LOD for each variant type included in the test’s indications for use.
Since different variant types may have different LODs, calculate the LOD for each variant type in different sequence contexts. The laboratory needs to establish the LOD separately for single-nucleotide variants, small and/or large indels, translocations/fusions, and copy number gains or losses as applicable. Additionally, CLSI recommends that for CNVs and fusions LOD be established on a per-gene basis.
Use sufficient samples to determine the LOD with statistical confidence.
The number of samples depends on the required statistical confidence and the number of variants per sample, since the calculation is performed for each detected variant. For example, the AMP-CAP guidelines provide a formula to calculate the number of samples required to establish confidence intervals and recommend including 59 samples in a typical scenario (see table below and (4). When using multiplexed contrived samples, this could translate to at least 60 variants across the different reference materials, dramatically reducing the amount of sequencing required.
The number of variants included in the validation affects the statistical confidence of the calculated analytical sensitivity. For example, a sample size of 60 variants is estimated to provide a maximum sensitivity of 95% (within a 95% CI), while a sample size of 300 variants increases the maximum sensitivity to 99% (within a 95% CI) (4,5). Given the difficulty in obtaining samples with certain variant types (such as indels), it is recommended that, of the 60 variants overall, at least 10 of each relevant type be assayed.
Another challenge for LOD validation is that the allele frequencies (AFs) of variants in clinical samples vary greatly, and not all variants are useful for establishing LOD. This further increases the number of clinical samples needed to gather enough variants of each type within the relevant AF range. Pooled clinical samples may be used, but contrived samples with precisely determined VAFs, such as Seraseq reference materials, offer greater confidence than sample mixtures.
The advantage of contrived materials is the number, types, and AFs of the variants align to the expected ones for LOD validation. They are also highly consistent and sustainable, supporting more extensive validation studies. The large number of variants per sample reduces the number of required sequencing runs. Although using contrived Seraseq reference materials has been successfully employed as a part of assay validation compliant with the New York State guidelines (6,8), different regulatory bodies might have different requirements and limitations regarding the use of biosynthetic samples for LOD validation.
Include clinically relevant variants and sample types in validation
If including all clinically relevant variants is not feasible, genomic regions containing these variants should meet minimum quality standards to guarantee accurate variant detection. All specimen types from the intended use population (e.g. FFPE or plasma) should be represented.
Samples or variants must have a representative distribution of reportable variants across all target areas, including GC-rich sequences, and must be verified by an independent reference method.
It is expected that the variants included in the validation are balanced across the targeted regions. New York State Department of Health stipulates that the variants should be representative of different target areas (7), with minimum number of variants validated located in regions with high probability of error (e.g., homopolymer or repetitive regions). Seraseq reference materials are designed to encompass a broad range of challenging genomic alterations, with individual variants quantified by dPCR, a widely accepted reference method for straightforward variant detection.
The positive percent agreement (PPA) is a metric related to LOD. Defined as the proportion of positive test results that truly have the genetic alteration being tested for, PPA is the number of true positives (TP), divided by the sum of TP and FN (false negatives). It is used primarily to compare test performance against an established gold standard reference method.
Different authoritative bodies offer different details on variant types, sample size, and sources used to determine the error rate. See the table below for a comparison.
|
Agency |
Required # of samples |
Variant types |
Sample source |
|
U.S FDA |
Not specified |
Separate LOD for each variant type included in the test’s indications for use |
Fresh clinical samples; if not available archival clinical samples, cell lines (or blends), or contrived reference materials |
|
CLSI |
10-20 positive samples for each variant type with both the assay and an orthogonal method |
LOD at 95% CI for all variant types included in assay claims |
Challenging clinical samples, but can also use contrived materials to augment or substitute them |
|
AMP-CAP |
59 samples to establish the 95% CI |
PPA should be calculated separately for all variant types |
Well-characterized cell lines or engineered cell lines/plasmids for rare variants |
|
New York State Department of Health |
3-5 clinical samples per variant type |
Sensitivity for each variant type at the lower limit of NA input. |
Cell line DNA mixtures (not plasmids), need to verify with 3-5 patient samples. Validate performance characteristics for each sample type (e.g., FFPE, ctDNA). |
Our upcoming article will discuss the precision validation of an NGS diagnostic assay. Subscribe to our blog so you won't miss it.
Read the first article in this series on accuracy determination.
Download our e-book Next-Generation Sequencing Assay Validation: A Practical Guide for the Clinical Genomics Laboratory.