Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

filter

Category: POST-CONSENSUS

Filter consensus reads based on quality metrics

Description

Filters consensus reads generated by simplex or duplex commands. Two kinds of filtering are performed:

  1. Masking/filtering of individual bases in reads
  2. Filtering out of reads (i.e. not writing them to the output file)

Base-level filtering/masking is only applied if per-base tags are present (see duplex and simplex for descriptions of these tags). Read-level filtering is always applied. When filtering reads, secondary alignments and supplementary records may be removed independently if they fail one or more filters; if either R1 or R2 primary alignments fail a filter then all records for the template will be filtered out.

The filters applied are as follows:

  1. Reads with fewer than min-reads contributing reads are filtered out
  2. Reads with an average consensus error rate higher than max-read-error-rate are filtered out
  3. Reads with mean base quality of the consensus read, prior to any masking, less than min-mean-base-quality are filtered out (if specified). A read with no bases (SEQ *) has a mean of 0, so it is filtered out whenever min-mean-base-quality is greater than 0 (fgbio keeps it).
  4. Bases with quality scores below min-base-quality are masked to Ns
  5. Bases with fewer than min-reads contributing raw reads are masked to Ns
  6. Bases with a consensus error rate (defined as the fraction of contributing reads that voted for a different base than the consensus call) higher than max-base-error-rate are masked to Ns
  7. Reads with a fraction or count of Ns higher than max-no-call-fraction after per-base filtering are filtered out. When max-no-call-fraction is a fraction (< 1.0), a read with no bases (SEQ *) is also filtered out.

A mapped record with no bases (SEQ *, e.g. a secondary alignment written by bwa mem -a) that is kept keeps its NM, UQ and MD tags unchanged, since there are no bases to recompute them from; fgbio fails on such a record. A record with bases but no qualities (QUAL *) gets NM and MD and keeps its UQ, as in fgbio.

Every read must carry base qualities: a read that has bases but no qualities (QUAL *) is an error.

When filtering single-umi consensus reads generated by simplex, a single value each should be supplied for –min-reads, –max-read-error-rate, and –max-base-error-rate.

When filtering duplex consensus reads generated by duplex, each of the three parameters may independently take 1-3 values. For example:

fgumi filter … –min-reads 10,5,3 –max-base-error-rate 0.1

In each case if fewer than three values are supplied, the last value is repeated (i.e. 80,40 -> 80 40 40 and 0.1 -> 0.1 0.1 0.1). The first value applies to the final consensus read, the second value to one single-strand consensus, and the last value to the other single-strand consensus. It is required that if values two and three differ, the more stringent value comes earlier.

In order to correctly filter reads in or out by template, the input BAM must be either queryname sorted or query grouped. If your BAM is not already in an appropriate order, this can be done in streaming fashion with:

fgumi sort -i in.bam –order queryname | fgumi filter -i /dev/stdin …

Output is written in the same order as the input. To emit coordinate-sorted output, pipe through fgumi sort.

The –reverse-per-base-tags option controls whether per-base tags should be reversed before being used on reads marked as being mapped to the negative strand. This is necessary if the reads have been mapped and the bases/quals reversed but the consensus tags have not. If true, the tags written to the output BAM will be reversed where necessary in order to line up with the bases and quals. It reverses the same tags as fgumi zipper –tags-to-reverse Consensus –tags-to-revcomp Consensus, so reverse in one of the two only: reversing in both restores the original, wrong orientation. fgumi realign (and fgumi runall –realign) always reverses them, so do not pass –reverse-per-base-tags on its output.

Arguments

FlagDescriptionDefault
-i, --input <INPUT>Input BAM filerequired
-o, --output <OUTPUT>Output BAM filerequired
--async-reader <ASYNC_READER>Enable async userspace prefetch on the input BAMfalse
--check-crc <CHECK_CRC>Verify each BGZF block’s CRC32 checksum while decoding the inputfalse
--no-check-crc <NO_CHECK_CRC>Skip CRC32 verification while decoding the inputfalse
-r, --ref <REFERENCE>Reference FASTA file for NM/UQ/MD tag regeneration. If not provided, alignment tag regeneration (NM/UQ/MD) is skipped. On a simplex methylation consensus (one carrying cu/ct), NM/UQ do not count the conversion of the read’s original strand, as a bisulfite-aware aligner does (directional libraries assumed); MD always lists every difference
-M, --min-reads <MIN_READS>Minimum number of raw reads to support a single-strand consensus base/read. For duplex: provide 1-3 values for [duplex, single-strand consensus, single-strand consensus]
-E, --max-read-error-rate <MAX_READ_ERROR_RATE>Maximum raw read error rate for a single-strand consensus base/read (0.0-1.0). For duplex: provide 1-3 values for [duplex, single-strand consensus, single-strand consensus]0.025
-e, --max-base-error-rate <MAX_BASE_ERROR_RATE>Maximum base error rate across raw reads (0.0-1.0). For duplex: provide 1-3 values for [duplex, AB consensus, BA consensus]0.1
-N, --min-base-quality <MIN_BASE_QUALITY>Minimum base quality score (after masking). Optional in fgumi (unlike fgbio, which requires it): when omitted, no per-base quality masking is performed
-q, --min-mean-base-quality <MIN_MEAN_BASE_QUALITY>Minimum mean base quality across the read, computed over the full read length prior to any masking (matching fgbio FilterConsensusReads). Must be a finite value >= 0
-n, --max-no-call-fraction <MAX_NO_CALL_FRACTION>Maximum no-calls (N bases) allowed in a read0.2
`-R, –reverse-per-base-tags <truefalse>`Reverse per-base tags for negative strand reads
--threads <THREADS>Number of worker threads for the processing pipeline
`–filter-by-template <truefalse>`Filter templates together (all primary reads must pass)
--rejects <REJECTS>Optional output BAM file for rejected reads
--stats <STATS>Optional output file for filtering statistics
`-s, –require-single-strand-agreement <truefalse>`Require single-strand agreement for duplex consensus (mask bases where AB and BA disagree)
--min-methylation-depth <MIN_METHYLATION_DEPTH>Minimum methylation depth (cu+ct) at methylation sites (EM-Seq/TAPs). Single-strand consensus: sites are aligned reference C/G read by the record’s strand, and those below the first value are masked to N (needs aligned records; unmapped ones are left unfiltered and counted). Duplex: sites are C/G in the record’s own SEQ, aligned or not, and 1-3 values [total, better strand, worse strand] apply: both calls of a CpG are dropped from MM/ML unless their summed depth >= total, the better half >= better and the worse half >= worse (on a single-strand duplex record the absent strand’s half counts as 0; use 0 for worse to keep them). Other duplex calls must reach the second value. Per-base tags must be in genomic orientation
`–require-strand-methylation-agreement <truefalse>`Require strand methylation agreement at CpG sites for duplex consensus (EM-Seq/TAPs). Drops the calls of both halves of a CpG from MM/ML when the strands calling its C and its G disagree on methylation status (majority of cu vs ct at each)
--min-conversion-fraction <MIN_CONVERSION_FRACTION>Minimum bisulfite/enzymatic conversion fraction at non-CpG cytosines. For EM-Seq: checks converted/total >= threshold (high conversion = good). For TAPs: checks unconverted/total >= threshold (low conversion = good). Requires –methylation-mode. Uses cu/ct tags
--methylation-mode <METHYLATION_MODE>Methylation mode for conversion fraction filtering. Required when using –min-conversion-fraction. Controls whether the conversion fraction check uses converted (em-seq) or unconverted (taps) counts as the numerator
--compression-level <COMPRESSION_LEVEL>Compression level for output BAM (0-12)1
--scheduler <SCHEDULER>Inert scheduler strategy (see [SchedulerStrategy]); the chain engine does not use it to assign thread work. Retained for backward compatibility only — the values below describe what each once meant to the removed legacy engine:balanced-chase-drain
--pool-scheduler <POOL_SCHEDULER>Override for the chain-builder pool dispatch scheduler (hidden; A/B knob)auto
`–pipeline-stats <truefalse>`Print detailed pipeline statistics at completion
--pipeline-trace <PIPELINE_TRACE>Per-edge instrumentation level: off | summary | timeline | deepoff
--pipeline-trace-out <PIPELINE_TRACE_OUT>Path for the --pipeline-trace timeline per-tick TSV (default pipeline-trace.tsv in the working directory)
--deadlock-timeout <SECONDS>Seconds without progress before the deadlock watchdog reports a stall10
`–deadlock-recover <truefalse>`Enable automatic deadlock recovery (default: false, detection only)
--pipeline-telemetry-out <PIPELINE_TELEMETRY_OUT>File stem for the per-tick thread-telemetry TSVs — six files, &lt;stem&gt;.ticks.{summary,edges,workers,steps,counters,counter_names}.tsv (summary/edges/workers/counters carry a row per tick; steps/counter_names are written once). Setting this (or the FGUMI_PIPELINE_TELEMETRY_OUT env var) enables telemetry; the default stem when neither is set but telemetry is otherwise requested is pipeline-telemetry
--pipeline-telemetry-interval <PIPELINE_TELEMETRY_INTERVAL>Sample/emit cadence for thread telemetry, e.g. 2ms, 500us, 1s (default: 2ms). Also settable via FGUMI_PIPELINE_TELEMETRY_INTERVAL
--max-memory <MAX_MEMORY>Maximum total memory for the pipeline queues768
--memory-reserve <MEMORY_RESERVE>Memory to reserve for other processes when –max-memory=autoauto
`–memory-per-thread <truefalse>`Scale the memory limit by thread count