### Pipeline run code and environment: * Command: `/home/mjs5kd/.local/bin/refgenie build mm10/bismark_bt2_index` * Compute host: udc-ba25-18 * Working dir: /sfs/qumulo/qproject/shefflab/deploy/rg.databio.org_full/rg.databio.org * Outfolder: /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/ * Pipeline started at: (04-07 20:26:39) elapsed: 0.0 _TIME_ ### Version log: * Python version: 3.6.6 * Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper` * Pypiper version: 0.12.1 * Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin` * Pipeline version: None ### Arguments passed to pipeline: * `asset_registry_paths`: `['mm10/bismark_bt2_index']` * `assets`: `None` * `command`: `build` * `config_file`: `refgenie.yaml` * `docker`: `False` * `files`: `None` * `genome`: `None` * `genome_config`: `None` * `genome_description`: `None` * `logdev`: `False` * `new_start`: `False` * `outfolder`: `/project/shefflab/deploy/rg.databio.org_full/genomes/data` * `params`: `None` * `recipe`: `None` * `recover`: `False` * `requirements`: `False` * `silent`: `False` * `skip_read_lock`: `False` * `tag_description`: `None` * `verbosity`: `None` * `volumes`: `None` ---------------------------------------- Target to produce: `/project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1_bismark_bt2_index__default.flag` > `ln -sf /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/fasta/default/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1.fa /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default` (15603)
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. PID: 15603; Command: ln; Return code: 0; Memory used: 0.001GB > `bismark_genome_preparation --bowtie2 /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default` (15604)
[33mWARNING:[0m Skipping mount /opt/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container Writing bisulfite genomes out into a single MFA (multi FastA) file Bisulfite Genome Indexer version v0.22.1 (last modified: 14 April 2019) Step I - Prepare genome folders - completed Total number of conversions performed: C->T: 552647817 G->A: 552690118 Step II - Genome bisulfite conversions - completed Bismark Genome Preparation - Step III: Launching the Bowtie 2 indexer Please be aware that this process can - depending on genome size - take several hours! Settings: Output files: "BS_CT.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.CT_conversion.fa Building a SMALL index Reading reference sizes Settings: Output files: "BS_GA.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.GA_conversion.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:43 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time reading reference sizes: 00:00:42 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:31 bmax according to bmaxDivN setting: 663195875 Using parameters --bmax 497396907 --dcv 1024 Doing ahead-of-time memory usage test Time to join reference sequences: 00:00:32 bmax according to bmaxDivN setting: 663195875 Using parameters --bmax 497396907 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 497396907 --dcv 1024 Constructing suffix-array element generator Passed! Constructing with these parameters: --bmax 497396907 --dcv 1024 Building DifferenceCoverSample Building sPrime Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder Building sPrimeOrder V-Sorting samples V-Sorting samples V-Sorting samples time: 00:04:50 Allocating rank array Ranking v-sort output V-Sorting samples time: 00:04:51 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:01:03 Invoking Larsson-Sadakane on ranks Ranking v-sort output time: 00:01:03 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:01:33 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 2.65278e+09 (target: 497396906) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Invoking Larsson-Sadakane on ranks time: 00:01:34 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 2.65278e+09 (target: 497396906) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Sorting block of length 2652783500 for bucket 1 (Using difference cover) Sorting block of length 2652783500 for bucket 1 (Using difference cover) Sorting block time: 01:57:49 Returning block of 2652783501 for bucket 1 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1325970242 fchr[G]: 1878618059 fchr[T]: 1878618059 fchr[$]: 2652783500 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 888465563 bytes to primary EBWT file: BS_GA.1.bt2 Wrote 663195880 bytes to secondary EBWT file: BS_GA.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 2652783500 bwtLen: 2652783501 sz: 663195875 bwtSz: 663195876 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 165798969 offsSz: 663195876 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 13816581 numLines: 13816581 ebwtTotLen: 884261184 ebwtTotSz: 884261184 color: 0 reverse: 0 Total time for call to driver() for forward index: 02:30:54 Reading reference sizes Time reading reference sizes: 00:00:27 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:31 Time to reverse reference sequence: 00:00:05 bmax according to bmaxDivN setting: 663195875 Using parameters --bmax 497396907 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 497396907 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples Sorting block time: 02:25:46 Returning block of 2652783501 for bucket 1 V-Sorting samples time: 00:05:04 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:01:09 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:01:44 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 2.65278e+09 (target: 497396906) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Sorting block of length 2652783500 for bucket 1 (Using difference cover) Exited Ebwt loop fchr[A]: 0 fchr[C]: 773280124 fchr[G]: 773280124 fchr[T]: 1325970242 fchr[$]: 2652783500 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 888465563 bytes to primary EBWT file: BS_CT.1.bt2 Wrote 663195880 bytes to secondary EBWT file: BS_CT.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 2652783500 bwtLen: 2652783501 sz: 663195875 bwtSz: 663195876 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 165798969 offsSz: 663195876 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 13816581 numLines: 13816581 ebwtTotLen: 884261184 ebwtTotSz: 884261184 color: 0 reverse: 0 Total time for call to driver() for forward index: 03:03:11 Reading reference sizes Time reading reference sizes: 00:00:29 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:32 Time to reverse reference sequence: 00:00:04 bmax according to bmaxDivN setting: 663195875 Using parameters --bmax 497396907 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 497396907 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:04:06 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:56 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:01:41 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 2.65278e+09 (target: 497396906) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Sorting block of length 2652783500 for bucket 1 (Using difference cover) Sorting block time: 02:27:49 Returning block of 2652783501 for bucket 1 Sorting block time: 02:08:09 Returning block of 2652783501 for bucket 1 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1325970242 fchr[G]: 1878618059 fchr[T]: 1878618059 fchr[$]: 2652783500 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 888465563 bytes to primary EBWT file: BS_GA.rev.1.bt2 Wrote 663195880 bytes to secondary EBWT file: BS_GA.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 2652783500 bwtLen: 2652783501 sz: 663195875 bwtSz: 663195876 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 165798969 offsSz: 663195876 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 13816581 numLines: 13816581 ebwtTotLen: 884261184 ebwtTotSz: 884261184 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 03:00:50 Exited Ebwt loop fchr[A]: 0 fchr[C]: 773280124 fchr[G]: 773280124 fchr[T]: 1325970242 fchr[$]: 2652783500 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 888465563 bytes to primary EBWT file: BS_CT.rev.1.bt2 Wrote 663195880 bytes to secondary EBWT file: BS_CT.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 2652783500 bwtLen: 2652783501 sz: 663195875 bwtSz: 663195876 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 165798969 offsSz: 663195876 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 13816581 numLines: 13816581 ebwtTotLen: 884261184 ebwtTotSz: 884261184 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 02:31:14Command completed. Elapsed time: 5:35:57. Running peak memory: 27.447GB. PID: 15604; Command: bismark_genome_preparation; Return code: 0; Memory used: 27.447GB > `touch /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1_bismark_bt2_index__default.flag` (13714)
Command completed. Elapsed time: 0:00:00. Running peak memory: 27.447GB. PID: 13714; Command: touch; Return code: 0; Memory used: 0.001GB Asset digest: 09ae85dcba522cfac08a392dc4c45fd1 Default tag for '0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index' set to: default ### Pipeline completed. Epilogue * Elapsed time (this run): 5:36:25 * Total elapsed time (all runs): 5:35:57 * Peak memory (this run): 27.4468 GB * Pipeline completed time: 2021-04-08 02:03:05 Finished building 'bismark_bt2_index' asset Created alias directories: - /project/shefflab/deploy/rg.databio.org_full/genomes/alias/mm10/bismark_bt2_index/default