### Pipeline run code and environment: * Command: `/home/mjs5kd/.local/bin/refgenie build mm10/bismark_bt2_index` * Compute host: udc-ba25-18 * Working dir: /sfs/qumulo/qproject/shefflab/deploy/rg.databio.org_full/rg.databio.org * Outfolder: /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/ * Pipeline started at: (04-07 20:26:39) elapsed: 0.0 _TIME_ ### Version log: * Python version: 3.6.6 * Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper` * Pypiper version: 0.12.1 * Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin` * Pipeline version: None ### Arguments passed to pipeline: * `asset_registry_paths`: `['mm10/bismark_bt2_index']` * `assets`: `None` * `command`: `build` * `config_file`: `refgenie.yaml` * `docker`: `False` * `files`: `None` * `genome`: `None` * `genome_config`: `None` * `genome_description`: `None` * `logdev`: `False` * `new_start`: `False` * `outfolder`: `/project/shefflab/deploy/rg.databio.org_full/genomes/data` * `params`: `None` * `recipe`: `None` * `recover`: `False` * `requirements`: `False` * `silent`: `False` * `skip_read_lock`: `False` * `tag_description`: `None` * `verbosity`: `None` * `volumes`: `None` ---------------------------------------- Target to produce: `/project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1_bismark_bt2_index__default.flag` > `ln -sf /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/fasta/default/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1.fa /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default` (15603)

Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB.  
  PID: 15603;	Command: ln;	Return code: 0;	Memory used: 0.001GB


> `bismark_genome_preparation --bowtie2 /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default` (15604)
WARNING: Skipping mount /opt/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
Writing bisulfite genomes out into a single MFA (multi FastA) file

Bisulfite Genome Indexer version v0.22.1 (last modified: 14 April 2019)

Step I - Prepare genome folders - completed



Total number of conversions performed:
C->T:	552647817
G->A:	552690118

Step II - Genome bisulfite conversions - completed


Bismark Genome Preparation - Step III: Launching the Bowtie 2 indexer
Please be aware that this process can - depending on genome size - take several hours!
Settings:
  Output files: "BS_CT.*.bt2"
  Line rate: 6 (line is 64 bytes)
  Lines per side: 1 (side is 64 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Max bucket size: default
  Max bucket size, sqrt multiplier: default
  Max bucket size, len divisor: 4
  Difference-cover sample period: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  genome_mfa.CT_conversion.fa
Building a SMALL index
Reading reference sizes
Settings:
  Output files: "BS_GA.*.bt2"
  Line rate: 6 (line is 64 bytes)
  Lines per side: 1 (side is 64 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Max bucket size: default
  Max bucket size, sqrt multiplier: default
  Max bucket size, len divisor: 4
  Difference-cover sample period: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  genome_mfa.GA_conversion.fa
Building a SMALL index
Reading reference sizes
  Time reading reference sizes: 00:00:43
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time reading reference sizes: 00:00:42
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:31
bmax according to bmaxDivN setting: 663195875
Using parameters --bmax 497396907 --dcv 1024
  Doing ahead-of-time memory usage test
  Time to join reference sequences: 00:00:32
bmax according to bmaxDivN setting: 663195875
Using parameters --bmax 497396907 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 497396907 --dcv 1024
Constructing suffix-array element generator
  Passed!  Constructing with these parameters: --bmax 497396907 --dcv 1024
Building DifferenceCoverSample
  Building sPrime
Constructing suffix-array element generator
Building DifferenceCoverSample
  Building sPrime
  Building sPrimeOrder
  Building sPrimeOrder
  V-Sorting samples
  V-Sorting samples
  V-Sorting samples time: 00:04:50
  Allocating rank array
  Ranking v-sort output
  V-Sorting samples time: 00:04:51
  Allocating rank array
  Ranking v-sort output
  Ranking v-sort output time: 00:01:03
  Invoking Larsson-Sadakane on ranks
  Ranking v-sort output time: 00:01:03
  Invoking Larsson-Sadakane on ranks
  Invoking Larsson-Sadakane on ranks time: 00:01:33
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 2.65278e+09 (target: 497396906)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
  No samples; assembling all-inclusive block
  Invoking Larsson-Sadakane on ranks time: 00:01:34
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 2.65278e+09 (target: 497396906)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
  No samples; assembling all-inclusive block
  Sorting block of length 2652783500 for bucket 1
  (Using difference cover)
  Sorting block of length 2652783500 for bucket 1
  (Using difference cover)
  Sorting block time: 01:57:49
Returning block of 2652783501 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 1325970242
fchr[G]: 1878618059
fchr[T]: 1878618059
fchr[$]: 2652783500
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 888465563 bytes to primary EBWT file: BS_GA.1.bt2
Wrote 663195880 bytes to secondary EBWT file: BS_GA.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 2652783500
    bwtLen: 2652783501
    sz: 663195875
    bwtSz: 663195876
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 165798969
    offsSz: 663195876
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 13816581
    numLines: 13816581
    ebwtTotLen: 884261184
    ebwtTotSz: 884261184
    color: 0
    reverse: 0
Total time for call to driver() for forward index: 02:30:54
Reading reference sizes
  Time reading reference sizes: 00:00:27
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:31
  Time to reverse reference sequence: 00:00:05
bmax according to bmaxDivN setting: 663195875
Using parameters --bmax 497396907 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 497396907 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
  Building sPrime
  Building sPrimeOrder
  V-Sorting samples
  Sorting block time: 02:25:46
Returning block of 2652783501 for bucket 1
  V-Sorting samples time: 00:05:04
  Allocating rank array
  Ranking v-sort output
  Ranking v-sort output time: 00:01:09
  Invoking Larsson-Sadakane on ranks
  Invoking Larsson-Sadakane on ranks time: 00:01:44
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 2.65278e+09 (target: 497396906)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
  No samples; assembling all-inclusive block
  Sorting block of length 2652783500 for bucket 1
  (Using difference cover)
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 773280124
fchr[G]: 773280124
fchr[T]: 1325970242
fchr[$]: 2652783500
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 888465563 bytes to primary EBWT file: BS_CT.1.bt2
Wrote 663195880 bytes to secondary EBWT file: BS_CT.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 2652783500
    bwtLen: 2652783501
    sz: 663195875
    bwtSz: 663195876
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 165798969
    offsSz: 663195876
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 13816581
    numLines: 13816581
    ebwtTotLen: 884261184
    ebwtTotSz: 884261184
    color: 0
    reverse: 0
Total time for call to driver() for forward index: 03:03:11
Reading reference sizes
  Time reading reference sizes: 00:00:29
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:32
  Time to reverse reference sequence: 00:00:04
bmax according to bmaxDivN setting: 663195875
Using parameters --bmax 497396907 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 497396907 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
  Building sPrime
  Building sPrimeOrder
  V-Sorting samples
  V-Sorting samples time: 00:04:06
  Allocating rank array
  Ranking v-sort output
  Ranking v-sort output time: 00:00:56
  Invoking Larsson-Sadakane on ranks
  Invoking Larsson-Sadakane on ranks time: 00:01:41
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 2.65278e+09 (target: 497396906)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
  No samples; assembling all-inclusive block
  Sorting block of length 2652783500 for bucket 1
  (Using difference cover)
  Sorting block time: 02:27:49
Returning block of 2652783501 for bucket 1
  Sorting block time: 02:08:09
Returning block of 2652783501 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 1325970242
fchr[G]: 1878618059
fchr[T]: 1878618059
fchr[$]: 2652783500
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 888465563 bytes to primary EBWT file: BS_GA.rev.1.bt2
Wrote 663195880 bytes to secondary EBWT file: BS_GA.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 2652783500
    bwtLen: 2652783501
    sz: 663195875
    bwtSz: 663195876
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 165798969
    offsSz: 663195876
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 13816581
    numLines: 13816581
    ebwtTotLen: 884261184
    ebwtTotSz: 884261184
    color: 0
    reverse: 1
Total time for backward call to driver() for mirror index: 03:00:50
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 773280124
fchr[G]: 773280124
fchr[T]: 1325970242
fchr[$]: 2652783500
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 888465563 bytes to primary EBWT file: BS_CT.rev.1.bt2
Wrote 663195880 bytes to secondary EBWT file: BS_CT.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 2652783500
    bwtLen: 2652783501
    sz: 663195875
    bwtSz: 663195876
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 165798969
    offsSz: 663195876
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 13816581
    numLines: 13816581
    ebwtTotLen: 884261184
    ebwtTotSz: 884261184
    color: 0
    reverse: 1
Total time for backward call to driver() for mirror index: 02:31:14
Command completed. Elapsed time: 5:35:57. Running peak memory: 27.447GB. PID: 15604; Command: bismark_genome_preparation; Return code: 0; Memory used: 27.447GB > `touch /project/shefflab/deploy/rg.databio.org_full/genomes/data/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index/default/_refgenie_build/0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1_bismark_bt2_index__default.flag` (13714)

Command completed. Elapsed time: 0:00:00. Running peak memory: 27.447GB.  
  PID: 13714;	Command: touch;	Return code: 0;	Memory used: 0.001GB

Asset digest: 09ae85dcba522cfac08a392dc4c45fd1
Default tag for '0f10d83b1050c08dd53189986f60970b92a315aa7a16a6f1/bismark_bt2_index' set to: default

### Pipeline completed. Epilogue
*        Elapsed time (this run):  5:36:25
*  Total elapsed time (all runs):  5:35:57
*         Peak memory (this run):  27.4468 GB
*        Pipeline completed time: 2021-04-08 02:03:05
Finished building 'bismark_bt2_index' asset
Created alias directories: 
 - /project/shefflab/deploy/rg.databio.org_full/genomes/alias/mm10/bismark_bt2_index/default