Settings: Output files: "/project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/hs38d1.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: /project/shefflab/genomes_parallel/hs38d1/fasta/default/hs38d1.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 8; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 0; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 2; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 342710 for bucket 1 (Using difference cover) Sorting block time: 00:00:01 Returning block of 342711 for bucket 1 Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 766753 for bucket 2 (Using difference cover) Sorting block time: 00:00:00 Returning block of 766754 for bucket 2 Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 787978 for bucket 3 (Using difference cover) Sorting block time: 00:00:00 Returning block of 787979 for bucket 3 Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 875857 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 875858 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 797637 for bucket 5 (Using difference cover) Sorting block time: 00:00:01 Returning block of 797638 for bucket 5 Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% bucket 6: 30% bucket 6: 40% bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 1000302 for bucket 6 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1000303 for bucket 6 Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 596532 for bucket 7 (Using difference cover) Sorting block time: 00:00:00 Returning block of 596533 for bucket 7 Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 624746 for bucket 8 (Using difference cover) Sorting block time: 00:00:00 Returning block of 624747 for bucket 8 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 2930027 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6406876 bytes to primary EBWT file: /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/hs38d1.1.bt2 Wrote 1448136 bytes to secondary EBWT file: /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/hs38d1.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 0 Total time for call to driver() for forward index: 00:00:03 Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 7; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 827502 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 7 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 1014832 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1014833 for bucket 1 Getting block 2 of 7 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 555985 for bucket 2 (Using difference cover) Sorting block time: 00:00:00 Returning block of 555986 for bucket 2 Getting block 3 of 7 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 854548 for bucket 3 (Using difference cover) Sorting block time: 00:00:01 Returning block of 854549 for bucket 3 Getting block 4 of 7 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 504658 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 504659 for bucket 4 Getting block 5 of 7 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 836947 for bucket 5 (Using difference cover) Sorting block time: 00:00:00 Returning block of 836948 for bucket 5 Getting block 6 of 7 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% bucket 6: 30% bucket 6: 40% bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 1073260 for bucket 6 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1073261 for bucket 6 Getting block 7 of 7 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 952286 for bucket 7 (Using difference cover) Sorting block time: 00:00:01 Returning block of 952287 for bucket 7 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 2930027 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6406876 bytes to primary EBWT file: /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/hs38d1.rev.1.bt2 Wrote 1448136 bytes to secondary EBWT file: /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/hs38d1.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:03Command completed. Elapsed time: 0:00:05. Running peak memory: 0.02GB. PID: 174092; Command: bowtie2-build; Return code: 0; Memory used: 0.02GB > `touch /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default/_refgenie_build/hs38d1_bowtie2_index__default.flag` (174094) Command completed. Elapsed time: 0:00:00. Running peak memory: 0.02GB. PID: 174094; Command: touch; Return code: 0; Memory used: 0.0GB > `cd /project/shefflab/genomes_parallel/hs38d1/bowtie2_index/default; find . -type f -not -path './_refgenie_build*' -exec md5sum {} \; | sort -k 2 | awk '{print $1}' | md5sum` Default tag for 'hs38d1/bowtie2_index' set to: default ### Pipeline completed. Epilogue * Elapsed time (this run): 0:00:06 * Total elapsed time (all runs): 0:00:05 * Peak memory (this run): 0.0205 GB * Pipeline completed time: 2019-10-17 08:59:45 Finished building asset 'bowtie2_index' Using 'default' as the default tag for 'hs38d1/bismark_bt1_index' Inputs required to build 'bismark_bt1_index': Building asset 'bismark_bt1_index' Output content: /project/shefflab/genomes_parallel/hs38d1; logs: /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default/_refgenie_build ### Pipeline run code and environment: * Command: `/home/mjs5kd/.local/bin/refgenie build --genome hs38d1 --asset star_index bwa_index bowtie2_index bismark_bt1_index bismark_bt2_index hisat2_index --fasta /project/shefflab/www/refgenie_raw/hs38d1.fa.gz --gencode_gtf /project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz --ensembl_gtf /project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz --gff /project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz --refgene /project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * Compute host: udc-ba34-36 * Working dir: /sfs/qumulo/qhome/mjs5kd/code/databio_genomes_parallel * Outfolder: /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default/_refgenie_build/ * Pipeline started at: (10-17 08:59:45) elapsed: 0.0 _TIME_ ### Version log: * Python version: 3.6.6 * Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper` * Pypiper version: 0.12.1 * Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin` * Pipeline version: None ### Arguments passed to pipeline: * `asset_registry_paths`: `['star_index', 'bwa_index', 'bowtie2_index', 'bismark_bt1_index', 'bismark_bt2_index', 'hisat2_index']` * `command`: `build` * `config_file`: `/home/mjs5kd/.local/lib/python3.6/site-packages/refgenie/refgenie.yaml` * `context`: `None` * `dbnsfp`: `None` * `docker`: `False` * `ensembl_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz` * `fasta`: `/project/shefflab/www/refgenie_raw/hs38d1.fa.gz` * `gencode_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz` * `genome`: `hs38d1` * `genome_config`: `None` * `genome_description`: `None` * `gff`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz` * `logdev`: `False` * `new_start`: `False` * `outfolder`: `/project/shefflab/genomes_parallel` * `recover`: `False` * `refgene`: `/project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * `requirements`: `False` * `silent`: `False` * `tag_description`: `None` * `tags`: `None` * `verbosity`: `None` * `volumes`: `None` ---------------------------------------- Target to produce: `/project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default/_refgenie_build/hs38d1_bismark_bt1_index__default.flag` > `ln -sf /project/shefflab/genomes_parallel/hs38d1/fasta/default/hs38d1.fa /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default` (174112) Command completed. Elapsed time: 0:00:00. Running peak memory: 0GB. PID: 174112; Command: ln; Return code: 0; Memory used: 0.0GB > `bismark_genome_preparation /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default` (174113)

Writing bisulfite genomes out into a single MFA (multi FastA) file Bisulfite Genome Indexer version v0.19.1 (last modified 26 April 2018) Step I - Prepare genome folders - completed Total number of conversions performed: C->T: 1274343 G->A: 1235129 Step II - Genome bisulfite conversions - completed Bismark Genome Preparation - Step III: Launching the Bowtie 2 indexer Please be aware that this process can - depending on genome size - take several hours! Settings: Output files: "BS_CT.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.CT_conversion.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 7; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 744104 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 744105 for bucket 1 Settings: Output files: "BS_GA.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.GA_conversion.fa Building a SMALL index Reading reference sizes Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences bucket 2: 70% bucket 2: 80% Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test bucket 2: 90% bucket 2: 100% Sorting block of length 911578 for bucket 2 (Using difference cover) Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:01 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Sorting block time: 00:00:01 Returning block of 911579 for bucket 2 Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 625300 for bucket 3 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 6; iterating... Sorting block time: 00:00:00 Returning block of 625301 for bucket 3 Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 929849 for bucket 4 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 0; iterating... Sorting block time: 00:00:01 Returning block of 929850 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab bucket 5: 70% Entering Ebwt loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 5: 80% bucket 1: 20% bucket 5: 90% bucket 1: 30% bucket 1: 40% bucket 5: 100% Sorting block of length 696686 for bucket 5 (Using difference cover) bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 432516 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 432517 for bucket 1 Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% Sorting block time: 00:00:00 Returning block of 696687 for bucket 5 bucket 2: 20% bucket 2: 30% bucket 2: 40% Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 2: 50% bucket 6: 10% bucket 2: 60% bucket 6: 20% bucket 2: 70% bucket 6: 30% bucket 2: 80% bucket 6: 40% bucket 2: 90% bucket 2: 100% Sorting block of length 1026532 for bucket 2 (Using difference cover) bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 409626 for bucket 6 (Using difference cover) Sorting block time: 00:00:01 Returning block of 409627 for bucket 6 Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% Sorting block time: 00:00:01 Returning block of 1026533 for bucket 2 bucket 7: 50% bucket 7: 60% Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 7: 70% bucket 3: 10% bucket 7: 80% bucket 3: 20% bucket 7: 90% bucket 3: 30% bucket 7: 100% Sorting block of length 869402 for bucket 7 (Using difference cover) bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 850511 for bucket 3 (Using difference cover) Sorting block time: 00:00:00 Returning block of 869403 for bucket 7 Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% Sorting block time: 00:00:00 Returning block of 850512 for bucket 3 bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 8: 80% bucket 4: 10% bucket 8: 90% bucket 4: 20% bucket 8: 100% Sorting block of length 605970 for bucket 8 (Using difference cover) bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 581250 for bucket 4 (Using difference cover) Sorting block time: 00:00:01 Returning block of 605971 for bucket 8 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 1655684 fchr[T]: 2890813 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_CT.1.bt2 Sorting block time: 00:00:01 Returning block of 581251 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 893494 for bucket 5 (Using difference cover) Wrote 1448136 bytes to secondary EBWT file: BS_CT.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 0 Total time for call to driver() for forward index: 00:00:05 Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Sorting block time: 00:00:00 Returning block of 893495 for bucket 5 Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% bucket 6: 30% bucket 6: 40% Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 221484 for bucket 6 (Using difference cover) V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Sorting block time: 00:00:00 Returning block of 221485 for bucket 6 Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 1023617 for bucket 7 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 6; iterating... Sorting block time: 00:00:00 Returning block of 1023618 for bucket 7 Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... bucket 8: 100% Sorting block of length 763111 for bucket 8 (Using difference cover) Sorting block time: 00:00:00 Returning block of 763112 for bucket 8 Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Exited Ebwt loop fchr[A]: 0 fchr[C]: 2890813 fchr[G]: 4165156 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_GA.1.bt2 Wrote 1448136 bytes to secondary EBWT file: BS_GA.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 0 Total time for call to driver() for forward index: 00:00:06 Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 827502 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 7 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 1039693 for bucket 1 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 6; iterating... Sorting block time: 00:00:01 Returning block of 1039694 for bucket 1 Getting block 2 of 7 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 882586 for bucket 2 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 0; iterating... Sorting block time: 00:00:00 Returning block of 882587 for bucket 2 Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 643613 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 9 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: Getting block 3 of 7 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 1: 10% bucket 3: 10% bucket 1: 20% bucket 3: 20% bucket 1: 30% bucket 1: 40% bucket 3: 30% bucket 1: 50% bucket 3: 40% bucket 1: 60% bucket 3: 50% bucket 1: 70% bucket 3: 60% bucket 1: 80% bucket 1: 90% bucket 3: 70% bucket 1: 100% Sorting block of length 81919 for bucket 1 (Using difference cover) bucket 3: 80% Sorting block time: 00:00:00 Returning block of 81920 for bucket 1 bucket 3: 90% Getting block 2 of 9 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 3: 100% Sorting block of length 467125 for bucket 3 (Using difference cover) bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% Sorting block time: 00:00:01 Returning block of 467126 for bucket 3 bucket 2: 100% Sorting block of length 1015263 for bucket 2 (Using difference cover) Getting block 4 of 7 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 1029608 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1015264 for bucket 2 Getting block 3 of 9 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% Sorting block time: 00:00:00 Returning block of 1029609 for bucket 4 bucket 3: 100% Sorting block of length 129125 for bucket 3 (Using difference cover) Sorting block time: 00:00:00 Returning block of 129126 for bucket 3 Getting block 4 of 9 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: Getting block 5 of 7 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 4: 10% bucket 5: 10% bucket 4: 20% bucket 5: 20% bucket 4: 30% bucket 5: 30% bucket 4: 40% bucket 5: 40% bucket 4: 50% bucket 5: 50% bucket 4: 60% bucket 5: 60% bucket 4: 70% bucket 4: 80% bucket 5: 70% bucket 4: 90% bucket 5: 80% bucket 4: 100% Sorting block of length 964506 for bucket 4 (Using difference cover) bucket 5: 90% bucket 5: 100% Sorting block of length 869072 for bucket 5 (Using difference cover) Sorting block time: 00:00:01 Returning block of 964507 for bucket 4 Sorting block time: 00:00:00 Returning block of 869073 for bucket 5 Getting block 5 of 9 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: Getting block 6 of 7 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 5: 10% bucket 6: 10% bucket 5: 20% bucket 6: 20% bucket 5: 30% bucket 6: 30% bucket 5: 40% bucket 6: 40% bucket 5: 50% bucket 6: 50% bucket 5: 60% bucket 6: 60% bucket 5: 70% bucket 5: 80% bucket 6: 70% bucket 5: 90% bucket 6: 80% bucket 5: 100% Sorting block of length 936058 for bucket 5 (Using difference cover) bucket 6: 90% bucket 6: 100% Sorting block of length 1031011 for bucket 6 (Using difference cover) Sorting block time: 00:00:00 Returning block of 936059 for bucket 5 Getting block 6 of 9 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% Sorting block time: 00:00:01 Returning block of 1031012 for bucket 6 bucket 6: 30% bucket 6: 40% Getting block 7 of 7 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 6: 50% bucket 7: 10% bucket 7: 20% bucket 6: 60% bucket 7: 30% bucket 6: 70% bucket 7: 40% bucket 7: 50% bucket 6: 80% bucket 7: 60% bucket 6: 90% bucket 7: 70% bucket 6: 100% Sorting block of length 1038279 for bucket 6 (Using difference cover) bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 473421 for bucket 7 (Using difference cover) Sorting block time: 00:00:00 Returning block of 473422 for bucket 7 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 1655684 fchr[T]: 2890813 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_CT.rev.1.bt2 Sorting block time: 00:00:00 Returning block of 1038280 for bucket 6 Wrote 1448136 bytes to secondary EBWT file: BS_CT.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:06 Getting block 7 of 9 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 368137 for bucket 7 (Using difference cover) Sorting block time: 00:00:00 Returning block of 368138 for bucket 7 Getting block 8 of 9 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 855159 for bucket 8 (Using difference cover) Sorting block time: 00:00:01 Returning block of 855160 for bucket 8 Getting block 9 of 9 Reserving size (1086098) for bucket 9 Calculating Z arrays for bucket 9 Entering block accumulator loop for bucket 9: bucket 9: 10% bucket 9: 20% bucket 9: 30% bucket 9: 40% bucket 9: 50% bucket 9: 60% bucket 9: 70% bucket 9: 80% bucket 9: 90% bucket 9: 100% Sorting block of length 404068 for bucket 9 (Using difference cover) Sorting block time: 00:00:00 Returning block of 404069 for bucket 9 Exited Ebwt loop fchr[A]: 0 fchr[C]: 2890813 fchr[G]: 4165156 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_GA.rev.1.bt2 Wrote 1448136 bytes to secondary EBWT file: BS_GA.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:05Command completed. Elapsed time: 0:00:17. Running peak memory: 0.05GB. PID: 174113; Command: bismark_genome_preparation; Return code: 0; Memory used: 0.05GB > `touch /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default/_refgenie_build/hs38d1_bismark_bt1_index__default.flag` (174171) Command completed. Elapsed time: 0:00:00. Running peak memory: 0.05GB. PID: 174171; Command: touch; Return code: 0; Memory used: 0.0GB > `cd /project/shefflab/genomes_parallel/hs38d1/bismark_bt1_index/default; find . -type f -not -path './_refgenie_build*' -exec md5sum {} \; | sort -k 2 | awk '{print $1}' | md5sum` Default tag for 'hs38d1/bismark_bt1_index' set to: default ### Pipeline completed. Epilogue * Elapsed time (this run): 0:00:17 * Total elapsed time (all runs): 0:00:17 * Peak memory (this run): 0.0498 GB * Pipeline completed time: 2019-10-17 09:00:02 Finished building asset 'bismark_bt1_index' Using 'default' as the default tag for 'hs38d1/bismark_bt2_index' Inputs required to build 'bismark_bt2_index': Building asset 'bismark_bt2_index' Output content: /project/shefflab/genomes_parallel/hs38d1; logs: /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default/_refgenie_build ### Pipeline run code and environment: * Command: `/home/mjs5kd/.local/bin/refgenie build --genome hs38d1 --asset star_index bwa_index bowtie2_index bismark_bt1_index bismark_bt2_index hisat2_index --fasta /project/shefflab/www/refgenie_raw/hs38d1.fa.gz --gencode_gtf /project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz --ensembl_gtf /project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz --gff /project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz --refgene /project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * Compute host: udc-ba34-36 * Working dir: /sfs/qumulo/qhome/mjs5kd/code/databio_genomes_parallel * Outfolder: /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default/_refgenie_build/ * Pipeline started at: (10-17 09:00:03) elapsed: 0.0 _TIME_ ### Version log: * Python version: 3.6.6 * Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper` * Pypiper version: 0.12.1 * Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin` * Pipeline version: None ### Arguments passed to pipeline: * `asset_registry_paths`: `['star_index', 'bwa_index', 'bowtie2_index', 'bismark_bt1_index', 'bismark_bt2_index', 'hisat2_index']` * `command`: `build` * `config_file`: `/home/mjs5kd/.local/lib/python3.6/site-packages/refgenie/refgenie.yaml` * `context`: `None` * `dbnsfp`: `None` * `docker`: `False` * `ensembl_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz` * `fasta`: `/project/shefflab/www/refgenie_raw/hs38d1.fa.gz` * `gencode_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz` * `genome`: `hs38d1` * `genome_config`: `None` * `genome_description`: `None` * `gff`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz` * `logdev`: `False` * `new_start`: `False` * `outfolder`: `/project/shefflab/genomes_parallel` * `recover`: `False` * `refgene`: `/project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * `requirements`: `False` * `silent`: `False` * `tag_description`: `None` * `tags`: `None` * `verbosity`: `None` * `volumes`: `None` ---------------------------------------- Target to produce: `/project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default/_refgenie_build/hs38d1_bismark_bt2_index__default.flag` > `ln -sf /project/shefflab/genomes_parallel/hs38d1/fasta/default/hs38d1.fa /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default` (174258) Command completed. Elapsed time: 0:00:00. Running peak memory: 0GB. PID: 174258; Command: ln; Return code: 0; Memory used: 0.0GB > `bismark_genome_preparation --bowtie2 /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default` (174260)

Writing bisulfite genomes out into a single MFA (multi FastA) file Bisulfite Genome Indexer version v0.19.1 (last modified 26 April 2018) Step I - Prepare genome folders - completed Total number of conversions performed: C->T: 1274343 G->A: 1235129 Step II - Genome bisulfite conversions - completed Bismark Genome Preparation - Step III: Launching the Bowtie 2 indexer Please be aware that this process can - depending on genome size - take several hours! Settings: Output files: "BS_CT.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.CT_conversion.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:01 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 7; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 744104 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 744105 for bucket 1 Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 911578 for bucket 2 (Using difference cover) Settings: Output files: "BS_GA.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: genome_mfa.GA_conversion.fa Building a SMALL index Reading reference sizes Sorting block time: 00:00:01 Returning block of 911579 for bucket 2 Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 bucket 3: 70% Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 625300 for bucket 3 (Using difference cover) V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Sorting block time: 00:00:00 Returning block of 625301 for bucket 3 Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 929849 for bucket 4 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 6; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Sorting block time: 00:00:01 Returning block of 929850 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 696686 for bucket 5 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 0; iterating... Sorting block time: 00:00:00 Returning block of 696687 for bucket 5 Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab bucket 6: 10% Entering Ebwt loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 6: 20% bucket 1: 20% bucket 1: 30% bucket 6: 30% bucket 1: 40% bucket 6: 40% bucket 1: 50% bucket 1: 60% bucket 6: 50% bucket 1: 70% bucket 6: 60% bucket 1: 80% bucket 1: 90% bucket 6: 70% bucket 1: 100% Sorting block of length 432516 for bucket 1 (Using difference cover) bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 409626 for bucket 6 (Using difference cover) Sorting block time: 00:00:01 Returning block of 432517 for bucket 1 Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% Sorting block time: 00:00:00 Returning block of 409627 for bucket 6 bucket 2: 30% Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 2: 40% bucket 7: 10% bucket 2: 50% bucket 2: 60% bucket 7: 20% bucket 2: 70% bucket 2: 80% bucket 7: 30% bucket 2: 90% bucket 7: 40% bucket 2: 100% Sorting block of length 1026532 for bucket 2 (Using difference cover) bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 869402 for bucket 7 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1026533 for bucket 2 Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% Sorting block time: 00:00:00 Returning block of 869403 for bucket 7 bucket 3: 30% bucket 3: 40% bucket 3: 50% Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 3: 60% bucket 8: 10% bucket 8: 20% bucket 3: 70% bucket 8: 30% bucket 3: 80% bucket 8: 40% bucket 3: 90% bucket 8: 50% bucket 3: 100% Sorting block of length 850511 for bucket 3 (Using difference cover) bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 605970 for bucket 8 (Using difference cover) Sorting block time: 00:00:01 Returning block of 850512 for bucket 3 Sorting block time: 00:00:00 Returning block of 605971 for bucket 8 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 1655684 fchr[T]: 2890813 fchr[$]: 5792522 Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_CT.1.bt2 bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 581250 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 581251 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: Wrote 1448136 bytes to secondary EBWT file: BS_CT.2.bt2 bucket 5: 10% Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 0 bucket 5: 20% Total time for call to driver() for forward index: 00:00:05 Reading reference sizes bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences bucket 5: 70% bucket 5: 80% Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test bucket 5: 90% bucket 5: 100% Sorting block of length 893494 for bucket 5 (Using difference cover) Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:01 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Sorting block time: 00:00:01 Returning block of 893495 for bucket 5 Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% bucket 6: 30% bucket 6: 40% bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 221484 for bucket 6 (Using difference cover) Sorting block time: 00:00:00 Returning block of 221485 for bucket 6 Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 6; iterating... bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 1023617 for bucket 7 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Sorting block time: 00:00:00 Returning block of 1023618 for bucket 7 Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 763111 for bucket 8 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Sorting block time: 00:00:00 Returning block of 763112 for bucket 8 Exited Ebwt loop fchr[A]: 0 fchr[C]: 2890813 fchr[G]: 4165156 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_GA.1.bt2 Wrote 1448136 bytes to secondary EBWT file: BS_GA.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 0 Total time for call to driver() for forward index: 00:00:05 Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 1448130 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 827502 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 7 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 bucket 1: 30% Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 1039693 for bucket 1 (Using difference cover) V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:01 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Sorting block time: 00:00:01 Returning block of 1039694 for bucket 1 Getting block 2 of 7 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 6; iterating... bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 882586 for bucket 2 (Using difference cover) Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 0; iterating... Sorting block time: 00:00:00 Returning block of 882587 for bucket 2 Getting block 3 of 7 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 643613 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 9 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 3: 90% bucket 1: 10% bucket 3: 100% Sorting block of length 467125 for bucket 3 (Using difference cover) bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 81919 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 81920 for bucket 1 Getting block 2 of 9 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% Sorting block time: 00:00:00 Returning block of 467126 for bucket 3 bucket 2: 20% Getting block 4 of 7 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 2: 30% bucket 2: 40% bucket 4: 10% bucket 2: 50% bucket 4: 20% bucket 2: 60% bucket 4: 30% bucket 2: 70% bucket 4: 40% bucket 2: 80% bucket 4: 50% bucket 2: 90% bucket 2: 100% Sorting block of length 1015263 for bucket 2 (Using difference cover) bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 1029608 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 1015264 for bucket 2 Getting block 3 of 9 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% Sorting block time: 00:00:00 Returning block of 1029609 for bucket 4 bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% Getting block 5 of 7 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 3: 70% bucket 3: 80% bucket 5: 10% bucket 3: 90% bucket 5: 20% bucket 3: 100% Sorting block of length 129125 for bucket 3 (Using difference cover) bucket 5: 30% bucket 5: 40% Sorting block time: 00:00:00 Returning block of 129126 for bucket 3 Getting block 4 of 9 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 5: 50% bucket 4: 10% bucket 5: 60% bucket 4: 20% bucket 4: 30% bucket 5: 70% bucket 4: 40% bucket 5: 80% bucket 4: 50% bucket 5: 90% bucket 4: 60% bucket 4: 70% bucket 5: 100% Sorting block of length 869072 for bucket 5 (Using difference cover) bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 964506 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 869073 for bucket 5 Getting block 6 of 7 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% Sorting block time: 00:00:00 Returning block of 964507 for bucket 4 bucket 6: 20% bucket 6: 30% Getting block 5 of 9 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 6: 40% bucket 5: 10% bucket 6: 50% bucket 5: 20% bucket 6: 60% bucket 5: 30% bucket 6: 70% bucket 5: 40% bucket 6: 80% bucket 5: 50% bucket 6: 90% bucket 5: 60% bucket 6: 100% Sorting block of length 1031011 for bucket 6 (Using difference cover) bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 936058 for bucket 5 (Using difference cover) Sorting block time: 00:00:01 Returning block of 936059 for bucket 5 Sorting block time: 00:00:01 Returning block of 1031012 for bucket 6 Getting block 6 of 9 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: Getting block 7 of 7 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 6: 10% bucket 7: 10% bucket 6: 20% bucket 7: 20% bucket 6: 30% bucket 7: 30% bucket 7: 40% bucket 6: 40% bucket 7: 50% bucket 7: 60% bucket 6: 50% bucket 7: 70% bucket 6: 60% bucket 7: 80% bucket 6: 70% bucket 7: 90% bucket 6: 80% bucket 7: 100% Sorting block of length 473421 for bucket 7 (Using difference cover) bucket 6: 90% bucket 6: 100% Sorting block of length 1038279 for bucket 6 (Using difference cover) Sorting block time: 00:00:00 Returning block of 473422 for bucket 7 Exited Ebwt loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 1655684 fchr[T]: 2890813 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_CT.rev.1.bt2 Wrote 1448136 bytes to secondary EBWT file: BS_CT.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:06 Sorting block time: 00:00:00 Returning block of 1038280 for bucket 6 Getting block 7 of 9 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 368137 for bucket 7 (Using difference cover) Sorting block time: 00:00:01 Returning block of 368138 for bucket 7 Getting block 8 of 9 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 855159 for bucket 8 (Using difference cover) Sorting block time: 00:00:00 Returning block of 855160 for bucket 8 Getting block 9 of 9 Reserving size (1086098) for bucket 9 Calculating Z arrays for bucket 9 Entering block accumulator loop for bucket 9: bucket 9: 10% bucket 9: 20% bucket 9: 30% bucket 9: 40% bucket 9: 50% bucket 9: 60% bucket 9: 70% bucket 9: 80% bucket 9: 90% bucket 9: 100% Sorting block of length 404068 for bucket 9 (Using difference cover) Sorting block time: 00:00:00 Returning block of 404069 for bucket 9 Exited Ebwt loop fchr[A]: 0 fchr[C]: 2890813 fchr[G]: 4165156 fchr[T]: 4165156 fchr[$]: 5792522 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 6228721 bytes to primary EBWT file: BS_GA.rev.1.bt2 Wrote 1448136 bytes to secondary EBWT file: BS_GA.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 5792522 bwtLen: 5792523 sz: 1448131 bwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 30170 numLines: 30170 ebwtTotLen: 1930880 ebwtTotSz: 1930880 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:05Command completed. Elapsed time: 0:00:16. Running peak memory: 0.05GB. PID: 174260; Command: bismark_genome_preparation; Return code: 0; Memory used: 0.05GB > `touch /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default/_refgenie_build/hs38d1_bismark_bt2_index__default.flag` (174275) Command completed. Elapsed time: 0:00:00. Running peak memory: 0.05GB. PID: 174275; Command: touch; Return code: 0; Memory used: 0.0GB > `cd /project/shefflab/genomes_parallel/hs38d1/bismark_bt2_index/default; find . -type f -not -path './_refgenie_build*' -exec md5sum {} \; | sort -k 2 | awk '{print $1}' | md5sum` Default tag for 'hs38d1/bismark_bt2_index' set to: default ### Pipeline completed. Epilogue * Elapsed time (this run): 0:00:17 * Total elapsed time (all runs): 0:00:16 * Peak memory (this run): 0.0499 GB * Pipeline completed time: 2019-10-17 09:00:20 Finished building asset 'bismark_bt2_index' Using 'default' as the default tag for 'hs38d1/hisat2_index' Inputs required to build 'hisat2_index': Building asset 'hisat2_index' Output content: /project/shefflab/genomes_parallel/hs38d1; logs: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/_refgenie_build ### Pipeline run code and environment: * Command: `/home/mjs5kd/.local/bin/refgenie build --genome hs38d1 --asset star_index bwa_index bowtie2_index bismark_bt1_index bismark_bt2_index hisat2_index --fasta /project/shefflab/www/refgenie_raw/hs38d1.fa.gz --gencode_gtf /project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz --ensembl_gtf /project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz --gff /project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz --refgene /project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * Compute host: udc-ba34-36 * Working dir: /sfs/qumulo/qhome/mjs5kd/code/databio_genomes_parallel * Outfolder: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/_refgenie_build/ * Pipeline started at: (10-17 09:00:20) elapsed: 0.0 _TIME_ ### Version log: * Python version: 3.6.6 * Pypiper dir: `/sfs/qumulo/qhome/mjs5kd/.local/lib/python3.6/site-packages/pypiper` * Pypiper version: 0.12.1 * Pipeline dir: `/sfs/qumulo/qhome/mjs5kd/.local/bin` * Pipeline version: None ### Arguments passed to pipeline: * `asset_registry_paths`: `['star_index', 'bwa_index', 'bowtie2_index', 'bismark_bt1_index', 'bismark_bt2_index', 'hisat2_index']` * `command`: `build` * `config_file`: `/home/mjs5kd/.local/lib/python3.6/site-packages/refgenie/refgenie.yaml` * `context`: `None` * `dbnsfp`: `None` * `docker`: `False` * `ensembl_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl.gtf.gz` * `fasta`: `/project/shefflab/www/refgenie_raw/hs38d1.fa.gz` * `gencode_gtf`: `/project/shefflab/www/refgenie_raw/hs38d1_gencode.gtf.gz` * `genome`: `hs38d1` * `genome_config`: `None` * `genome_description`: `None` * `gff`: `/project/shefflab/www/refgenie_raw/hs38d1_ensembl_rb.gff.gz` * `logdev`: `False` * `new_start`: `False` * `outfolder`: `/project/shefflab/genomes_parallel` * `recover`: `False` * `refgene`: `/project/shefflab/www/refgenie_raw/hs38d1_refGene.txt.gz` * `requirements`: `False` * `silent`: `False` * `tag_description`: `None` * `tags`: `None` * `verbosity`: `None` * `volumes`: `None` ---------------------------------------- Target to produce: `/project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/_refgenie_build/hs38d1_hisat2_index__default.flag` > `hisat2-build /project/shefflab/genomes_parallel/hs38d1/fasta/default/hs38d1.fa /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1` (174302)

Settings: Output files: "/project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1.*.ht2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Local offset rate: 3 (one in 8) Local fTable chars: 6 Local sequence length: 57344 Local sequence overlap between two consecutive indexes: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: /project/shefflab/genomes_parallel/hs38d1/fasta/default/hs38d1.fa Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 Time to read SNPs and splice sites: 00:00:00 Using parameters --bmax 1086098 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1086098 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 12 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 8; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 2, merged 0; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 2; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 1; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 724064 (target: 1086097) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering GFM loop Getting block 1 of 8 Reserving size (1086098) for bucket 1 Calculating Z arrays for bucket 1 Entering block accumulator loop for bucket 1: bucket 1: 10% bucket 1: 20% bucket 1: 30% bucket 1: 40% bucket 1: 50% bucket 1: 60% bucket 1: 70% bucket 1: 80% bucket 1: 90% bucket 1: 100% Sorting block of length 342710 for bucket 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 342711 for bucket 1 Getting block 2 of 8 Reserving size (1086098) for bucket 2 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 2: bucket 2: 10% bucket 2: 20% bucket 2: 30% bucket 2: 40% bucket 2: 50% bucket 2: 60% bucket 2: 70% bucket 2: 80% bucket 2: 90% bucket 2: 100% Sorting block of length 766753 for bucket 2 (Using difference cover) Sorting block time: 00:00:00 Returning block of 766754 for bucket 2 Getting block 3 of 8 Reserving size (1086098) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70% bucket 3: 80% bucket 3: 90% bucket 3: 100% Sorting block of length 787978 for bucket 3 (Using difference cover) Sorting block time: 00:00:00 Returning block of 787979 for bucket 3 Getting block 4 of 8 Reserving size (1086098) for bucket 4 Calculating Z arrays for bucket 4 Entering block accumulator loop for bucket 4: bucket 4: 10% bucket 4: 20% bucket 4: 30% bucket 4: 40% bucket 4: 50% bucket 4: 60% bucket 4: 70% bucket 4: 80% bucket 4: 90% bucket 4: 100% Sorting block of length 875857 for bucket 4 (Using difference cover) Sorting block time: 00:00:00 Returning block of 875858 for bucket 4 Getting block 5 of 8 Reserving size (1086098) for bucket 5 Calculating Z arrays for bucket 5 Entering block accumulator loop for bucket 5: bucket 5: 10% bucket 5: 20% bucket 5: 30% bucket 5: 40% bucket 5: 50% bucket 5: 60% bucket 5: 70% bucket 5: 80% bucket 5: 90% bucket 5: 100% Sorting block of length 797637 for bucket 5 (Using difference cover) Sorting block time: 00:00:00 Returning block of 797638 for bucket 5 Getting block 6 of 8 Reserving size (1086098) for bucket 6 Calculating Z arrays for bucket 6 Entering block accumulator loop for bucket 6: bucket 6: 10% bucket 6: 20% bucket 6: 30% bucket 6: 40% bucket 6: 50% bucket 6: 60% bucket 6: 70% bucket 6: 80% bucket 6: 90% bucket 6: 100% Sorting block of length 1000302 for bucket 6 (Using difference cover) Sorting block time: 00:00:01 Returning block of 1000303 for bucket 6 Getting block 7 of 8 Reserving size (1086098) for bucket 7 Calculating Z arrays for bucket 7 Entering block accumulator loop for bucket 7: bucket 7: 10% bucket 7: 20% bucket 7: 30% bucket 7: 40% bucket 7: 50% bucket 7: 60% bucket 7: 70% bucket 7: 80% bucket 7: 90% bucket 7: 100% Sorting block of length 596532 for bucket 7 (Using difference cover) Sorting block time: 00:00:00 Returning block of 596533 for bucket 7 Getting block 8 of 8 Reserving size (1086098) for bucket 8 Calculating Z arrays for bucket 8 Entering block accumulator loop for bucket 8: bucket 8: 10% bucket 8: 20% bucket 8: 30% bucket 8: 40% bucket 8: 50% bucket 8: 60% bucket 8: 70% bucket 8: 80% bucket 8: 90% bucket 8: 100% Sorting block of length 624746 for bucket 8 (Using difference cover) Sorting block time: 00:00:00 Returning block of 624747 for bucket 8 Exited GFM loop fchr[A]: 0 fchr[C]: 1655684 fchr[G]: 2930027 fchr[T]: 4165156 fchr[$]: 5792522 Exiting GFM::buildToDisk() Returning from initFromVector Wrote 6406896 bytes to primary GFM file: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1.1.ht2 Wrote 1448136 bytes to secondary GFM file: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1.2.ht2 Re-opening _in1 and _in2 as input streams Returning from GFM constructor Returning from initFromVector Wrote 21448813 bytes to primary GFM file: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1.5.ht2 Wrote 1451082 bytes to secondary GFM file: /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/hs38d1.6.ht2 Re-opening _in5 and _in5 as input streams Returning from HierEbwt constructor Headers: len: 5792522 gbwtLen: 5792523 nodes: 5792523 sz: 1448131 gbwtSz: 1448131 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 0 eftabSz: 0 ftabLen: 1048577 ftabSz: 4194308 offsLen: 362033 offsSz: 1448132 lineSz: 64 sideSz: 64 sideGbwtSz: 48 sideGbwtLen: 192 numSides: 30170 numLines: 30170 gbwtTotLen: 1930880 gbwtTotSz: 1930880 reverse: 0 linearFM: Yes Total time for call to driver() for forward index: 00:00:04Command completed. Elapsed time: 0:00:05. Running peak memory: 0.02GB. PID: 174302; Command: hisat2-build; Return code: 0; Memory used: 0.02GB > `touch /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default/_refgenie_build/hs38d1_hisat2_index__default.flag` (174306) Command completed. Elapsed time: 0:00:00. Running peak memory: 0.02GB. PID: 174306; Command: touch; Return code: 0; Memory used: 0.0GB > `cd /project/shefflab/genomes_parallel/hs38d1/hisat2_index/default; find . -type f -not -path './_refgenie_build*' -exec md5sum {} \; | sort -k 2 | awk '{print $1}' | md5sum` Default tag for 'hs38d1/hisat2_index' set to: default ### Pipeline completed. Epilogue * Elapsed time (this run): 0:00:05 * Total elapsed time (all runs): 0:00:05 * Peak memory (this run): 0.0198 GB * Pipeline completed time: 2019-10-17 09:00:25 Finished building asset 'hisat2_index'