RAM issue when running large genome

[yl-lu]

Thanks for releasing SICER2. It's very easy to use.

When I running sicer, I got the error as follows. It seems to be a RAM issue. Any solutions, excepting for spliting chromosomes, will be helpful. Thanks in advance. And I am also wondering which step eats so much RAM.

---

Running SICER with given arguments

Preprocess the sample.sort.filter.bed file to remove redundancy with threshold of 1

---

chrom Total plus reads Retained plus reads Total minus reads Retained minus reads

chr1A 2334560 1911462 2334705 1911372
chr1B 1748245 1429378 1748368 1429319
chr1D 2116681 1730043 2117066 1730218
chr2A 2973120 2438267 2973290 2438125
chr2B 2976395 2440205 2976392 2440360
chr2D 2776080 2272251 2775903 2271996
chr3A 3071998 2520283 3071642 2519954
chr3B 3311932 2708761 3311451 2708444
chr3D 2634732 2159777 2634646 2159560
chr4A 2926229 2398594 2925700 2398413
chr4B 2795149 2243862 2797695 2244249
chr4D 2183479 1790446 2183363 1790651
chr5A 2855854 2339924 2862450 2340632
chr5B 2836048 2325507 2835977 2325796
chr5D 2474294 2024849 2474465 2025120
chr6A 2463988 2022030 2464361 2022027
chr6B 2657886 2174715 2657654 2174627
chr6D 2074008 1690806 2073279 1690430
chr7A 2902987 2384111 2903140 2383982
chr7B 3030855 2476157 3030789 2476418
chr7D 2737207 2243202 2737338 2243360
chrUn 872120 691180 871182 690569

Preprocess the sample.sort.filter.bed file to remove redundancy with threshold of 1

Removing temporary directory and all files in it.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/export/home/yllu/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/export/home/yllu/anaconda3/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/net/home/igz/yllu/software/SICER2/sicer/src/remove_redundant_reads.py", line 132, in find_and_filter_reads
return strand_broken_remove(chrom, cutoff, file_name, chrom_reads)
File "/net/home/igz/yllu/software/SICER2/sicer/src/remove_redundant_reads.py", line 92, in strand_broken_remove
np.save(name_for_save, filtered_reads)
File "<array_function internals>", line 5, in save
File "/export/home/yllu/anaconda3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 528, in save
format.write_array(fid, arr, allow_pickle=allow_pickle,
File "/export/home/yllu/anaconda3/lib/python3.8/site-packages/numpy/lib/format.py", line 675, in write_array
array.tofile(fp)
OSError: Not enough free space to write 1428442200 bytes
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/net/home/igz/yllu/software/SICER2/bin/sicer", line 331, in
main()
File "/net/home/igz/yllu/software/SICER2/bin/sicer", line 325, in main
run_SICER.main(args)
File "/net/home/igz/yllu/software/SICER2/sicer/main/run_SICER.py", line 103, in main
total_control_read_count = remove_redundant_reads.main(args, args.control_file, pool)
File "/net/home/igz/yllu/software/SICER2/sicer/src/remove_redundant_reads.py", line 145, in main
filtered_result = pool.map(find_and_filter_reads_partial, chroms)
File "/export/home/yllu/anaconda3/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/export/home/yllu/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
OSError: Not enough free space to write 1428442200 bytes