This is what it looks like when you do a genome assembly and run out of memory…
The machine in question actually has 384GB of RAM (not much, as far as machines which do genome assembly go!). Assembling a genome is like doing a massive puzzle; you need to have all the “pieces” in contiguous memory, and with the advances of genome sequencing technology there are always more “pieces” to put together!
The graph comes from Ganglia, a cluster monitoring tool we use on our high-performance computing cluster at ILRI, Nairobi. Ganglia won’t tell you when machines are down, slow, or whatever… but it’s great for a heads-up look at the health of your cluster (ie, why is there such a massive drop?), as well as hourly, daily, weekly trend graphs.
As the graph was quite sharp to rise and fall, I assumed whatever job the user was running probably hadn’t ended amicably. A quick look at the machine’s
dmesg verifies that assumption:
Out of memory: Kill process 6447 (velvetg) score 981 or sacrifice child Killed process 6447, UID 901, (velvetg) total-vm:430619680kB, anon-rss:390267636kB, file-rss:4kB
The tool in question is velvet, a genome assembler. While genome assembly in general is memory intensive, I hear that velvet is especially so. 😉