Linux applications

Unfortunately, a lot of application developers like overcommit, presumably for two reasons:

  1. It allows you to allocate a ridiculous amounts of memory as long as you know you’ll only make sparse use of it. In our credit card analogy, this is like a contractor going to a building supply store and buying twice the amount of materials they expect to need using a credit card, knowing they’ll be safe as long as they go back and return the unused materials for a refund before the credit card bill is due.
  2. It gives you an excuse to be lazy handling errors. You can rationalize ignoring the return value of malloc on the basis that, due to overcommit, even if you check the return value you can’t be sure to avoid crashing later when the kernel doesn’t have enough physical memory to instantiate your virtual memory

This comment/assertion (if true) from http://www.etalabs.net/overcommit.html is rather alarming in terms of using/building/deploying linux for mission-critical systems.. The article further says

Overcommit is harmful because it encourages, and provides a wrong but plausible argument for, writing bad software. While the number of applications that completely ignore the failure of malloc seems to be shrinking, plenty of applications and even libraries intended for use in serious software utilize “xmalloc” wrappers that abort (!!) the caller when malloc returns a null pointer, and the justification is almost always that, since the program could OOM-crash anyway if allocation fails, it’s no worse to abort. And of course this line of reasoning completely neglects systems that were intentionally configured to be robust under memory exhaustion.

 

OK that’s it. no more blindly just deploying or installing new applications on a production linux box.. Every single new application that needs to be installed has to go through a process designed to ensure clean memory allocation. that means load testing in a staging environment (with similar memory hardware) and test cases designed to run out of memory…

I think i now understand the wisdom of DevOps movement better…

This seems to be a good starter for memory management http://jamesgolick.com/2013/5/15/memory-allocators-101.html

And from the same blog http://jamesgolick.com/2013/5/19/how-tcmalloc-works.html. Hmm.. TCMalloc sounds interesting.. also http://goog-perftools.sourceforge.net/doc/tcmalloc.html.. gotta check this out sometime…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s