Home Developer "Too many open files" exception: how to fix the bug

“Too many open files” exception: how to fix the bug

Author

Date

Category

“Too many open files” exception is a component Codacy started displaying from time to time. This was one of the hardest bug we’ve had to face to date, getting us to the point of pulling out our hair. Today, we share how we fixed it so that you can avoid pulling out yours.

Close opened resources

The go to solution when you start getting this kind of exceptions is to look for resources like files, sockets, etc that weren’t properly closed.

In our code, we follow the golden rule of closing whatever we open, but still, there we’re some details of our Scala + PlayFramework stack, that we were unaware that caused leaking of resources.

Source.fromFile

Although the scala.io.Source.fromFile method is often used in oneliners and seductive to use in a functional style, it does not close the file opened unless you do it explicitly.

val source = scala.io.Source.fromFile("file.txt")
val lines = try { source.mkString } finally { source.close() }

play-ws

This is another sneaky one. If you’re using play-ws outside a play application, and you’re creating an instance of NingWSClient and casting it to WSClient, you’ll care about this case. Previous to version 2.4, the WSClient interface didn’t exposed a close method, so one would assume that it is managed automatically. Wrong, the file descriptor was left open. To close it you should cast the client to NingWSClient as:

wsClient.underlying[NingWSClient].close()

In newer versions than 2.4, the method is exposed in the WSClient interface.

Increasing the system limits

If you got this far, maybe you’re starting to feel let down. You’ve done everything right, but still your application is crashing. Don’t worry, it’s normal. Some applications just have to handle more files than what OS defaults are set. The rest of the guide is for specific for an Ubuntu stack.

ulimit

The system limits for open file descriptors are set in /etc/security/limits.conf. You can change those by editing the file or by running:

ulimit -Hn 65536
ulimit -Sn 65536

You can now check if the limits were applied to your play application by running:

https://gist.github.com/mrfyda/27a99c1c8a1924bf14d1

In case it didn’t maybe you’re missing this line in the file /etc/pam.d/common-session:

session required pam_limits.so

System services

If after these changes your app is still using the original limits, you may be running it as a service. Our app was being initialised with an initscript which doesn’t honor the limits set in /etc/security/limits.conf. In order to get it working we had to hardcode the limits in the init script.

At last, success! It was a very hard debugging session that spawn over a couple of days. I have to thank to my debugging buddy, the rubber duckling, who keep me sane during this journey.


Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!


References:
https://stackoverflow.com/questions/26220391/how-to-have-a-scala-standalone-application-that-uses-the-playframework-libraries
https://www.playframework.com/documentation/2.4.x/api/scala/index.html#play.api.libs.ws.WSClient
https://rtcamp.com/tutorials/linux/increase-open-files-limit/
http://ubuntuforums.org/archive/index.php/t-1274847.html
http://upstart.ubuntu.com/cookbook/

About Codacy

Codacy is used by thousands of developers to analyze billions of lines of code every day!

Getting started is easy – and free! Just use your  GitHub, Bitbucket or Google account to sign up.

GET STARTED

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Recent posts

7 drawbacks of linting tools

Linting tools (also known as linters or static analyzers) help automate the code review process. They perform basic static code analysis by flagging programming...

Using the API to add Codacy Grade details to the Readme

Some context Codacy has a badge mechanism that can be included in your Readme file. It gives you an idea of the grade of your repository, from...

March Product Update: Support for Cloud Infrastructure-as-code, Custom Reports with API endpoints & more 🚀

Here are some fresh updates from March! This month we bring you a new product offering, new features, and product updates, interesting reads, and...

Top 10 ways to perform fast code review

We always want to be fast at code review.. How frequent is it for you to be reviewing code at 3am? When code reviewing, do you...

Interview with Gary McKay, Somos’ Director of Agile Service Delivery

Somos is a proven leader in registry management and data solutions. Somos fosters meaningful connections by delivering value, innovation and confidence to consumers. We...