Home Developer "Too many open files" exception: how to fix the bug

"Too many open files" exception: how to fix the bug

Author

Date

Category

“Too many open files” exception is a component Codacy started displaying from time to time. This was one of the hardest bug we’ve had to face to date, getting us to the point of pulling out our hair. Today, we share how we fixed it so that you can avoid pulling out yours.

Close opened resources

The go to solution when you start getting this kind of exceptions is to look for resources like files, sockets, etc that weren’t properly closed.

In our code, we follow the golden rule of closing whatever we open, but still, there we’re some details of our Scala + PlayFramework stack, that we were unaware that caused leaking of resources.

Source.fromFile

Although the scala.io.Source.fromFile method is often used in oneliners and seductive to use in a functional style, it does not close the file opened unless you do it explicitly.

val source = scala.io.Source.fromFile("file.txt")
val lines = try { source.mkString } finally { source.close() }

play-ws

This is another sneaky one. If you’re using play-ws outside a play application, and you’re creating an instance of NingWSClient and casting it to WSClient, you’ll care about this case. Previous to version 2.4, the WSClient interface didn’t exposed a close method, so one would assume that it is managed automatically. Wrong, the file descriptor was left open. To close it you should cast the client to NingWSClient as:

wsClient.underlying[NingWSClient].close()

In newer versions than 2.4, the method is exposed in the WSClient interface.

Increasing the system limits

If you got this far, maybe you’re starting to feel let down. You’ve done everything right, but still your application is crashing. Don’t worry, it’s normal. Some applications just have to handle more files than what OS defaults are set. The rest of the guide is for specific for an Ubuntu stack.

ulimit

The system limits for open file descriptors are set in /etc/security/limits.conf. You can change those by editing the file or by running:

ulimit -Hn 65536
ulimit -Sn 65536

You can now check if the limits were applied to your play application by running:

https://gist.github.com/mrfyda/27a99c1c8a1924bf14d1

In case it didn’t maybe you’re missing this line in the file /etc/pam.d/common-session:

session required pam_limits.so

System services

If after these changes your app is still using the original limits, you may be running it as a service. Our app was being initialised with an initscript which doesn’t honor the limits set in /etc/security/limits.conf. In order to get it working we had to hardcode the limits in the init script.

At last, success! It was a very hard debugging session that spawn over a couple of days. I have to thank to my debugging buddy, the rubber duckling, who keep me sane during this journey.


Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!


References:
https://stackoverflow.com/questions/26220391/how-to-have-a-scala-standalone-application-that-uses-the-playframework-libraries
https://www.playframework.com/documentation/2.4.x/api/scala/index.html#play.api.libs.ws.WSClient
https://rtcamp.com/tutorials/linux/increase-open-files-limit/
http://ubuntuforums.org/archive/index.php/t-1274847.html
http://upstart.ubuntu.com/cookbook/

About Codacy

Codacy is used by thousands of developers to analyze billions of lines of code every day!

Getting started is easy – and free! Just use your  GitHub, Bitbucket or Google account to sign up.

GET STARTED

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Recent posts

New Public Product Roadmap – See What's In Store For Q1

We’re excited to release a public product roadmap for 2020.  Now, those outside Codacy can see what our team is working on...

4 Trends To Boost Software Quality in 2020

Today’s abundance of technology has created an increased demand for higher software quality. With companies aware of this, spending on IT worldwide...

7 Reasons Dev Teams Choose Codacy Over Other Automated Code Review Tools

Today more than 100,000 developers at leading global organizations like PayPal and Adobe across industries use Codacy (read customer case studies). ...

Introducing GitHub Sync For Seamless User Management

Announcing GitHub Sync: Synced Organizations For GitHub We are thrilled to announce "GitHub Sync" -- automatic user and repository...

Ringing In 2020 With Codacy

It’s been an exciting 2019 at Codacy and we expect more excitement in 2020. Over the last 12 months we’ve built out...