Home Developer "Too many open files" exception: how to fix the bug

“Too many open files” exception: how to fix the bug

Author

Date

Category

“Too many open files” exception is a component Codacy started displaying from time to time. This was one of the hardest bug we’ve had to face to date, getting us to the point of pulling out our hair. Today, we share how we fixed it so that you can avoid pulling out yours.

Close opened resources

The go to solution when you start getting this kind of exceptions is to look for resources like files, sockets, etc that weren’t properly closed.

In our code, we follow the golden rule of closing whatever we open, but still, there we’re some details of our Scala + PlayFramework stack, that we were unaware that caused leaking of resources.

Source.fromFile

Although the scala.io.Source.fromFile method is often used in oneliners and seductive to use in a functional style, it does not close the file opened unless you do it explicitly.

val source = scala.io.Source.fromFile("file.txt")
val lines = try { source.mkString } finally { source.close() }

play-ws

This is another sneaky one. If you’re using play-ws outside a play application, and you’re creating an instance of NingWSClient and casting it to WSClient, you’ll care about this case. Previous to version 2.4, the WSClient interface didn’t exposed a close method, so one would assume that it is managed automatically. Wrong, the file descriptor was left open. To close it you should cast the client to NingWSClient as:

wsClient.underlying[NingWSClient].close()

In newer versions than 2.4, the method is exposed in the WSClient interface.

Increasing the system limits

If you got this far, maybe you’re starting to feel let down. You’ve done everything right, but still your application is crashing. Don’t worry, it’s normal. Some applications just have to handle more files than what OS defaults are set. The rest of the guide is for specific for an Ubuntu stack.

ulimit

The system limits for open file descriptors are set in /etc/security/limits.conf. You can change those by editing the file or by running:

ulimit -Hn 65536
ulimit -Sn 65536

You can now check if the limits were applied to your play application by running:

https://gist.github.com/mrfyda/27a99c1c8a1924bf14d1

In case it didn’t maybe you’re missing this line in the file /etc/pam.d/common-session:

session required pam_limits.so

System services

If after these changes your app is still using the original limits, you may be running it as a service. Our app was being initialised with an initscript which doesn’t honor the limits set in /etc/security/limits.conf. In order to get it working we had to hardcode the limits in the init script.

At last, success! It was a very hard debugging session that spawn over a couple of days. I have to thank to my debugging buddy, the rubber duckling, who keep me sane during this journey.


Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!


References:
https://stackoverflow.com/questions/26220391/how-to-have-a-scala-standalone-application-that-uses-the-playframework-libraries
https://www.playframework.com/documentation/2.4.x/api/scala/index.html#play.api.libs.ws.WSClient
https://rtcamp.com/tutorials/linux/increase-open-files-limit/
http://ubuntuforums.org/archive/index.php/t-1274847.html
http://upstart.ubuntu.com/cookbook/

About Codacy

Codacy is used by thousands of developers to analyze billions of lines of code every day!

Getting started is easy – and free! Just use your  GitHub, Bitbucket or Google account to sign up.

GET STARTED

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Recent posts

Free Codacy Pro account to fight COVID-19

Our hearts go out to everyone who has been directly or indirectly impacted by the global coronavirus (COVID-19) pandemic. We are committed...

Introducing GitHub Apps for improved user access control

We are very excited to announce our recent migration to GitHub Apps to improve the experience of GitHub Cloud users. Now, you...

How Agile & Container Technology led to the rise of enterprise DevSecOps

New development processes and open-source technologies have shifted the technology security landscape for enterprises. Previously a separate security department often dealt with...

Spring tech events with Codacy: Coverage & what's next

Although many Spring tech events are being canceled or turning virtual due to coronavirus we are fortunate to have already participated in...

Top 6 items for your code review checklist

At Codacy we set high standards, and care about the quality of the code we produce. In order to provide optimal experiences...