“Too many open files” exception is a component Codacy started displaying from time to time. This was one of the hardest bug we’ve had to face to date, getting us to the point of pulling out our hair. Today, we share how we fixed it so that you can avoid pulling out yours.
Close opened resources
The go to solution when you start getting this kind of exceptions is to look for resources like files, sockets, etc that weren’t properly closed.
In our code, we follow the golden rule of closing whatever we open, but still, there we’re some details of our Scala + PlayFramework stack, that we were unaware that caused leaking of resources.
Source.fromFile
Although the scala.io.Source.fromFile method is often used in oneliners and seductive to use in a functional style, it does not close the file opened unless you do it explicitly.
val source = scala.io.Source.fromFile("file.txt")
val lines = try { source.mkString } finally { source.close() }
play-ws
This is another sneaky one. If you’re using play-ws outside a play application, and you’re creating an instance of NingWSClient and casting it to WSClient, you’ll care about this case. Previous to version 2.4, the WSClient interface didn’t exposed a close method, so one would assume that it is managed automatically. Wrong, the file descriptor was left open. To close it you should cast the client to NingWSClient as:
wsClient.underlying[NingWSClient].close()
In newer versions than 2.4, the method is exposed in the WSClient interface.
Increasing the system limits
If you got this far, maybe you’re starting to feel let down. You’ve done everything right, but still your application is crashing. Don’t worry, it’s normal. Some applications just have to handle more files than what OS defaults are set. The rest of the guide is for specific for an Ubuntu stack.
ulimit
The system limits for open file descriptors are set in /etc/security/limits.conf. You can change those by editing the file or by running:
ulimit -Hn 65536
ulimit -Sn 65536
You can now check if the limits were applied to your play application by running:
https://gist.github.com/mrfyda/27a99c1c8a1924bf14d1
In case it didn’t maybe you’re missing this line in the file /etc/pam.d/common-session:
session required pam_limits.so
System services
If after these changes your app is still using the original limits, you may be running it as a service. Our app was being initialised with an initscript which doesn’t honor the limits set in /etc/security/limits.conf. In order to get it working we had to hardcode the limits in the init script.
At last, success! It was a very hard debugging session that spawn over a couple of days. I have to thank to my debugging buddy, the rubber duckling, who keep me sane during this journey.
Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!
References:
https://stackoverflow.com/questions/26220391/how-to-have-a-scala-standalone-application-that-uses-the-playframework-libraries
https://www.playframework.com/documentation/2.4.x/api/scala/index.html#play.api.libs.ws.WSClient
https://rtcamp.com/tutorials/linux/increase-open-files-limit/
http://ubuntuforums.org/archive/index.php/t-1274847.html
http://upstart.ubuntu.com/cookbook/
About Codacy
Codacy is used by thousands of developers to analyze billions of lines of code every day!
Getting started is easy – and free! Just use your GitHub, Bitbucket or Google account to sign up.
GET STARTED
