Don’t make your users wait for GC

One of the weaknesses of Ruby is garbage collection. Although some improvements have been made (tunable GC settings, lazy sweep, bitmap marking), it is still a simple stop-the-world GC. This can be problematic especially for large applications.

We have a rather large application running on Passenger 3 using REE 1.8.7 with tuned GC. As the application grew, the GC performance has been degrading until we were faced with the following situation where GC averages 130 ms per request.

It was time to address our obvious GC problem.

We could have continued to tune the GC and optimize memory usage to reduce the impact of GC on response times. But why? Why have the GC impact the response time at all? Instead, we decided to trigger GC after one request finishes but before the next request starts. This is not a new idea. It is actually generically referred to as out of band work (OOBW). It has been implemented by Unicorn and discussed here. However, it is not supported by Passenger. So we decided to add OOBW support to Passenger.

Our patch allows the application to respond with an additional header, namely X-Passenger-OOB-Work. When Passenger sees this header, it will stop sending new requests to that application process and tell that application process to perform the out of band work. The application registers oob_work callback for the work it wants done using Passenger’s event mechanism. When the work is done, the application process resumes handling of normal requests.

All the application code can be packaged in an initializer,

PhusionPassenger.on_event(:oob_work) do
 t0 =
 GC.start "Out-Of-Bound GC finished in #{ - t0} sec"

class RequestOOBWork
 def initialize(app, frequency)
   @app = app
   @frequency = frequency
   @request_count = 0

 def call(env)
   @request_count += 1
   status, headers, body =
   if @request_count % @frequency == 0
     headers['X-Passenger-Request-OOB-Work'] = 'true'
   [status, headers, body]

Rails.application.config.middleware.use RequestOOBWork, 5

After experimentation, we decided to trigger GC after every 5 requests. If you set the parameter too high, then GC will continue to occur in the middle of requests. If you set the parameter too low, then application processes will be busy performing their out-of-band GC causing Passenger to spawn more workers.

With these changes in place, our graphs show 10x improvement in GC average per request,

The patch we implemented and deployed to production is for Passenger 3. It is available here.

Passenger 4 is currently under active development. In fact, beta 1 has already been released. We’ve ported our patch to Passenger 4. It is merged and expected to be included in beta 2.

27 thoughts on “Don’t make your users wait for GC

  1. Hi pkmiec, thanks for contributing this patch, it was a pleasure working with you!

    I’m thinking how much cooler it would be if there’s way to do *delayed GC* on top of out-of-band GC. Whenever the Ruby GC detects the need to trigger a garbage collection, it would set a flag. The code can then later on invoke the garbage collector when needed. This requires patching Ruby though.

    Maintaining REE and keeping quality assurance was an extremely resource intensive task, which was one of the reasons why we had to End-Of-Life it. Such a patch should be accepted by upstream for it to become viable. Maybe they’d accept a patch that introduces GC hooks so that new GC functionality can be implemented in extensions. That would make maintenance and development much easier.

    Just thinking out loud.

    • Thank you for all you help and support on this patch.

      Along the lines of the *delayed GC* idea, I initially tried to implement some sort of heuristic on when to respond with the X-Passenger-OOB-Work header, but could not get it to work in a reasonable way. So I just settled on every nth request.

      I feel a lot of effort has gone into the ruby gc, but integrating that effort into the mainstream ruby has been very very slow. You guys made REE incorporating the railsbench gc patch and copy-on-write friendliness. Twitter invested into Kiji which implements a limited version of generation gc. Rubinius, to my understanding, has a much more sophisticated generational gc. Yet, MRI 1.9 only included the tunable GC and lazy sweep. Never mind, it wasn’t really usable until 1.9.3 and we lost tools like memprof. Ruby 2.0 will have bitmap marking which is great but also disappointing it took so long to effectively implement the same thing that REE had.

      I totally agree pluggable GC would be a huge improvement. I recall this having been proposed on the mailing list at some point. I believe one issue is the lack of proper way of dealing with c extensions, which Brian Ford talked about at the last ruby conf:

  2. Pingback: Phusion Passenger 4 Technology Preview: Out-Of-Band Work – Phusion Corporate BlogPhusion Corporate Blog

  3. Pingback: Phusion Passenger now supports the new Ruby 2.1 Out-Of-Band GC – Phusion Corporate Blog

  4. Hello there! I could have ssworn I’ve besen tto this site before but after reading throujgh some
    of the post I realizd it’s new to me. Anyways, I’m defnitely
    glad I found it aand I’ll be bookmarkiing annd checking back often!

  5. If the man yyou aree buying for likes to make a fashhion statement, youu may wish to pick out a sport
    coat in a bold fabric, such as velvet. If yokur shirts develop sweat stains,
    pre-treat them before laundering them by dabbing them ith white vinegar or a launddry stain treatment.
    Can you get away with wearing a black suit, rather than a black tuxedo, at a black tie wedding.

  6. Attractive component of content. I just stumbled upon your website and in accession capital to assert that I get inn fact loved account your
    blog posts. Anyay I will be subscribing to your feeds
    and even I success you get admission to constantly quickly.

  7. SECURITY HARBOR – The city of Security Harbor is internet hosting
    the annual Holiday Discount Bazaar on Saturday, Dec.
    ehow, Lineman apprentice packages near On one hand, there may be
    higher demand for electricians throughout peak periods of construction constructing and upkeep., Employment Information Paper/ Rozgaar Samachaar published every week E-Paper for candidates who’re eligible
    for Govt./ Bank/ SSC/ PSC Job Vacancies 2015.

  8. Check out your links below and you will find that
    there are many more ideas certainly burn the fat. Being overweight
    is debilitating situation by which one could be at and the higher chances of
    developing serious health hazards like non insulin dependent
    diabetes,hypertension,stroke and heart disease.
    Drinking lots of water will eliminate being hungry.

  9. View the actual problem is within the actual complicated as well as
    diverse, so long as all of us very carefully notice
    appropriate will discover what the law states from the location. If you want it
    for a week or so then you can look at reputed Chennai car rental companies.
    a few seconds of spare journey, to every location to various course of action motion.

  10. Hello there I am so happy I found your web site, I
    really found you by mistake, while I was looking on Bing for
    something else, Regardless I am here now and would just like to
    say kudos for a tremendous post and a all round enjoyable blog
    (I also love the theme/design), I don’t have time to go through
    it all at the minute but I have book-marked it and also
    added in your RSS feeds, so when I have time I will
    be back to read a great deal more, Please do keep up the awesome jo.

  11. I am now not certain the place you are getting your info, however great topic.

    I needs to spend a while finding out much more
    or figuring out more. Thank you for great information I used to
    be searching for this information for my mission.

  12. Most common capacities are 5000 litres, 10000 litres, and
    20000 litres. In other words, structures would be designed or modified to collect the natural rainfall that
    falls onto the property, purify it and then store it in cisterns until the water is
    needed by the occupants of the building. The most popular types and
    styles of these include the following:.

  13. Hey, I think your site might be having browser compatibility
    issues. When I look at your website in Opera, it looks fine but when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up!
    Other then that, wonderful blog!

  14. What i don’t realize is in truth how you’re now not
    actually much more smartly-preferred than you may be now.
    You’re so intelligent. You know thus considerably in the case
    of this matter, produced me personally believe it from a lot
    of varied angles. Its like women and men are not interested until it is one thing to do with Woman gaga!
    Your individual stuffs excellent. Always handle
    it up!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s