How Fast is the Released Ruby 2.6.0?
/If you’ve been following me recently, there won’t be a lot of big shocks here.
I generally run Rails Ruby Bench, a big concurrent Rails benchmark based on Discourse, a high-quality piece of open-source forum software that uses Rails. I run 10 processes and 60 threads on an Amazon EC2 m4.2xlarge dedicated instance, then seen how fast I can run a lot of pseudorandom generated HTTP requests through it. This will all be familiar to you if you’ve read much here in the last couple of years.
Later this year there will be some new benchmark that doesn’t work that way. But for right now, let’s check out Ruby 2.6 with good old RRB and see how it stacks up.
On Christmas, Ruby 2.6.0 was released, following its release candidates, which I also speed-tested.
Another Test, Another Graph
The short version is: plain, JIT-less Ruby 2.6.0 is about the same speed as 2.5.3, or maybe just slightly faster. But it’s close enough that it’s hard for me to measure the difference, so I’m going to say “same speed.” It’s unlikely that it’s a full 2% faster (or slower) for instance. And running the benchmark at the accuracy below takes all day, so telling for sure is probably a two- or three-day benchmark run to accurately tell the difference. It’s very similar.
Here are those numbers:
You may get deja vu if you looked at the rc1 and rc2 graphs.
This isn’t bad. Like with rc1 and rc2, the results were very stable - no 500s, no segfaults. Those occasionally occur just randomly, and more on some EC2 instances than others, and I exclude them from the results. But in this case the benchmark just quietly churned away for a whole day without a hiccup on any of the tested Ruby versions - this was a stable, (good kind of) boring test.
JITless 2.6.0 looks a bit faster on this test. But again, it’s hard to tell and these numbers are very close. That Y axis starts at around 47, seconds and the slowest runs are around 55 seconds, not counting the odd bump at the high end of the 2.5.3 numbers. That may have been one random bad run, or it may be that 2.5.3 is slower than 2.5.0 in some cases - this is the first time I have specifically run 2.5.3 as opposed to 2.5.0. Either way, it’s a very small effect, and it doesn’t seem to be present in 2.6.0.
The big difference is with JIT. As of 2.6.0preview3 it looked like JIT wasn’t too far off from JITless performance - maybe 10% or 15% slower? What we’re seeing here is much slower, more like 50% or 60%. Takashi knows about the regression, but it’s basically doing to be awhile before we see JIT helping Rails out. It’s just not there yet. He’s working on it.
Conclusions
For Rails Ruby Bench, 2.6.0 has been a solid, unexciting release - no real Rails speedup, or perhaps a tiny one. The stability is good. JIT’s not useful for big Rails apps yet.
To see more about how 2.6 and JIT stack up we’ll need to look at smaller benchmarks. I have some of that planned, and you’ll see it over the next few months — and at RubyKaigi, if they accept my talk proposal. I have some interesting numbers gathered, and many more to come.
For this year, I’m trying to get my release schedule on a simple track - one post every two weeks, written and scheduled ahead of time. I tried weekly, and it’s just too much. But I feel like last year was a pretty darn good writing year, and it came to almost exactly one post every two weeks. I have a good feeling about this.
Talk to you in two weeks!