Ruby 2.7preview2, a Quick Speed Update
/As you know, I like to check Ruby’s speed for running big Rails apps. Recently, Ruby 2.7 preview2 was released. Normally Ruby releases a new version every Christmas, so it was about time.
I’ve run Rails Ruby Bench on it to check a few things - first, is the speed significantly different? Second, any change in Ruby’s JIT?
Today’s is a pretty quick update since there haven’t been many changes.
Speed and Background
Mostly, Ruby’s speed jumps are between minor versions - 2.6 is different from 2.5 is different from 2.4, but there’s not much change between 2.4.0 and 2.4.5, for instance. I’ve done some checking of this, and it’s held pretty true over time. It's much less true of prerelease Ruby versions, as you’d expect - they’re often still getting big new optimisations, so 2.5’s prereleases were quite different from each other, and from the released 2.5. That’s appropriate and normal.
But I went ahead and speed-checked 2.6.5 against 2.6.0. While these small changes don’t usually make a significant difference, 2.6.0 was one I checked carefully.
And of course, over time I’ve checked how JIT is doing with Rails. Rails is still too tough for it, but there’s a difference in how close do breakeven it is, depending on both what code I’m benchmarking and exactly what revision of JIT I’m testing.
Numbers First
While I’ve run a lot of trials of this, the numbers are fairly simple - what’s the median performance of Ruby, running Discourse flat-out, for this number of samples? This is code I’ve benchmarked many times in roughly this configuration, and it turns out to be well-summarised by the median.
In this case, the raw data is small enough that I can just hand it to you. Here’s my data for 90 runs per configuration with 10,000 HTTP requests per run, with everything else how I generally do it:
| Ruby version | Median reqs/sec | Std. Dev. | Variance |
|---|---|---|---|
| 2.6.0 | 174.0 | 1.47 | 2.17 |
| 2.6.5 | 170.1 | 1.69 | 2.86 |
| 2.7.0 | 175.6 | 1.63 | 2.67 |
| 2.7.0 w/ JIT | 110.4 | 1.05 | 1.11 |
One of the first things you’re likely to notice: except for 2.7 with JIT, which we expect to be slow, these are all pretty close together. The difference between 2.6.5 and 2.7.0 is only 5.5 reqs/second, which is a little over three standard deviations - not a huge difference.
I’ve made a few trials, though, and these seem to hold up. 2.6.5 does seem just a touch slower than 2.6.0. The just-about-2% slower that you’re seeing here seems typical. 2.7.0 seems to be a touch faster than 2.6.0, but as you see here, it would take a lot of samples to show it convincingly. One standard deviation apart like this could easily be measurement error, even with the multiple runs I’ve done separately. This is simply too close to call without extensive measurement.
Conclusions
Sometimes when you do statistics, you get the simple result: overall, Ruby 2.7 preview 2 is the same speed as 2.6.0. There might be a regression in 2.6.5, but if so, it’s a small one and there’s a small optimisation in 2.7 that’s balancing it out. Alternately, all these measurements are so close that they may all, in effect, be the same speed.