My Experience With Mob Programming

At Appfolio we often have guest speakers from the software community come in and present topics at our internal tech talks. Three months ago we had guest speaker Woody Zuill here to describe a practice he and his team use called Mob Programming.

We already encourage our developers to pair program due to the numerous advantages we find that it provides. These advantages include having two sets of eyes on a problem and having mentorship between more and less senior developers. Mob programming takes the idea of pair programming to its extreme. The basic concept is to have everyone on the team take part in a collaborative coding session, with one person as the driver and everyone else as navigators.

When Woody began talking to us, he showed us a time-lapse video of his team during their typical work day, with large text over the screen pitching the concept of mob programming. The letters at one point read “Using this technique the team has achieved 10x productivity”. Needless to say, I was a bit skeptical. The rest of the presentation dove into the details of how Woody and his team use mob programming, but the basic concept was easy to understand. The question is, can having your entire team (usually 5-7 people in Woody’s video) focused on the same task using the same screen, keyboard, mouse, and editor actually be better than more traditional methods?

After the talk, my team discussed what we liked and disliked about the idea of mob programming. Since at Appfolio teams can adopt and practice new techniques without any sort of approval, we decided that we’d give mob programming a shot. If it worked then we’d have a valuable new tool. If not then no real harm done besides possibly being a few commits behind. With that in mind, we hijacked a conference room and got to work.

The first feature we tried the new technique out on was very UI heavy. Having the whole team (three developers and one QA engineer) present helped out not only in visual design decisions, but also in remembering which CSS styles to use where. Not to mention, asking product managers or UX specialists to come join the mob for questions and group conversation was very easy.

We finished the feature in considerably less time than we thought it would take, yet still had a very high quality, well tested piece of software. Having other developers around kept us accountable for having good test coverage, using good design patterns, and refactoring any nasty code we’re currently touching.

We’ve since worked on many other features using the technique and found our experience to, for the most part, be more of the same. Recently, it has been especially useful having so many eyes to look at some of the more critical changes we have been working on, such as enhancements to the payments platform. Despite our success, it doesn’t mean that there aren’t downsides or that mob programming works for everyone.

How can we evaluate whether or not a software development technique is better than any other? As far as I can tell, there are at least four things we can try to measure.

1) How productive is the team? (i.e. how much gets done per unit of time)

2) How high quality is the product? (not only customer facing quality but also code quality)

3) How much are team members learning? (do less experienced team members get mentoring)

4) How happy are the individuals on the team? (do you go home happier at the end of the day)

Unsurprisingly, the team did not achieve 10x productivity. In fact, we found our productivity to be almost the same as it was before. We’re not sure how Woody’s team measured their productivity increase, or what their workflow looked like before, but it seems that our findings were not the same here. Is it worth mob programming solely based on productivity boosts? Your mileage may vary, but as far as we’re concerned it’s a resounding no.

Is the product higher quality? Is there better test coverage? Is the code idiomatic and does it follow best practices? Are the chances of a bug crawling into the product minimized? From our experience this is the most emphatic yes of all the concerns listed above. Not only does having everyone together increase accountability and awareness, but mistakes that may be made by more junior developers are more likely to be caught. Furthermore, when our QA engineer was in the mob, he gained a much better sense of how to go about testing the feature as thoroughly as possible.

How much learning is going on? This question doesn’t really have a straightforward answer. Yes, more senior developers get to spread expertise to the less senior ones, and that is enormously useful, but that’s not all that is involved in learning. Personally, I know that sometimes I just need some time to myself to dive into unfamiliar code and figure out what is going on. When the mob is working, people not at the keyboard can’t pause time and open a rails console to experiment, and they can’t go down the rabbit hole of some large stack of functions to find out what’s really going on. And while those at the keyboard may physically be able to do so, they probably shouldn’t if they want the other team members to stay engaged.

Do we go home happier at the end of the day? It’s hard to tell really. I do enjoy getting to interact with all of my team members all day, but it’s honestly a little draining sometimes. Being a fairly introverted person myself, the constant interaction is more of an energy drain than a lot of my previous work at the company, despite possibly being more fun. By the end of the work day I often find myself anxious to get home and recharge with a good book or a show on Netflix.

What other pitfalls might exist? For one, we’ve found that having three developers seems to be the magic number, at least for us. Any more and we’d experience a decrease in productivity without an equally valuable increase in quality, learning, or happiness. Most of our teams have four developers, so this might explain why we are the only team that I am aware of who has tried mob programming and stuck with it. On that note, is it still a mob with only three developers? (Lately our QA engineer seems to get caught up in other work much of the time, despite spending a lot of time in the mob initially) In addition, keeping everyone engaged is important. If one or two people dominate the conversation, it’s pretty easy for others to completely lose focus or get bored. If you find yourself losing focus often, try to be the driver more.

If you do decide to try mob programming, here are a few suggestions that we found work for us:

1) Keep track of time. We have timer cubes that allow whoever is at the keyboard to pick how long they want to drive. Once the timer goes off, switch to a new person.

2) Get a big screen. We started out using large television screens in conference rooms and after getting kicked out of enough rooms we decided to just swipe a nice 50 inch LED screen and put it in our team’s normal work area.

3) Try to merge everyones’ preferred bash/git aliases together to minimize “command not found” errors at the terminal. On the same note, try to have everyones’ editor of choice available. We have two VIM users and one RubyMine user on the team, but we’re all comfortable with both after working together.

So that’s about it. We plan on sticking with mob programming on our team for the short term. However, none of the other teams are still using it. Will we keep using it? Who knows. Yes for the short term, but probably not forever, especially considering how often we rotate teams to spread knowledge. Should you use mob programming? It’s hard to say for sure, but I definitely recommend giving it a try.

Ruby Mixins & ActiveSupport::Concern

A few people have asked: what is the dealio with ActiveSupport::Concern? My answer: it encapsulates a few common patterns for building modules intended for mixins. Before understanding why ActiveSupport::Concern is useful, we first need to understand Ruby mixins.

Here we go…!

First, the Ruby object model:

http://blog.jcoglan.com/2013/05/08/how-ruby-method-dispatch-works/
http://www.ruby-doc.org/core-1.9.3/Class.html

As you can see mixins are “virtual classes” that have been injected in a class’s or module’s ancestor chain. That is, this:

module MyMod
end

class Base
end

class Child < Base
  include MyMod
end

# irb> Child.ancestors
#  => [Child, MyMod, Base, Object, Kernel, BasicObject]

results in the same ancestor chain as this:

class Base
end

class MyMod < Base
end

class Child < MyMod
end

# irb> Child.ancestors
#  => [Child, MyMod, Base, Object, Kernel, BasicObject]

Great? Great.

Modules can also be used to extend objects, for example:

my_obj = Object.new
my_obj.extend MyMod

# irb> my_obj.singleton_class.ancestors
#  => [MyMod, Object, Kernel, BasicObject]

In Ruby, every object has a singleton class. Object#extend does what Module#include does but on an object’s singleton class. That is, the following is equivalent to the above:

my_obj = Object.new
my_obj.singleton_class.class_eval do
  include MyMod
end

# irb> my_obj.singleton_class.ancestors
#  => [MyMod, Object, Kernel, BasicObject]

This is how “static” or “class” methods work in Ruby. Actually, there’s no such thing as static/class methods in Ruby. Rather, there are methods on a class’s singleton class. For example, w.r.t. the ancestors chain, the following are equivalent:

class MyClass
  extend MyMod
end

# irb> MyClass.singleton_class.ancestors
#  => [MyMod, Class, Module, Object, Kernel, BasicObject]

class MyClass
  class << self
    include MyMod
  end
end

# irb> MyClass.singleton_class.ancestors
#  => [MyMod, Class, Module, Object, Kernel, BasicObject]

Classes are just objects “acting” as “classes”.

Back to mixins…

Ruby provides some hooks for modules when they are being mixed into classes/modules:
http://www.ruby-doc.org/core-1.9.3/Module.html#method-i-included
http://www.ruby-doc.org/core-1.9.3/Module.html#method-i-extended

For example:

module MyMod
  def self.included(target)
    puts "included into #{target}"
  end

  def self.extended(target)
    puts "extended into #{target}"
  end
end

class MyClass
  include MyMod
end
# irb>
# included into MyClass

class MyClass2
  extend MyMod
end
# irb>
# extended into MyClass2

# irb> MyClass.ancestors
# => [MyClass, MyMod, Object, Kernel, BasicObject]
# irb> MyClass.singleton_class.ancestors
# => [Class, Module, Object, Kernel, BasicObject]

# irb> MyClass2.ancestors
# => [MyClass2, Object, Kernel, BasicObject]
# irb> MyClass2.singleton_class.ancestors
# => [MyMod, Class, Module, Object, Kernel, BasicObject]

Great? Great.

Back to ActiveSupport::Concern…

Over time it became a common pattern in the Ruby worldz to create modules intended for use as mixins like this:

module MyMod
  def self.included(target)
    target.send(:include, InstanceMethods)
    target.extend ClassMethods
    target.class_eval do
      a_class_method
    end
  end

  module InstanceMethods
    def an_instance_method
    end
  end

  module ClassMethods
    def a_class_method
      puts "a_class_method called"
    end
  end
end

class MyClass
  include MyMod
# irb> end
# a_class_method called
end

# irb> MyClass.ancestors
#  => [MyClass, MyMod::InstanceMethods, MyMod, Object, Kernel, BasicObject]
# irb> MyClass.singleton_class.ancestors
#  => [MyMod::ClassMethods, Class, Module, Object, Kernel, BasicObject]

As you can see, this single module is adding instance methods, “class” methods, and acting directly on the target class (calling a_class_method() in this case).

ActiveSupport::Concern encapsulates this pattern. Here’s the same module rewritten to use ActiveSupport::Concern:

module MyMod
  extend ActiveSupport::Concern

  included do
    a_class_method
  end

  def an_instance_method
  end

  module ClassMethods
    def a_class_method
      puts "a_class_method called"
    end
  end
end

You’ll notice the nested InstanceMethods is removed and an_instance_method() is defined directly on the module. This is because this is standard Ruby; given the ancestors of MyClass ([MyClass, MyMod::InstanceMethods, MyMod, Object, Kernel]) there’s no need for a MyMod::InstanceMethods since methods on MyMod are already in the chain.

So far ActiveSupport::Concern has taken away some of the boilerplate code used in the pattern: no need to define an included hook, no need to extend the target class with ClassMethods, no need to class_eval on the target class.

The last thing that ActiveSupport::Concern does is what I call lazy evaluation. What’s that?

Back to Ruby mixins…

Consider:

module MyModA
end

module MyModB
  include MyModA
end

class MyClass
  include MyModB
end

# irb> MyClass.ancestors
#  => [MyClass, MyModB, MyModA, Object, Kernel, BasicObject]

Let’s say MyModA wanted to do something special when included into the target class, say:

module MyModA
  def self.included(target)
    target.class_eval do
      has_many :squirrels
    end
  end
end

When MyModA is included in MyModB, the code in the included() hook will run, and if has_many() is not defined on MyModB things will break:

irb :050 > module MyModB
irb :051?>   include MyModA
irb :052?> end
NoMethodError: undefined method `has_many' for MyModB:Module
 from (irb):46:in `included'
 from (irb):45:in `class_eval'
 from (irb):45:in `included'
 from (irb):51:in `include'

ActiveSupport::Concern skirts around this issue by delaying all the included hooks from running until a module is included into a non-ActiveSupport::Concern. Redefining the above using ActiveSupport::Concern:

module MyModA
  extend ActiveSupport::Concern

  included do
    has_many :squirrels
  end
end

module MyModB
  extend ActiveSupport::Concern
  include MyModA
end

class MyClass
  def self.has_many(*args)
    puts "has_many(#{args.inspect}) called"
  end

  include MyModB
# irb>
# has_many([:squirrels]) called
end

# irb> MyClass.ancestors
#  => [MyClass, MyModB, MyModA, Object, Kernel, BasicObject]
# irb> MyClass.singleton_class.ancestors
#  => [Class, Module, Object, Kernel, BasicObject]

Great? Great.

But why is ActiveSupport::Concern called “Concern”? The name Concern comes from AOP (http://en.wikipedia.org/wiki/Aspect-oriented_programming). Concerns in AOP encapsulate a “cohesive area of functionality”. Mixins act as Concerns when they provide cohesive chunks of functionality to the target class. Turns out using mixins in this fashion is a very common practice.

ActiveSupport::Concern provides the mechanics to encapsulate a cohesive chunk of functionality into a mixin that can extend the behavior of the target class by annotating the class’ ancestor chain, annotating the class’ singleton class’ ancestor chain, and directly manipulating the target class through the included() hook.

So….

Is every mixin a Concern? No. Is every ActiveSupport::Concern a Concern? No.

While I’ve used ActiveSupport::Concern to build actual Concerns, I’ve also used it to avoid writing out the boilerplate code mentioned above. If I just need to share some instance methods and nothing else, then I’ll use a bare module.

Modules, mixins and ActiveSupport::Concern are just tools in your toolbox to accomplish the task at hand. It’s up to you to know how the tools work and when to use them.

I hope that helps somebody.

Discovery of a MySQL bug

Recently, one of our MySQL instances crashed. We responded to the alert, but by the time we logged into the server, the MySQL instance has already restarted and recovered. So, we started to investigate the cause of the crash.

Our MySQL data volume is nfs-mounted on a NetApp filer in order to allow for reliable crash-recovery and data replication. Our monitoring system showed a spike of near-Gb-line-rate output from the MySQL server to the NetApp filer and it quickly became apparent that MySQL crashed because it ran out of disk space. However, the MySQL disk usage did not change. Instead, the space was used by the NetApp snapshots, each on the order of 20GBs. Snapshots, much like source control, keep track of changes made to files allowing you to view past versions. The NetApp is configured to automatically create snapshots and garbage collect old ones. However, as the rate of change grows, the size of snapshots grows as well. The writing of GBs of data caused enormous snapshots to be created. This meant that whatever wrote those massive files in a short time, also deleted them in that same short time.

Additional metrics in our monitoring system pointed us towards MySQL’s internal temporary tables. MySQL uses these internal temporary tables to satisfy certain queries. When an internal temporary table becomes too big for memory, MySQL flushes it to disk. Once the query finishes, the internal temporary table is deleted. So, in this case, MySQL was processing a query and needed a 20GB internal temporary table!? Naturally, we need to find out what query is causing this and on which data set (each customer is stored in a separate database using the multi-tenancy principle).

Based on mytop and the MySQL slow query log, we were able to single out one suspicious query. It was a query executed by Solr’s data import handler during delta reindex. The application knows which models and which attributes are stored in Solr. Whenever any of those models and attributes change, the application sends a request to Solr asking it to delta reindex. We have an extension in Solr which waits 60 seconds before actually delta reindexing, which effectively coalesces multiple delta reindex requests into a single delta reindex and lightening the load on the MySQL

It turned out that this suspicious query was changed at the beginning of December, but, due to the Christmas / New Year, was only in production for a couple of weeks. The change was rather unusual as it added a join from table A to table X to table B to a query that was already joining from table A to table Y to table B. This was a deliberate change allowing us to migrate from table X to table Y over a few development cycles and the fact that we now have a double join (and a cartesian product) from table A to table B was discussed at development time and was deemed okay. We assumed less than 50 rows resulting from the join of table A to table B, and so in the worse case we’d end up with 2,500 (50 x 50) rows because of the double join.

We found the internal temporary table that MySQL flushed to disk in one of the NetApp’s snapshots. We then matched the data in that file to a particular data set. In this data set, the join from table A to table B had 500 rows, which would produce 250,000 (500 x 500) rows because of the double join. Clearly 250,000 rows is significantly larger than the expected 2,500 rows, but still nothing that MySQL should not be able to handle.

We manually explained and executed this query and we saw no issues whatsoever. Hmmm??? Perhaps this query was not the cause after all?

Nevertheless, we proceeded to copy the data set to our staging servers and see whether we can reproduce the issue there. Based on the suspicious delta reindex query, we knew what to edit in the UI in order to trigger that query. We made the edit and waited. The application asked Solr to delta reindex. Solr waited 60 seconds and started to delta reindex. After few seconds, Solr started to execute our suspicious query and after another 30 seconds, MySQL dumped 20GB internal temporary table to disk. Bingo!

So again we tried to manually explain and execute that query and again we saw no issues whatsoever. Wat!?

We speculated there must be some difference between Solr running the query and us running the query manually. We enabled the MySQL full query log and repeated the edit in the UI. This is what Solr executed,

711723 Query /* mysql-connector-java-5.1.21 ( Revision: ${bzr.revision-id} ) */SELECT @@session.auto_increment_increment
711723 Query SHOW COLLATION
711723 Query SET NAMES utf8mb4
711723 Query SET character_set_results = NULL
711723 Query SET autocommit=1
711723 Query SET sql_mode='STRICT_TRANS_TABLES'
711723 Query SET autocommit=0
711723 Query <delta reindex query>

We quickly focused on SET NAMES utf8mb4. Running the delta reindex query from MySQL client by itself did not cause the problem, but running the delta reindex query after SET NAMES utf8mb4 caused the 20GB file. In fact, we tried other encodings, including utf8, and all of them seem to cause the 20GB file. The only one that worked seemed to be latin1.

This led us to realize that by default our MySQL client uses latin1 whereas our application and Solr use utf8 and utf8mb4, respectively. So in theory, if Solr used latin1 encoding it would avoid the 20GB file. We quickly reconfigured Solr to use latin1 and sure enough, it worked.

Our attention now turned to understanding why that particular delta reindex query on that particular data set with utf8mb4 encoding was causing MySQL to write 20GB file. It turns out MySQL has a bug when a GROUP_CONCAT(DISTINCT …) is too big to fit in memory and requires MySQL to flush to disk. In our case, the bug is triggered because utf8 / utf8mb4 use multiple bytes per character where as latin1 uses one byte per character. You can read more details about this bug here.

Don’t make your users wait for GC

One of the weaknesses of Ruby is garbage collection. Although some improvements have been made (tunable GC settings, lazy sweep, bitmap marking), it is still a simple stop-the-world GC. This can be problematic especially for large applications.

We have a rather large application running on Passenger 3 using REE 1.8.7 with tuned GC. As the application grew, the GC performance has been degrading until we were faced with the following situation where GC averages 130 ms per request.

It was time to address our obvious GC problem.

We could have continued to tune the GC and optimize memory usage to reduce the impact of GC on response times. But why? Why have the GC impact the response time at all? Instead, we decided to trigger GC after one request finishes but before the next request starts. This is not a new idea. It is actually generically referred to as out of band work (OOBW). It has been implemented by Unicorn and discussed here. However, it is not supported by Passenger. So we decided to add OOBW support to Passenger.

Our patch allows the application to respond with an additional header, namely X-Passenger-OOB-Work. When Passenger sees this header, it will stop sending new requests to that application process and tell that application process to perform the out of band work. The application registers oob_work callback for the work it wants done using Passenger’s event mechanism. When the work is done, the application process resumes handling of normal requests.

All the application code can be packaged in an initializer,

PhusionPassenger.on_event(:oob_work) do
 t0 = Time.now
 GC.start
 Rails.logger.info "Out-Of-Bound GC finished in #{Time.now - t0} sec"
end

class RequestOOBWork
 def initialize(app, frequency)
   @app = app
   @frequency = frequency
   @request_count = 0
 end

 def call(env)
   @request_count += 1
   status, headers, body = @app.call(env)
   if @request_count % @frequency == 0
     headers['X-Passenger-Request-OOB-Work'] = 'true'
   end
   [status, headers, body]
 end
end

Rails.application.config.middleware.use RequestOOBWork, 5

After experimentation, we decided to trigger GC after every 5 requests. If you set the parameter too high, then GC will continue to occur in the middle of requests. If you set the parameter too low, then application processes will be busy performing their out-of-band GC causing Passenger to spawn more workers.

With these changes in place, our graphs show 10x improvement in GC average per request,

The patch we implemented and deployed to production is for Passenger 3. It is available here.

Passenger 4 is currently under active development. In fact, beta 1 has already been released. We’ve ported our patch to Passenger 4. It is merged and expected to be included in beta 2.

CSS Architecture

To many Web developers, being good at CSS means you can take a visual mock-up and replicate it perfectly in code. You don’t use tables, and you pride yourself on using as few images as possible. If you’re really good, you use the latest and greatest techniques like media queries, transitions and transforms. While all this is certainly true of good CSS developers, there’s an entirely separate side to CSS that rarely gets mentioned when assessing one’s skill.

Interestingly, we don’t usually make this oversight with other languages. A Rails developer isn’t considered good just because his code works to spec. This is considered baseline. Of course it must work to spec; its merit is based on other things: Is the code readable? Is it easy to change or extend? Is it decoupled from other parts of the application? Will it scale?

These questions are natural when assessing other parts of the code base, and CSS shouldn’t be any different. Today’s web applications are larger than ever, and a poorly thought-out CSS architecture can cripple development. It’s time to evaluate CSS the same way we evaluate every other part of the application. It cannot be an afterthought or written off as merely the “designer’s” problem.

The Goals of Good CSS Architecture

In the CSS community, a general consensus of best practices is very difficult to come by. Judging purely by the comments on Hacker News and the reaction of developers to the release of CSS Lint, it’s clear that many people disagree over even the basic things CSS authors should and shouldn’t do.

So instead of laying out an argument for my own set of best practices, I think we should start by defining our goals. If we can agree upon the goals, hopefully we can start to spot bad CSS not because it breaks our preconceived notions of what’s good but because it actually hinders the development process.

I believe the goals of good CSS architecture shouldn’t be that different from the goals of all good software development. I want my CSS to be predictable, reusable, maintainable, and scalable.

Predictable

Predictable CSS means your rules behave as you’d expect. When you add or update a rule, it shouldn’t affect parts of your site that you didn’t intend. On small sites that rarely change, this isn’t as important, but on large sites with tens or hundreds of pages, predictable CSS is a must.

Reusable

CSS rules should be abstract and decoupled enough that you can build new components quickly from existing parts without having to recode patterns and problems you’ve already solved.

Maintainable

When new components and features need to be added, updated or rearranged on your site, doing so shouldn’t require refactoring existing CSS. Adding component X to the page shouldn’t break component Y by its mere presence.

Scalable

As your site grows in size and complexity it usually requires more developers to maintain. Scalable CSS means it can be easily managed by a single person or a large engineering team. It also means your site’s CSS architecture is easily approachable without requiring an enormous learning curve. Just because you’re the only developer touching the CSS today doesn’t mean that will always be the case.

Common Bad Practices

Before we look at ways to achieve the goals of good CSS architecture, I think it can be helpful to look at common practices that get in the way of our goals. It’s often only through repeated mistakes that we can begin to embrace an alternate path.

The following examples are all generalizations of code I’ve actually written, and, while technically valid, each has lead to disaster and headache. Dispite my best intentions and the promise that this time would be different, these patterns consistently got me into trouble.

Modifying Components Based On Who Their Parents Are

In almost every site on the web there will be a particular visual element that looks exactly the same with each occurrence, except one. And when faced with this one-off situation almost every new CSS developer (and even experienced ones) handles it the same way. You figure out some unique parent for this one particular occurrence (or you create one), and you write a new rule to handle it.

.widget {
  background: yellow;
  border: 1px solid black;
  color: black;
  width: 50%;
}

#sidebar .widget {
  width: 200px;
}

body.homepage .widget {
  background: white;
}

At first this might seem like fairly harmless code, but let’s examine it based on the goals established above.

First, the widget in the examle is not predictable. A developer who’s made several of these widgets will expect it to look a certain way, yet when she uses it in the sidebar or on the homepage, it will look different, despite the markup being exactly the same.

It’s also not very reusable or scalable. What happens when the way it looks on the homepage is requested on some other page? New rules will have to be added.

Lastly, it’s not easily maintainable because if the widget were to get redesigned, it would have to get updated in several places in the CSS, and unlike the above example, the rules that commit this particular anti-pattern rarely appear right next to each other.

Imagine if this type of coding were done in any other language. You’re essentially making a class definition, and then in another part of the code you’re reaching into that class definition and changing it for a particular use case. This directly violates the open/closed principle of software development:

Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.

Later in this article we’ll look at how to modify components without relying on parent selectors.

Overly Complicated Selectors

Occasionally an article will make its way around the Internet showcasing the power of CSS selectors and proclaiming that you can style an entire site without using any classes or IDs.

While technically true, the more I develop with CSS, the more I stay away from complex selectors. The more complicated a selector the more coupled it is to the HTML. Relying on HTML tags and combinators keeps your HTML squeaky clean, but it makes your CSS gross and dirty.

#main-nav ul li ul li div { }
#content article h1:first-child { }
#sidebar > div > h3 + p { }

All of the above examples make logical sense. The first is probably styling a dropdown menu, the second says that the article’s main heading should look different than all other h1 elements, and the last example is likely adding some extra spacing for the first paragraph in the sidebar sections.

If this HTML were never going to change, an argument could be made for its merits, but how realistic is it to assume the HTML will never change? Overly complicated selectors can be impressive and they can rid the HTML of all presentational hooks, but they rarely help us achieve our goals for good CSS architecture.

These examples above are not reusable at all. Since the selector is pointing to a very particular place in the markup, how could another component with a different HTML structure reuse those styles? Taking the first selector (the dropdown) as an example, what if a similar looking dropdown were needed on a different page and it wasn’t inside of the #main-nav element? You’d have to recreate the entire style.

These selectors are also very unpredictable if the HTML needs to change. Imagine that a devloper wanted to change the div in the third example to the HTML5 section tag, the whole rule would break.

Finally, since these selectors only work when the HTML remains constant, they’re by definition not maintainable or scalable.

In large applictaions you have to make trade-offs and compromises. The fragility of complex selectors are rarely worth the price in the name of keeping your HTML “clean”.

Overly Generic Class Names

When creating reusable design components, it’s very common to scope (as it were) the component’s sub-elements inside the component’s class name. For example:

<div class="widget">
  <h3 class="title">...</h3>
  <div class="contents">
    Lorem ipsum dolor sit amet, consectetur adipiscing elit.
    In condimentum justo et est dapibus sit amet euismod ligula ornare.
    Vivamus elementum accumsan dignissim.
    <button class="action">Click Me!</button>
  </div>
</div>

.widget {}
.widget .title {}
.widget .contents {}
.widget .action {}

The idea is that the .title, .contents, and .action sub-element classes can be safely styled without having to worry about those styles spilling over to any other elements styled with the same classes. This is true, but it doesn’t prevent the styling of classes with those same names from spilling into this component.

On a large project it’s very likely that a class name like .title would get used in another context or even by itself. If that happened, the widget’s title would suddenly look very different than intended.

Overly generic class names lead to very unpredictable CSS.

Making a Rule Do Too Much

Sometimes you make a visual component that needs to be 20 pixels from the top left corner of a section in your site:

.widget {
  position: absolute;
  top: 20px;
  left: 20px;
  background-color: red;
  font-size: 1.5em;
  text-transform: uppercase;
}

Then down the road you need to use this exact same component in a different location. The above CSS won’t work because it’s not reusable in different contexts.

The problem is that you’re making this one selector do too much. You’re defining the look and feel as well as the layout and position within the same rule. The look and feel is reusable but the layout and position is not. And since they’re used together, the whole rule is compromised.

While this may seem harmless at first, it often leads to copying and pasting from less CSS-savvy developers. If a new team member wants something to look like a particular component, say an .infobox, they’ll probably start by trying that class. But if that doesn’t work because it positions that new infobox in an undesired way, what are they likely to do? In my experience, most new developers will not break the rule up into its reusable parts. Instead they’ll simply copy and paste the lines of code needed for this particular instance into a new selector, unnecessarily duplicating code.

The Cause

All of the above bad practices share one similarity, they place far too much of the styling burden on the CSS.

That may seem like strange statement. After all, it is a stylesheet; shouldn’t it bear most (if not all) of the styling burden? Isn’t that what we want?

The simple answer to this question is “yes”, but, as usual, things aren’t always so simple. Separating content from presentation is a good thing, but just because your CSS is separate from your HTML doesn’t mean your content is separate from your presentation. Put another way, striping all presentational code from your HTML doesn’t fulfill the goal if your CSS requires an intimate knowledge of your HTML structure in order to work.

Furthermore, HTML is rarely just content; it’s almost always structure too. And often that structure consists of container elements with no purpose other than to allow the CSS to isolate a certain group of elements. Even without presentational classes, this is still clearly presentation mixed into the HTML. But is it necessarily mixing presentation with content?

I believe, given the current state of HTML and CSS, it’s necessary and often wise to have the HTML and CSS work together as a presentational layer. The content layer can still be abstracted away via templates and partials.

The Solution

If your HTML and your CSS are going to work together to form the presentation layer of a web application, they need to do so in a way that promotes all of the principles of good CSS architecture.

The best approach that I’ve found is for the CSS include as little HTML structure as possible. The CSS should define how a set of visual elements look and (in order to minimize coupling with the HTML) those elements should look as they’re defined regardless of where they appear in the HTML. If a certain component needs to look different in a different scenario, it should be called something different and it’s the HTML’s responsibility to call it that.

As an example, the CSS might define a button component via the .button class. If the HTML wants a particular element to look like a button, it should use that class. If there’s a situation were the button needs to look different (perhaps larger and full-width), then the CSS needs to define that look as well with a new class, and the HTML can include that new class to employ the new look.

The CSS defines what your components look like, and the HTML assigns those looks to the elements on the page. The less the CSS needs to know about the HTML structure the better.

A huge benefit of declaring exactly what you want in the HTML is it allows other developers to look at the markup and know exactly what the element is supposed to look like. The intent is obvious. Without this practice it’s impossible to tell if the look of an element is intentional or accidental, and this leads to confusion on the team.

A common objection to putting a lot of classes in the markup is the extra effort required to do so. A single CSS rule could target a thousand instances of a particular component. Is it really worth writing that classes a thousand times just to have it explicitly declared in the markup?

While this concern is clearly valid, it can be misleading. The implication is that either you use a parent selector in the CSS or you write that HTML class 1000 times by hand, but there are obviously other alternatives. View level abstractions in Rails or other frameworks can go a long way toward keeping the visual look explicitly declared in the HTML without having to write the same class over and over again.

Best Practices

After making the above mistakes over and over again, and paying the consequences later on down the road, I’ve come up with the following bits of advice. While by no means comprehensive, my experience has shown that sticking to these principles will help you better achieve the goals of good CSS architecture.

Be intentional

The best way to make sure your selectors don’t style unwanted elements is to not give them the opportunity. A selector like #main-nav ul li ul li div could very easily end up applying to unwanted elements as your markup changes down the road. A style like .subnav, on the other hand, will have absolutely no chance of accidentally applying to an unwanted element. Applying classes directly to the elements you want to style is the best way to keep your CSS predictable.

/* Grenade */
#main-nav ul li ul { }

/* Sniper Rifle */
.subnav { }

Given the two examples above, think of the first one like a grenade and the second like a sniper rifle. The grenade might work just fine today, but you never know when an innocent civilian could move inside the blast radius.

Separate your concerns

I’ve already mentioned that a well organized component layer can help loosen the coupling of HTML structure in the CSS. In addition to that, your CSS components themselves should be modular. Components should know how to style themselves and do that job well, but they should not be responsible for their layout or positioning nor should they make too many assumptions about how they’ll be spaced in relation to surrounding elements.

In general, components should define how they look, but not their layout or position. Be careful when you see properties like background, color, and font in the same rule as position, width, height, and margin.

Layout and position should be handled by either a separate layout class or a separate container element. (Remember that to effectively separate content from presentation it’s often essential to separate content from its container.)

Namespace your classes

We already examined why parent selectors aren’t 100% effective at encapsulation and preventing style cross-contamination. A much better approach is applying namespaces to the classes themselves. If an element is a member of a visual component, every one of its sub-element classes should use the component’s base class name as a namespace.


/* High risk of style cross-contamination */
.widget { }
.widget .title { }

/* Low risk of style cross-contamination */
.widget { }
.widget-title { }

Namespacing your classes keeps your components self-contained and modular. It minimizes the likelihood that an existing class will conflict, and it lowers the specificity required to style child elements.

Extend components with modifier classes

When an existing component needs to look slightly different in a certain context, create a modifier class to extend it.

/* Bad */
.widget { }
#sidebar .widget { }

/* Good */
.widget { }
.widget-sidebar { }

We’ve already seen the downsides of modifying components based on one of their parent elements, but to reiterate: A modifier class can be used anywhere. Location based overrides can only be used in a specific location. Modifier classes can also be reused as many times as you need. Lastly, modifier classes express the intention of the developer very clearly right in the HTML. Location based classes, on the other hand, are completely invisible to a developer only looking at the HTML, greatly increasing the probability that it will be overlooked.

Organize Your CSS Into a Logical Structure

Jonathan Snook, in his excellent book SMACSS, argues for organizing your CSS rules into four separate categories: base, layout, modules, and state. Base consists of reset rules and element defaults. Layout is for positioning of site-wide elements as well as generic layout helpers like grid systems. Modules are reusable visual elements, and state refers to styling that can be toggled on or off via JavaScript.

In the SMACSS system, modules (which are equivalent to what I call components) comprise the vast majority of all the CSS rules, so I often find it necessary to break them down even further into abstract templates.

Components are standalone visual elements. Templates, on the other hand, are building blocks. Templates don’t stand on their own and rarely describe look and feel. Instead, they’re single, repeatable patterns that can be put together to form a component.

To provide a concrete example, a component might be a modal dialog box. The modal might have the site’s signature background gradient in the header, it might have a drop shadow around it, it might have a close button in the top right corner, and it might be positioned fixed and centered vertically and horizontally. Each of these four patterns might be used again and again all over the site, so you wouldn’t want to have to recode those patterns each time. As such they’re all templates, and together they comprise the modal component.

I typically don’t use template classes in the HTML unless I have a good reason. Instead I use a preprocessor to include the template styles in the component definition. I’ll discuss this and my rational for doing so in more detail later.

Use Classes For Styling And Styling Only

Anyone who has worked on a large project has come across an HTML element with a class whose purpose was completely unknown. You want to remove it, but you’re hesitant because it may have some purpose that you’re not aware of. As this happens again and again, over time, your HTML become filled with classes that serve no purpose just because team members are afraid to delete them.

The problem is that classes are generally given too many responsibilities in front-end web development. They style HTML elements, they act as JavaScript hooks, they’re added to the HTML for feature detections, they’re used in automated tests, etc.

This is a problem. When classes are used by too many parts of the application, it becomes very scary to remove them from your HTML.

However, with an established convention, this problem can be completely avoided. When you see a class in the HTML, you should be able to tell instantly what its purpose is. My recommendation is to give all non-styled classes a prefix. I use .js- for JavaScript and I use .supports- for Modernizr classes. All classes without a prefix are for styling and styling only.

This makes finding unused classes and removing them from the HTML as easy as searching the stylesheets directory. You can even automate this process in JavaScript by cross referencing the classes in the HTML with the classes in the document.styleSheets object. Classes that aren’t in document.styleSheets can be safely removed.

In general, just as it’s a best practice to separate your content from your presentation, it’s also important to separate your presentation from your functionality. Using styled classes as JavaScript hooks deeply couples your CSS and JavaScript in a way that can make it hard or impossible to update the look of certain elements without breaking functionality.

Name your classes with a logical structure

These days most people write CSS with hyphens as word separators. But hyphens alone are usually not enough to distinguish between different types of classes.

Nicolas Gallagher recently wrote about his solution to this problem which I have also adopted (with slight changes) with great success. To illustrate the need for a naming convention consider the following:

/* A component */
.button-group { }

/* A component modifier (modifying .button) */
.button-primary { }

/* A component sub-object (lives within .button) */
.button-icon { }

/* Is this a component class or a layout class? */
.header { }

From looking at the above classes, it’s impossible to tell what type of rule they apply to. This not only increases confusion during development, but it also makes it harder to test your CSS and HTML in an automated way. A structured naming convention allows you to look at a class name and know exactly what its relationship is to other classes and where it should appear in the HTML — making naming easier and testing possible where it previously was not.

/* Templates Rules (using Sass placeholders) */
%template-name
%template-name--modifier-name
%template-name__sub-object
%template-name__sub-object--modifier-name

/* Component Rules */
.component-name
.component-name--modifier-name
.component-name__sub-object
.component-name__sub-object--modifier-name

/* Layout Rules */
.l-layout-method
.grid

/* State Rules */
.is-state-type

/* Non-styled JavaScript Hooks */
.js-action-name

The first example redone:

/* A component */
.button-group { }

/* A component modifier (modifying .button) */
.button--primary { }

/* A component sub-object (lives within .button) */
.button__icon { }

/* A layout class */
.l-header { }

Tools

Maintaining an effective and well-organized CSS architecture can be very difficult, especially on large teams. A few bad rules here and there can snowball into an unmanageable mess. Once your application’s CSS has entered into the realm of specificity wars and !important trumps, it can be next to impossible to recover without starting over. The key is to avoid those problems from the beginning.

Fortunately, there are tools that can make controlling your site’s CSS architecture much easier.

Preprocessors

These days it’s impossible to talk about CSS tools without mentioning preprocessors, so this article won’t be any different. But before I praise their usefulness, I should offer a few words of caution.

Preprocessors help you write CSS faster, not better. Ultimately it gets turned into plain CSS, and the same rules should apply. If a preprocessor lets you write your CSS faster then it also lets you write bad CSS faster, so it’s important to understand good CSS architecture before thinking a preprocessor will solve your problems.

Many of the so-called “features” of preprocessors can actually be very bad for CSS architecture. The following are some of the “features” I avoid at all costs (and though the general ideas apply to all preprocessor languages, these guidelines apply specifically to Sass).

  • Never nest rules purely for code organization. Only nest when the outputted CSS is what you want.
  • Never use a mixin if you’re not passing an argument. Mixins without arguments are much better used as templates which can be extended.
  • Never use @extend on a selector that isn’t a single class. It doesn’t make sense from a design perspective and it bloats the compiled CSS.
  • Never use @extend for UI components in component modifier rules because you lose the inheritance chain (more on this in a bit).

The best parts of preprocessors are functions like @extend and %placeholder. Both allow you to easily manage CSS abstraction without adding bloat to your CSS or a huge list of base classes in your HTML that can be very hard to manage.

@extend should be used with care though because sometime you want those classes in your HTML. For example, when you first learn about @extend it might be tempting to use it with all of your modifier classes like so:

.button {
  /* button styles */
}

/* Bad */
.button--primary {
  @extend .button;
  /* modification styles */
}

The problem with doing this is you lose the inheritance chain in the HTML. Now it’s very difficult to select all button instances with JavaScript.

As a general rule, I never extend UI components or anything that I might want to know the type of later. This is what templates are for and another way to help distinguish between templates and components. A template is something you wouldn’t ever need to target in your application logic, and therefore can be safely extended with a preprocessor.

Here’s how it might look using the modal example referenced above:

.modal {
  @extend %dialog;
  @extend %drop-shadow;
  @extend %statically-centered;
  /* other modal styles */
}

.modal__close {
  @extend %dialog__close;
  /* other close button styles */
}

.modal__header {
  @extend %background-gradient;
  /* other modal header styles */
}

CSS Lint

Nicole Sullivan and Nicholas Zakas created CSS Lint as a code quality tool to help developers detect bad practices in their CSS. Their site describes it as such:

CSS Lint points out problems with your CSS code. It does basic syntax checking as well as applying a set of rules to the code that look for problematic patterns or signs of inefficiency. The [rules] are all pluggable, so you can easily write your own or omit ones you don’t want.

While the general ruleset may not be perfect for most projects, the best feature of CSS Lint is its ability to be customized exactly how you want it. This means you can pick and choose the rules you want from their default list as well as write your own.

A tool like CSS Lint is essential for any large team to ensure at least a baseline of consistency and convention compliance. And like I hinted at previously, one of the great reasons for conventions is they allow for tools like CSS Lint to easily identify anything that breaks them.

Based on the conventions I’ve proposed above, it becomes very easy to write rules to detect particular antipatterns. Here are a few suggestions that I use:

  • Don’t allow IDs in your selectors.
  • Don’t use non-semantic type selectors (e.g. DIV, SPAN) in any multi-part rule.
  • Don’t use more than 2 combinators in a selector.
  • Don’t allow any class names that begin with “js-”.
  • Warn if frequently using layout and positioning for non “l-” prefixed rules.
  • Warn if a class defined by itself is later redefined as a child of something else.

These are obviously just suggestions, but they’re intended to get you thinking about how to enforce the standards you want on your projects.

HTML Inspector

Earlier I suggested that it would be easy to search your HTML classes and all linked stylesheets and warn if a class was used in the HTML but not defined in any stylesheet. I’m currently developing a tool called the HTML Inspector to make this process easier.

HTML Inspector traverses your HTML and (much like CSS Lint) allows you to write your own rules that throw errors and warnings when some convention is being broken. I currently use the following rules:

  • Warn if the same ID is used more than once on a page.
  • Don’t use any classes that aren’t mentioned in any stylesheet or pass a whitelist (like “js-” prefixed classes).
  • Modifer classes shouldn’t be used without their base class.
  • Sub-object classes shouldn’t be used when no ancestor contains the base class.
  • Plain old DIV or SPAN elements, without classes attached, should not be used in the HTML.

Summary

CSS isn’t just visual design. Don’t throw out programming best practices just because you’re writing CSS. Concepts like OOP, DRY, the open/closed principle, separation of concerns, etc. still apply to CSS.

The bottom line is that whatever you do to organize your code, make sure you judge your methods by whether or not they actually help make your development easier and more maintainable in the long term.

About the Author

Philip Walton is a Front End Software Engineer at AppFolio. For more of this thoughts you can read his blog or follow him on Twitter:

http://philipwalton.com
@philwalton