(This post is really about hearing the thoughts of others who have gone through this, so if you have anything to add, please do so!)
While we’re still a relatively small site, we’re growing pretty quickly (onappstorehq.com as well as running a decent number of white labels and also recently taking over the iPhoneDevSDK forums) and I’m beginning to think about the architecture I’d like to see long-term. I don’t have to make any decisions yet (we’re a fairly standard Rails site so we can scale using fairly standard techniques for some time), but I’m beginning to form some opinions for when the time does come.
The problem I am most conflicted about is arguably the highest-level decision I’ll need to make: Should we have an opinionated architecture or should we have an agnostic architecture? Should we force structure upon ourselves in an effort to simplify, or should we take each case individually and answer it as necessary.
We can represent this problem well by comparing the architectures of Amazon (both the retail site and AWS) with those of Google and Facebook. Amazon takes the agnostic approach, while Google and Facebook lean heavily towards the opinionated approach.(As an ex-Amazonian, I know the ups-and-downs of the Amazon approach, but I’m partially guessing on the Google/Facebook approach so feel free to correct me.)
The Amazon Way
Amazon has an extremely services based architecture. Each team operates autonomously and only interacts with other teams through a set of well-defined APIs. Almost every bit of architecture behind those APIs is up to the owners to decide. The only things that are forced upon a developer are the hardware and the network infrastructure.
The upside to this approach is that each problem is looked at with a blank slate and can be solved and implemented with the approach that fits best. Moreover, with the full stack in your control, debugging operational issues can be less hairy with the full view of what’s occurring.
The two main downsides to this approach that I saw were that (1) the majority of services were very similar, so each team was essentially reinventing the wheel, and (2) each team had higher-than-necessary operational costs because they were fully responsible for everything except the hardware and network (and even some of those because as a developer redundancy/failover is your responsibility).
The Google/Facebook Way
Unlike Amazon, Google and Facebook appear to be much more opinionated. If you work at Google, your background heavy-lifting will likely be done with MapReduce, large storage will be with GFS or BigTable, etc. There are massive clusters of those services already set up and teams who manage them. Similarly, at Facebook, you’re probably storing data in MySQL with a big memcached in front of it.
The advantage here is that service owners can focus on the work that matters to them and their users, while teams dedicated to the core infrastructure pieces (GFS, MapReduce, BigTable, etc) can focus on scaling and managing those for the company. You can also have teams who focus on supporting the live software while other teams focus on developing new features (this is almost impossible in the Amazon way as the only people who know how things work for each service are the developers themselves).
The major disadvantage I see is that not all problems are alike and you may be shoehorning your problem into these solutions. Additionally, debugging operational issues from your end can be more difficult when you lose control of some of these data and services (you don’t have the ability to view the whole stack).
Conclusion
I still have no good answer to this problem, and probably won’t for a long time. However, right now I lean towards the opinionated architecture approach as I think freeing up the developers to focus on solving problems for users is the best use of time, even if it’s at the cost of less than ideal solutions.
I’d love to hear your thoughts on this process, especially if you’ve gone through a scaling like this and had to make decisions…
I don’t have a preference one way or the other, except to advise “Solve the problem you have, not the problem you wish you have” which is sort of a corollary to Eric Ries’ “The #1 reason startups fail is from building working code that no one wants.”
