zCloud: The 'Hidden Jewel Inside Zynga'Over the last year, Zynga has quietly gone from having 80% of its games hosted on Amazon Web Services and 20% on their own servers, to 80% on Zynga's own zCloud and only 20% on Amazon Web Services. This transition, said Allan Leinwand, CTO of Infrastructure at Zynga, has been very important to Zynga. Leinwand calls zCloud “the hidden jewel inside of Zynga.”
Leinwand noted that Zynga has a huge scale to deal with. “We need to make our games accessible to millions of players, which requires an amazing amount of infrastructure,” said Leinwand. “We started by setting up our own servers, but back in 2009 FarmVille became wildly successful. In the first six weeks it went from 0 users to over 10 million daily users. That was a ramp that took a lot of people by surprise.”
That speedy ramp is something that distinguishes social gaming from most other web services. “Social gaming is something that can take off quickly; FarmVille hit 25 million DAU in five months,” noted Leinwand. Dealing with that reality was an eye-opener for Zynga. “We realized we couldn't continue to scale up an infrastructure at that pace,” said Leinwand. “So we ended up leveraging public cloud, specifically Amazon Web Services. We used the public cloud's flexibility to really grow our games.”
This led Zynga to develop its own infrastructure, but it had to work seamlessly with Amazon's so that users would never notice any issues. “In 2009 and through the middle of 2010, we would launch games in AWS and scale them up there. We had our private data centers and Amazon's centers. Over the course of 2010 we started to change that, and grow a private cloud we called zCloud. We were really the first people to build a hybrid public-private cloud. We continued to build out zCloud over the course of 2011. zCloud feels a lot like Amazon Web Services,” noted Leinwand.
This was not an easy process, and required a lot of tuning to make it work properly. As the zCloud has been built out, Zynga has shifted its infrastructure strategy to take advantage of this. Zynga would launch games on Amazon, and then when they could plot the slope of the capacity demand, they could put the game on zCloud. Zynga's mobile games also rely on zCloud as the social games do. “Social games, regardless of the end device, need to have infrastructure that's flexible and scalable,” said Leinwand.
Zynga is constantly rolling out updates and new features to all of its games, and “All these new features affect the infrastructure,” said Leinwand. Zynga's games are available worldwide, which poses its own set of problems. Leinwand wouldn't give details of server locations, but he did affirm that Zynga leverages other services in various areas to take care of the latency of slower ISPs in a given location. “When you launch you need to be able to architect the game to deal with latencies that may or may not be in the system. I can tell you we spend a lot of time working with our studios and thinking about reaching the globe.”
Securtity is always an issue for Zynga. ”Across the globe I don't think there's any shortage of people who like to do malicious activities on the Internet,” Leinwand pointed out. “Since Zynga is such a large company we've seen our fair share of such activities. We spend a lot of time making sure the infrastructure is safe for our players. We always have to be vigilant.” Perhaps a good measure of their success is that we've had no stories about Zynga's games being hacked, or servers being down because of hackers.
In sum, 2011 has been a big year for Zynga's CTO. “We used to outsource 80% of our players, and now we host 80% of them. In 2011 we were trying to change the way we used the public cloud, and we were trying to tune up our performance,” said Leinwand. What are the challenges for 2012? “Making sure the infrastructure continues to scale. I like to say, you have to get to silicon eventually. There are servers that are getting racked and stacked, there is fiber to interconnect. We want to make sure that we can scale ahead of the demand, and make sure it's reliable. We need to keep optimizing the performance of everything we have to make it a finely tuned race car for social games. We can add a thousand servers to zCloud in 24 hours. Of course, those servers have to live in a data center that has to have power and cooling, and that has to be planned in advance. Concrete doesn't dry that fast.”
Leinwand is justifiably proud of the infrastructure Zynga has built. “We've built automation tools, development platforms, monitoring capabilities. Behind Zynga games there's technology and innovation and tools and automation that we really think we're leading on.”
