Every computer science student learns the basics of graph theory--a set of mathematical abstractions for modeling networks and the connections within them. But few developers ever work with network data on the scale of Facebook's social graph, with its more than 1 billion users and hundreds of billions of connections, or edges, between them. So in 2012, when Facebook engineers started thinking in earnest about ways to efficiently process all the network data they'd amassed, they found themselves in uncharted territory.
"As far as I know, there's been no work to be cited working with graphs as large as a half a trillion or a trillion edges," says Facebook engineer Avery Ching.
But Facebook still wanted to be able to measure basic statistics about their users' connections, and efficiently do computations like measuring affinities between users to suggest potential friends, or finding mutual friends of sets of users. To make this possible, they turned to an open source graph theory toolkit called Apache Giraph, derived from code originally donated to the Apache Software Foundation by Yahoo.
Making Giraph work at Facebook's scale required a bit of engineering, which Facebook documented on a company blog and contributed back to the open source effort, but Giraph's overall computing model proved to be the ticket to parallelizing huge network graph computations across Facebook's servers.
"It was really easy for us to use," Ching says. "The model itself was very expressive--it allowed you to do a lot of different things."
How Giraph Works
Giraph is based on what's called the bulk synchronous parallel model, a class of parallel computing algorithm developed in the 1980s by researchers led by Harvard computer science professor Leslie Valiant. BSP algorithms take place across multiple computers over a series of steps, and at each step, each computer does a bit of computation and, if it wishes, sends messages about its results to others in the system, which they can incorporate into their computations at the next step.
In Giraph's case, the individual computations are done as if from the perspective of the individual nodes, or vertices, of a graph. And at the end of each step, each node can choose whether to send a message to its immediate neighbors. Before Giraph, that model had been proposed in a Google paper describing an internal tool called Pregel, and Facebook engineers found it's an easy way to describe many graph theory problems in a way that makes them possible to parallelize, by having different computers handle simultaneous computation on behalf of different sets of nodes.
"For large-scale graph computation at Facebook prior to Giraph, people were trying to [use] frameworks such as Hive and MapReduce," says Ching. "You can do it, but it's just really, really slow--think 50 to 100 times slower than it is today."
Finding The Degrees Between People
As a simple example, Giraph's manual presents an implementation of the single-source shortest path algorithm, which measures the distance from any particular node to every other node in the network.
Logically, the length of the shortest path to any particular node is the length of the shortest path to one of its neighbors, plus the distance from that neighbor. So, under Giraph's model, each node begins each step by figuring out the shortest path it knows to itself from the start node, notifying its neighbors if it's learned of a shorter path since last messaging them. Those messages let each node redo its own computation in the next step and again see if it's learned of a shorter route; when a step ends without any nodes needing to send revised paths to their neighbors, each node knows the best path from the source.
And Giraph works for more complex problems, too: A Facebook blog post from earlier this month described how Giraph can be used to assign Facebook users' data to particular database servers. Facebook runs more efficiently when users' information is stored on the same servers as that of their friends, since it means fewer cross-server lookups for common operations, but maximizing the number of users on the same server as their friends is hard with a billion accounts.
The Giraph approach assigns each node to a particular set, representing one server, and lets it tell its neighbors where it's located. At each step, it randomly decides whether to move to another set, with the probability of a move proportional to the number of neighbors in that set.
After a few dozens iterations, that algorithm improved handily on simply assigning nodes to sets based on geographic location, according to the post.
"An early adopter within the company was a service in which typical queries involve fetching a considerable amount of data for each of a person's friends," according to the blog post.
And Facebook remains committed to improving Giraph and contributing the changes it makes back to the open source project, so that it can be used by others dealing with graph data, although some of the algorithms it runs on top of the platform are only shared internally, Ching says.
Before any of Facebook's improvements to Giraph itself are deployed internally, the changes are pushed to the Apache project's open source code repositories, he says.
"We contribute everything we do first to Apache, and then we pull it back through trunk," he says, referring to the project's development branch.
"We work with some researchers and other folks outside the community to just advance Giraph generally," he says. "If people are going to use Apache Giraph, they should use it in the best way possible, which is the version that we're using."