PayPal Central Data Backbone: Couchbase and Kafka

Running a site like PayPal requires both huge scale and a lot of processing to support a complex growing business. The PayPal business over time faces a challenge in managing the user information required to run PayPal services. By leveraging Couchbase the PayPal Data Service team is able to accomodate fast access to user information at scale while streaming data into Hadoop. The solution is able to process millions of updates a day while leveraging Kafka’s high throughput capabilities to get this user data into the Hadoop cluster.

Justin Michaels of Couchbase and Shibi Sudhakaran of PayPal will cover the couchbasekafka project. They will define the problem domain. Followed by an over view and demo of the technology used to create a solution. Couchbase has a number of methods to meet the high availability and durability requirements the PayPal business demands. Justin will review how this is handled within Couchbase while providing real-time access to user data. The PayPal Kafka adapter is leveraged to keep up with the data feed coming out of the Couchbase cluster and transport to Hadoop. Shibi will talk about what makes Kafka unique in it’s ability to handle high throughput to feed the Hadoop pipeline.

Justin has over 15 years of industry experience filling a variety of roles from DBA, Data Integration Development and leadership roles. During this tenure he has experienced and understands the architecture challenges in deploying and maintaining mission critical systems. He is a Solution Architect at Couchbase and responsible for driving customer success, actively managing technical architecture and deployments. Filling the architect role facilitates regular interactions with a wide variety of customers providing enterprise planning and support.

Shibi Sudhakaran is engineer at Paypal, responsible for the development and management of Paypal's server side cookie platform. He has been a software developer since 1999 primarily in Java and related technologies. Prior to joining Paypal, he worked with companies like Cisco, Sun Microsystems, NESS, NEC, Epiphany and many others. He is a contributor to multiple Open Source projects. You can follow him on Twitter @s007.