Rule Engines, Cassandra, ElasticSearch and the Complex Join
Share this Session:
  Igor Okulist   Igor Okulist
Senior Software Engineer
Netflix
 


 

Tuesday, August 18, 2015
03:00 PM - 03:45 PM

Level:  Technical - Advanced


Behind the scenes of the Netflix Media Processing pipeline lies a distributed Rule Engine, executing the complex workflows involved in maintaining a large catalog of media and its distribution for a vast number of devices. The most recent generation has evolved from relational databases into the realms of NoSQL, providing significant speed and scalability improvements. However, rule engines are, by definition, complex Join machines, and Joins are not so welcome in the world of NoSQL. We present our solution, which involves a common forward chaining rule engine, a fast and reliable Cassandra store, an ElasticSearch index and a rule engine query optimizer that combines them all together. - Implementing Netflix' Media Processing workflow using a Rule Engine - Implementing a high performance, high reliability, large scale Rule Engine over Cassandra - Adding efficient near-real-time ElasticSearch indexing - Optimizing a global Rule Engine Join query language, with ElasticSearch and Cassandra


Igor Okulist is Senior Software Engineer at Netflix, Media Content Engineering focusing on media asset processing at scale. Prior experience imaging, healthcare, visualization with local and global teams. Specializing in software architecture, large scale development and applying common sense to complex projects.


   
Close Window