Looking to provide insight to our non-production infrastructure resource usage, the database services team was tasked to put together a solution for developers, business stakeholders and infrastructure administrators to see real-time metrics across the application stack. We used multiple open source technologies including Hadoop and NoSQL databases to provide a "console" that they could access and leverage the data to make sizing production infrastructure easier. This presentation will go through our technology selection process, modeling the time series data, hurdles we encountered along the way and the ways we are gathering the data to push into the solution.

Many companies, small and large, do not have robust solutions to monitor non-production infrastructure. Our use case for using HBase/Hadoop, OpenTSDB, Riemann, RabbitMQ, and Logstash was that they were all open source, ran on commodity hardware or small virtual servers and had large community support groups.