About this role
About the Role As a Staff Software Engineer on the Data Platform team, you will help build the Databricks Data Intelligence Platform and automate decision-making across the company. You will collaborate with Product, Data Science, Applied AI, and other teams to design scalable data infrastructure and tooling across multi-cloud environments.
What You'll Do
- Design and run the Databricks metrics store that enables all business units and engineering teams to bring their detailed metrics into a common platform with high-quality, introspection, and fast query performance.
- Design and run the cross-company Data Intelligence Platform, which contains every business and product metric used to run Databricks.
- Develop tooling and infrastructure to efficiently manage and run Databricks on Databricks at scale, across multiple clouds, geographies and deployment types, including CI/CD processes, data quality test frameworks, and infrastructure-as-code tooling.
- Design the base ETL framework used by all pipelines developed at the company.
- Partner with engineering teams to provide leadership in developing the long-term vision and requirements for the Databricks product.
- Build reliable data pipelines and solve data problems using Databricks, partner products, and other OSS tools.
- Provide early feedback on the design and operations of these products.
- Establish conventions and create new APIs for telemetry, debugging, feature and audit event log data, and evolve them as the product and underlying services change.
- Represent Databricks at academic and industrial conferences & events.
What We're Looking For
- 12+ years of industry experience
- 4+ years of experience building large-scale distributed systems
- 5+ years providing technical leadership on large projects similar to the ones described above (ETL frameworks, metrics stores, infrastructure management, data security)
- Experience building, shipping and operating reliable multi-geo data pipelines at scale
- Experience working with and operating workflow or orchestration frameworks, including open-source tools like Airflow and DBT or commercial enterprise tools
- Experience with large-scale messaging systems like Kafka or RabbitMQ or commercial systems
- Excellent cross-functional and communication skills, consensus builder
- Passion for data infrastructure and enabling others by making their data easier to access
Compensation & Benefits
- Benefits: Comprehensive benefits and perks; region-specific details available via the provided link
- Salary: Not disclosed in the posting