Sr. Data Engineer

Sr. Data Engineer

Sr. Data Engineer, Archer Daniels Midland Company, Erlanger, KY. Design, build, refactor, and maintain data pipelines to meet business requirements for reporting, analysis, and data science. Teach, adhere to, and contribute to DataOps and MLOps standards and best practices and continuously improve and accelerate data system performance. Design and integrate fault tolerance and enhancements into data pipelines to improve quality and performance. Perform root cause analysis and solve problems using analytical and technical skills to optimize data delivery and reduce costs. Engage business end users and shares responsibility leading a delivery team. Leverage in depth knowledge of HR systems to build efficient and robust analytical solutions. Demonstrate in depth platform understanding of Azure to design an automated FinOps solution using secure REST APIs. Utilize PySpark for scalable transformation of billing data, which enables real-time spend visibility and advanced analytics, detect cost anomalies, model usage forecasts, and uncover optimization opportunities across computers, storage, and data pipelines. Integrate real-time and event-driven workflows, building APIs and process objects to orchestrate data flows across enterprise systems. Demonstrate domain expertise in agricultural nutrition and Procure-to-Consume (P2C), with a clear understanding of how customer nutrition orders, and commercial demand drive commodity sourcing, manufacturing execution, inventory consumption, supplier invoicing, and financial outcomes across an ag services enterprise. Apply P2C and Nutrition business understanding to design and govern end-to-end data solutions, integrated with SAP and core ERP platforms, that connect orders, production and inventory movements, purchase orders, supplier invoices, settlement and payment, providing leadership with reliable visibility into cost-to-serve, margins, and working capital. Deliver executive-level analytics using Microsoft Power BI on nutrition product volumes, input cost variability, supplier performance, invoice accuracy, and consumption trends, translating complex operational and financial data into clear insights that support cost management, and strategic planning.

40 hrs/week, Mon-Fri, 8:30 a.m. - 5:30 p.m. MINIMUM REQUIREMENTS: Bachelor’s degree or foreign equivalent degree in Engineering (any), Information Systems, Business, Computer Science, Software Engineering, or a related field, and seven (7) years of post-bachelor’s, progressive, related work experience.

Must have seven (7) years of experience with/in:

• At least three (3) of the following cloud-based data systems: Microsoft Azure, SQL, Azure Data Factory, Azure Synapse, Databricks, Python, Pyspark, SAP Datasphere, and/or Power BI;
• Azure DevOps, GitHub, and CI/CD implementation;
• Designing and developing real-time and batch data pipelines using Qlik Replicate and Compose, enabling seamless integration across heterogeneous systems (e.g., Oracle, SQL Server, or JDEdwards) with full CDC (Change Data Capture) support;
• Designing and coding complex data transformations using advanced processing techniques to extract actionable business insights from large, disparate, and disconnected datasets, aligning data solutions with domain requirements, KPIs, and strategic decision-making objectives;
• Experience in building scalable ETL/ELT frameworks utilizing REST APIs, Graph APIs, event-based ingestion, CDC patterns, file-based ingestion (e.g., JSON, Parquet, Avro, or CSV), SFTP, and cross-platform database connectivity within hybrid and multi-cloud environments;
• Cloud-native storage, data lakes and distributed computing, dimensional modeling, enterprise data warehouses, and master data processes;
• Integrating robust security measures, such as end-to-end encryption, PGP encryption methods, and data masking for sensitive information to ensure compliance with data privacy regulations; and
• Integrating pipeline security controls such as secret management using Azure Key Vault, role-based approvals, artifact integrity checks, and environmental isolation.

Telecommuting permitted on a hybrid schedule as determined by the employer.