Data Engineer

Description

Job Summary: Design, build, and maintain scalable cloud-based data architectures, ensuring data ingestion, processing, and availability for advanced analytics, BI, and AI models. Key Highlights: 1. Cloud-based data pipeline design and implementation 2. Data lake and data warehouse management and optimization 3. Real-time data ingestion and deployment automation **PURPOSE:** Design, build, and maintain scalable cloud-based data architectures (primarily Azure), ensuring batch and streaming data ingestion, processing, and availability for advanced analytics, BI, and AI models. **RESPONSIBILITIES:** * Design data pipelines * Implement real-time (event-driven) data ingestion solutions * Develop efficient and scalable ETL/ELT processes * Manage storage in data lakes and data warehouses * Optimize performance and costs on cloud platforms * Ensure data quality, governance, and lineage * Integrate diverse sources (APIs, databases, events, IoT) * Automate data pipeline deployments (CI/CD) * Monitor and resolve incidents in production pipelines **TECHNOLOGY STACK:** * Azure product suite * Databricks * Apache Spark (PySpark / Scala) * Apache Kafka (preferred) * Python (required, advanced level) * SQL (advanced level) * CI/CD pipelines **DOMAIN EXPERTISE:** * Data modeling * Big Data handling * Partitioning and optimization * Handling of formats (Parquet, Avro, JSON) * Data security and governance (RBAC, policies) **CONDITIONS:** * Duration: 3 months (with possibility to extend to 6\) * Experience: minimum 2–3 years effective experience in this role * Office attendance: 3 times per week Send CV with photo (mandatory) to reclutamiento@programate.pe with subject line: Data Engineer Salary: S/.4,800\.00 \- S/.5,500\.00 per month Work location: Hybrid in San Borja, Lima