Data Engineer

Description

Job Summary: Design, build, and maintain scalable cloud-based data architectures, ensuring reliable data ingestion, processing, and availability for advanced analytics, BI, and AI models. Key Highlights: 1. Cloud-based data pipeline design and implementation 2. Big Data management and performance optimization on Azure 3. ETL/ELT process development and real-time data ingestion solutions **PURPOSE:** Design, build, and maintain scalable cloud-based data architectures (primarily on Azure), ensuring batch and streaming data ingestion, processing, and availability for advanced analytics, BI, and AI models. **RESPONSIBILITIES:** * Design data pipelines * Implement real-time data ingestion solutions (event-driven) * Develop efficient and scalable ETL/ELT processes * Manage storage in data lakes and data warehouses * Optimize performance and costs on cloud platforms * Ensure data quality, governance, and lineage * Integrate diverse data sources (APIs, databases, events, IoT) * Automate data pipeline deployments (CI/CD) * Monitor and resolve incidents in production pipelines **TECHNOLOGY STACK:** * Azure product suite * Databricks * Apache Spark (PySpark / Scala) * Apache Kafka (preferred) * Python (required, advanced level) * SQL (advanced level) * CI/CD pipelines **DOMAIN EXPERTISE:** * Data modeling * Big Data handling * Partitioning and optimization * Handling of formats (Parquet, Avro, JSON) * Data security and governance (RBAC, policies) **CONDITIONS:** * Duration: 3 months (with possibility to extend to 6) * Experience: minimum 2–3 years of proven experience in this role * Office attendance: 3 times per week Send CV with photo (mandatory) to reclutamiento@programate.pe with subject line: data engineer Salary: S/.4,800.00 – S/.5,500.00 per month Work location: Hybrid in San Borja, Lima