Brillio
Senior Lead Data Engineer
Primary Skills
- Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQL
Specialization
- Hadoop and Spark SQL
Key Responsibilities
- Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
- Manage and optimize the infrastructure ensuring reliability and performance
- Design and implement data models and schemas for enterprise data warehouse and data marts
- Write and maintain scripts in Python for various applications
- Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
- Work closely with analysts and product managers to understand data requirements and deliver high-quality data
- Provide technical guidance and mentorship to other team members fostering best practices in data engineering
- Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
Required Skills
- High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
- Good knowledge in Python for data manipulation and scripting
- Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
- Hands‑on experience with workflow management tools like Apache Airflow
- Familiarity with Unix shell scripting for automation and system management