Brillio
Senior Lead Data Engineer - R01557349
Brillio, San Francisco, California, United States, 94199
Senior Lead Data Engineer
Primary Skills
Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQL
Specialization
Hadoop and Spark SQL
Key Responsibilities
Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
Manage and optimize the infrastructure ensuring reliability and performance
Design and implement data models and schemas for enterprise data warehouse and data marts
Write and maintain scripts in Python for various applications
Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
Work closely with analysts and product managers to understand data requirements and deliver high-quality data
Provide technical guidance and mentorship to other team members fostering best practices in data engineering
Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
Required Skills
High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
Good knowledge in Python for data manipulation and scripting
Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
Hands‑on experience with workflow management tools like Apache Airflow
Familiarity with Unix shell scripting for automation and system management
#J-18808-Ljbffr
Primary Skills
Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQL
Specialization
Hadoop and Spark SQL
Key Responsibilities
Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
Manage and optimize the infrastructure ensuring reliability and performance
Design and implement data models and schemas for enterprise data warehouse and data marts
Write and maintain scripts in Python for various applications
Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
Work closely with analysts and product managers to understand data requirements and deliver high-quality data
Provide technical guidance and mentorship to other team members fostering best practices in data engineering
Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
Required Skills
High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
Good knowledge in Python for data manipulation and scripting
Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
Hands‑on experience with workflow management tools like Apache Airflow
Familiarity with Unix shell scripting for automation and system management
#J-18808-Ljbffr