Wfnen
This is an exciting opportunity to join a dynamic and growing organization working at the forefront of technology trends and developments in the social impact sector. Wadhwani Center for Government Digital Transformation (WGDT) collaborates with government ministries and state departments in India with a mission of " Enabling digital transformation to enhance the impact of government policy, initiatives, and programs ".
We are seeking a highly motivated and detail-oriented individual to join our team as a
Data Engineer . The role involves designing, constructing, and maintaining the architecture and infrastructure necessary for data generation, storage, and processing. You will contribute to the successful implementation of digital government policies and programs, developing robust, scalable, and efficient systems to manage large data volumes, facilitate analysis and decision-making, and drive innovation across various government entities in India. Key Responsibilities:
Data Architecture Design:
Design, develop, and maintain scalable data pipelines and infrastructure for ingesting, processing, storing, and analyzing large datasets efficiently. Translate business requirements into technical solutions. Data Integration:
Integrate data from various sources such as databases, APIs, streaming platforms, and third-party systems, ensuring data reliability, quality, and integrity per government standards. Data Modeling:
Design data models to organize and structure data for efficient storage and retrieval, using techniques like dimensional modeling, normalization, and denormalization. Data Pipeline Development/ETL:
Develop ETL processes to extract data from sources, transform it, and load into target systems, automating workflows to ensure data accuracy and consistency. Data Quality and Governance:
Implement data quality checks and governance policies, including data lineage, stewardship, metadata management, and building business glossaries. Data Lakes or Warehousing:
Design and maintain data lakes and warehouses to store structured, semi-structured, and unstructured data, integrating with big data frameworks and analytics tools. Data Security:
Apply security practices to protect data throughout its lifecycle, including access controls, encryption, masking, anonymization, and compliance with regulations like DPDP and GDPR. Database Management:
Administer and optimize relational and NoSQL databases for effective data management. Data Migration:
Plan and execute data migration projects ensuring minimal downtime and data consistency. Performance Optimization:
Enhance data pipeline and query performance through tuning, caching, and indexing strategies. Collaboration:
Work with data scientists, analysts, and IT teams to meet data requirements and maintain infrastructure in production. Documentation and Reporting:
Document data models, pipelines, and configurations; create training materials for team sustainability. Continuous Learning:
Keep abreast of latest data engineering technologies and trends, participating in training and community engagement. Desired Skills/Competencies:
Education: Bachelor's or Master's in Computer Science, Data Science, or related fields with 5+ years of experience. Database Management: Expertise in SQL and NoSQL databases like MySQL, PostgreSQL, MongoDB, Cassandra. Big Data Technologies: Familiarity with Hadoop, Spark, and related tools. ETL Tools: Experience with Apache NiFi, Talend, Airflow, etc. Data Modeling and Warehousing: Knowledge of data modeling and warehousing solutions like Redshift, BigQuery, Snowflake. Data Governance and Security: Understanding of data quality and security best practices. Cloud Computing: Experience with AWS, Azure, Google Cloud and their data services. Streaming Data Processing: Knowledge of Kafka, Flink, and real-time data processing frameworks. KPIs:
Data Pipeline Efficiency Data Quality Metrics System Uptime and Availability Data Storage Efficiency Data Security and Compliance Data Processing Performance Scalability and Performance Tuning Resource Utilization and Cost Optimization Incident Response and Resolution Documentation and Knowledge Sharing Experience: 3-5 years in similar roles. Career progression to CTO WGDT (Head of Incubation Centre). WF is a global non-profit working in a fast-paced, innovative environment emphasizing Quality, Speed, and Scale. We value excellence, creativity, collaboration, and societal impact, fostering an inclusive culture that promotes continuous learning and adaptation. We are an equal opportunity organization committed to diversity and inclusion. #J-18808-Ljbffr
Data Engineer . The role involves designing, constructing, and maintaining the architecture and infrastructure necessary for data generation, storage, and processing. You will contribute to the successful implementation of digital government policies and programs, developing robust, scalable, and efficient systems to manage large data volumes, facilitate analysis and decision-making, and drive innovation across various government entities in India. Key Responsibilities:
Data Architecture Design:
Design, develop, and maintain scalable data pipelines and infrastructure for ingesting, processing, storing, and analyzing large datasets efficiently. Translate business requirements into technical solutions. Data Integration:
Integrate data from various sources such as databases, APIs, streaming platforms, and third-party systems, ensuring data reliability, quality, and integrity per government standards. Data Modeling:
Design data models to organize and structure data for efficient storage and retrieval, using techniques like dimensional modeling, normalization, and denormalization. Data Pipeline Development/ETL:
Develop ETL processes to extract data from sources, transform it, and load into target systems, automating workflows to ensure data accuracy and consistency. Data Quality and Governance:
Implement data quality checks and governance policies, including data lineage, stewardship, metadata management, and building business glossaries. Data Lakes or Warehousing:
Design and maintain data lakes and warehouses to store structured, semi-structured, and unstructured data, integrating with big data frameworks and analytics tools. Data Security:
Apply security practices to protect data throughout its lifecycle, including access controls, encryption, masking, anonymization, and compliance with regulations like DPDP and GDPR. Database Management:
Administer and optimize relational and NoSQL databases for effective data management. Data Migration:
Plan and execute data migration projects ensuring minimal downtime and data consistency. Performance Optimization:
Enhance data pipeline and query performance through tuning, caching, and indexing strategies. Collaboration:
Work with data scientists, analysts, and IT teams to meet data requirements and maintain infrastructure in production. Documentation and Reporting:
Document data models, pipelines, and configurations; create training materials for team sustainability. Continuous Learning:
Keep abreast of latest data engineering technologies and trends, participating in training and community engagement. Desired Skills/Competencies:
Education: Bachelor's or Master's in Computer Science, Data Science, or related fields with 5+ years of experience. Database Management: Expertise in SQL and NoSQL databases like MySQL, PostgreSQL, MongoDB, Cassandra. Big Data Technologies: Familiarity with Hadoop, Spark, and related tools. ETL Tools: Experience with Apache NiFi, Talend, Airflow, etc. Data Modeling and Warehousing: Knowledge of data modeling and warehousing solutions like Redshift, BigQuery, Snowflake. Data Governance and Security: Understanding of data quality and security best practices. Cloud Computing: Experience with AWS, Azure, Google Cloud and their data services. Streaming Data Processing: Knowledge of Kafka, Flink, and real-time data processing frameworks. KPIs:
Data Pipeline Efficiency Data Quality Metrics System Uptime and Availability Data Storage Efficiency Data Security and Compliance Data Processing Performance Scalability and Performance Tuning Resource Utilization and Cost Optimization Incident Response and Resolution Documentation and Knowledge Sharing Experience: 3-5 years in similar roles. Career progression to CTO WGDT (Head of Incubation Centre). WF is a global non-profit working in a fast-paced, innovative environment emphasizing Quality, Speed, and Scale. We value excellence, creativity, collaboration, and societal impact, fostering an inclusive culture that promotes continuous learning and adaptation. We are an equal opportunity organization committed to diversity and inclusion. #J-18808-Ljbffr