City and County of San Francisco
Senior Data Engineer (1043) - DataSF
City and County of San Francisco, San Francisco, California, United States, 94199
Apply using SmartRecruiters, the City and County of San Francisco's application portal.
12/24/25 - Amended to include updated filing deadline
Department:
DataSF Job class:
1043 Senior Data Engineer Salary range:
$153,686/year - $193,388/year Role type:
Permanent Exempt Hours:
Full-time (Hybrid work schedule)
About:
Permanent Exempt: This is an Exempt position excluded by the Charter from the competitive Civil Service examination process, pursuant to the City and County of San Francisco, Charter Section 10.104. It is considered "at will" and shall serve at the discretion of the Appointing Officer.
Application Opening:
Wednesday, November 19, 2025 Application Closing: Interested candidates are encouraged to apply as soon as possible, as this job announcement will close at any time, but not earlier than
Friday, January 9, 2026 at 11:59pm
The Office of the City Administrator
and its 25+ divisions and departments operate core internal and public-facing services in San Francisco.
Mission and Vision Our vision is to lead the nation in public administration and to enable City departments to effectively deliver critical public services. We aim to help the city run better, connect residents to the vital public services they seek, and create a meaningful and diverse work culture that is the place of choice for people invested in public service.
We are committed to ensuring that City services are inclusive, efficient, equitable, and culturally competent for all residents.
Want to build robust and scalable data infrastructure to power data-driven City services? Join the DataSF team to empower and expand the use of data in government!
Company Description
DataSF’s mission is to transform the way San Francisco works with data. We believe that data, when harnessed effectively, can improve transparency, resident engagement, and government performance. We actively seek team members whose diverse life experiences reflect the residents we serve.
DataSF is seeking a Senior Data Engineer with 3+ years of experience to join our growing team. Reporting to the Principal Data Engineer, you will design, build, and maintain the City’s data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs.
Responsibilities
Platform Administration : Manage the central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization.
Pipeline Development : Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources, supporting streaming and batch processes.
Infrastructure as Code (IaC) : Use Terraform to define, deploy, and manage data infrastructure, ensuring pipelines are reproducible, version-controlled, and production-ready.
Best Practices & Innovation : Champion best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing. Provide guidance to department partners on data engineering best practices.
Collaboration : Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders to understand data requirements and build solutions that meet their needs.
Monitoring & Support : Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.
Desirable Qualifications
Technical Knowledge
Hands‑on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
Expertise in writing advanced, performant SQL and using tools like dbt for SQL‑based data transformation and modeling.
Strong programming skills in Python with pandas, PySpark, or similar libraries for data processing and automation.
Proficiency with Terraform and an Infrastructure as Code tool.
Experience building and deploying data pipelines using orchestration tools such as Airflow, Dagster, or similar technologies.
Deep understanding of data warehousing concepts, data modeling, and modern ELT principles.
Understanding of data governance, data security, and data privacy principles.
Experience with real‑time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe).
Experience deploying and managing data pipelines for machine learning models.
Collaboration and Communication Skills
Strong problem‑solving skills with ability to design practical and effective data solutions.
Excellent verbal and written communication skills, including the ability to explain technical concepts to non‑technical stakeholders.
A collaborative mindset with enthusiasm to work across diverse, cross‑functional teams.
Mission Alignment
Commitment to equity, transparency, and ethical data use.
Passion for public service and using data to improve government services.
Empathy for San Francisco’s diverse communities and a drive to make data and services more accessible for SF residents.
Interest or experience in public sector data or social impact work.
How to qualify Education : Associate degree in computer science, computer engineering, information systems, or a closely related field from an accredited college or university, or its equivalent in terms of total course credits/units (at least 60 semester or 90 quarter credits, with a minimum of 20 semester or 30 quarter credits in one of the fields above).
Experience : Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.
Substitution : Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/forty‑five (45) quarter units with a minimum of 10 semester / 15 quarter units in one of the fields above.
Verification : Applicants must submit verification of qualifying education and experience during the recruitment and selection process. For information on how to verify experience and/or education requirements, visit Verification of Experience and/or Education.
Note : Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.
Selection Procedures The selection process will include evaluation of applications in relation to minimum requirements and assessment of candidates’ job‑related knowledge, skills and abilities. Depending on the number of applicants, the Department may establish and implement additional screening mechanisms to evaluate candidate qualifications. Typically, this includes an oral interview and/or a written or performance exercise.
Applicants meeting the minimum requirements are not guaranteed advancement in the selection process.
EEO Statement All your information will be kept confidential according to EEO guidelines. The City and County of San Francisco encourages women, minorities, and persons with disabilities to apply. Applicants will be considered regardless of protected categories under applicable laws.
#J-18808-Ljbffr
12/24/25 - Amended to include updated filing deadline
Department:
DataSF Job class:
1043 Senior Data Engineer Salary range:
$153,686/year - $193,388/year Role type:
Permanent Exempt Hours:
Full-time (Hybrid work schedule)
About:
Permanent Exempt: This is an Exempt position excluded by the Charter from the competitive Civil Service examination process, pursuant to the City and County of San Francisco, Charter Section 10.104. It is considered "at will" and shall serve at the discretion of the Appointing Officer.
Application Opening:
Wednesday, November 19, 2025 Application Closing: Interested candidates are encouraged to apply as soon as possible, as this job announcement will close at any time, but not earlier than
Friday, January 9, 2026 at 11:59pm
The Office of the City Administrator
and its 25+ divisions and departments operate core internal and public-facing services in San Francisco.
Mission and Vision Our vision is to lead the nation in public administration and to enable City departments to effectively deliver critical public services. We aim to help the city run better, connect residents to the vital public services they seek, and create a meaningful and diverse work culture that is the place of choice for people invested in public service.
We are committed to ensuring that City services are inclusive, efficient, equitable, and culturally competent for all residents.
Want to build robust and scalable data infrastructure to power data-driven City services? Join the DataSF team to empower and expand the use of data in government!
Company Description
DataSF’s mission is to transform the way San Francisco works with data. We believe that data, when harnessed effectively, can improve transparency, resident engagement, and government performance. We actively seek team members whose diverse life experiences reflect the residents we serve.
DataSF is seeking a Senior Data Engineer with 3+ years of experience to join our growing team. Reporting to the Principal Data Engineer, you will design, build, and maintain the City’s data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs.
Responsibilities
Platform Administration : Manage the central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization.
Pipeline Development : Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources, supporting streaming and batch processes.
Infrastructure as Code (IaC) : Use Terraform to define, deploy, and manage data infrastructure, ensuring pipelines are reproducible, version-controlled, and production-ready.
Best Practices & Innovation : Champion best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing. Provide guidance to department partners on data engineering best practices.
Collaboration : Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders to understand data requirements and build solutions that meet their needs.
Monitoring & Support : Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.
Desirable Qualifications
Technical Knowledge
Hands‑on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
Expertise in writing advanced, performant SQL and using tools like dbt for SQL‑based data transformation and modeling.
Strong programming skills in Python with pandas, PySpark, or similar libraries for data processing and automation.
Proficiency with Terraform and an Infrastructure as Code tool.
Experience building and deploying data pipelines using orchestration tools such as Airflow, Dagster, or similar technologies.
Deep understanding of data warehousing concepts, data modeling, and modern ELT principles.
Understanding of data governance, data security, and data privacy principles.
Experience with real‑time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe).
Experience deploying and managing data pipelines for machine learning models.
Collaboration and Communication Skills
Strong problem‑solving skills with ability to design practical and effective data solutions.
Excellent verbal and written communication skills, including the ability to explain technical concepts to non‑technical stakeholders.
A collaborative mindset with enthusiasm to work across diverse, cross‑functional teams.
Mission Alignment
Commitment to equity, transparency, and ethical data use.
Passion for public service and using data to improve government services.
Empathy for San Francisco’s diverse communities and a drive to make data and services more accessible for SF residents.
Interest or experience in public sector data or social impact work.
How to qualify Education : Associate degree in computer science, computer engineering, information systems, or a closely related field from an accredited college or university, or its equivalent in terms of total course credits/units (at least 60 semester or 90 quarter credits, with a minimum of 20 semester or 30 quarter credits in one of the fields above).
Experience : Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.
Substitution : Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/forty‑five (45) quarter units with a minimum of 10 semester / 15 quarter units in one of the fields above.
Verification : Applicants must submit verification of qualifying education and experience during the recruitment and selection process. For information on how to verify experience and/or education requirements, visit Verification of Experience and/or Education.
Note : Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.
Selection Procedures The selection process will include evaluation of applications in relation to minimum requirements and assessment of candidates’ job‑related knowledge, skills and abilities. Depending on the number of applicants, the Department may establish and implement additional screening mechanisms to evaluate candidate qualifications. Typically, this includes an oral interview and/or a written or performance exercise.
Applicants meeting the minimum requirements are not guaranteed advancement in the selection process.
EEO Statement All your information will be kept confidential according to EEO guidelines. The City and County of San Francisco encourages women, minorities, and persons with disabilities to apply. Applicants will be considered regardless of protected categories under applicable laws.
#J-18808-Ljbffr