San Francisco
Senior Data Engineer (1043) - DataSF
San Francisco, San Francisco, California, United States, 94199
Company Overview
DataSF is a part of the Office of the City Administrator and operates core internal and public‑facing services in San Francisco. The mission is to transform how the city works with data, improving transparency, resident engagement, and government performance.
Position Summary Title:
Senior Data Engineer (Job class: 1043)
Type:
Permanent Exempt
Salary:
$153,686 – $193,388 per year
Hours:
Full‑time (Hybrid schedule)
Application Opening:
Wednesday, November 19, 2025
Application Closing:
Friday, January 9, 2026 at 11:59 pm (no earlier than this date)
Applicants must also complete a supplemental questionnaire by the filing deadline: https://forms.office.com/g/Dh3S0377ZV.
Job Description DataSF seeks a Senior Data Engineer with 3+ years of experience, reporting to the Principal Data Engineer. The role is responsible for designing, building, and maintaining the City’s data infrastructure, creating robust data pipelines and reliable data access for analytical and operational needs. The position applies advanced data engineering techniques to complex urban challenges, supporting the City’s commitment to efficient, equitable, and ethical service delivery.
Essential Duties
Platform Administration:
Manage the central Snowflake data warehouse – including access control, security policies, resource monitoring, performance tuning, and cost optimization – while promoting data democratization and protecting privacy.
Pipeline Development:
Build and maintain scalable, resilient pipelines for ingesting and structuring data from diverse sources, supporting both streaming and batch processes, and handling structured and unstructured data.
Infrastructure as Code (IaC):
Use Terraform to define, deploy, and manage data infrastructure, ensuring pipelines are reproducible, version‑controlled, and production‑ready.
Best Practices & Innovation:
Champion documentation, data modeling, warehouse architecture, SQL optimization, and testing. Drive creative improvements to platform capabilities and efficiency, and guide department partners on data engineering best practices.
Collaboration:
Work closely with data scientists, analysts, product managers, software engineers, and non‑technical stakeholders to understand data requirements and build solutions.
Monitoring & Support:
Proactively monitor platform and pipeline health, troubleshoot issues, and uphold high standards of data quality and availability.
Desirable Qualifications
Technical Knowledge:
Experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
Strong SQL skills, including performance tuning, and familiarity with dbt for data transformation.
Proficiency in Python (pandas, PySpark) for data processing and automation.
Experience with Terraform and orchestration tools like Azure Data Factory, Airflow, or Dagster.
Solid understanding of data warehousing, data modeling, and ELT principles.
Knowledge of data governance, security, and privacy.
Experience with real‑time streaming (Kafka, Kinesis, Snowpipe) and deploying pipelines for machine learning models.
Collaboration & Communication:
Strong problem‑solving skills and the ability to design practical data solutions.
Excellent verbal and written communication, including explaining technical concepts to non‑technical stakeholders.
Collaborative mindset with enthusiasm for cross‑functional work.
Mission Alignment:
Commitment to equity, transparency, and ethical data use.
Passion for public service and using data to improve government services.
Empathy for San Francisco’s diverse communities and a drive to make data and services more accessible to residents.
Interest or experience in public‑sector data or social impact work.
Qualifications Education:
Associate degree in computer science, engineering, information systems, or a closely related field, or equivalent in credit units (minimum 60 semester or 90 quarter credits, with at least 20 semester or 30 quarter credits in a related field).
Experience:
Three years of experience analyzing, installing, configuring, enhancing, and/or maintaining enterprise network components.
Substitution:
Up to two years of additional experience may replace the required degree. One year of experience equals 30 semester units or 45 quarter units, with a minimum of 10 semester or 15 quarter units in a related field.
Completion of the 1010 Information Systems Trainee Program may substitute for the required degree.
Verification Applicants must clearly demonstrate how they meet the minimum qualifications. Verification of education and experience will be required during the recruitment process. For more information on verifying foreign credentials or degree equivalency: Verification of Experience and/or Education.
Additional Information Selection Procedures:
The process includes evaluating applications against minimum requirements and assessing job‑related knowledge, skills, and abilities. Depending on the number of applicants, the department may use additional screening mechanisms, such as written or performance exercises, and oral interviews. Only those whose qualifications most closely meet the Department’s needs will be invited to continue.
Related to this classification: Position Counts by Job Codes and Department FY 2022‑23 (https://sfdhr.org/sites/default/files/documents/Forms-Documents/Position-Counts-by-Job-Codes-and-Department-FY-2022-23.pdf).
Benefits Overview, Equal Employment Opportunity, ADA Accommodation, Veteran’s Preference, Right to Work, etc.
– Visit the City’s job portal for details.
How to Apply All applications are accepted only through the online process at https://careers.sf.gov. Applicants must apply as soon as possible, but no earlier than Friday, January 9, 2026 (11:59 PM PST). The process includes:
Select the “Apply Now” button and follow screen instructions.
Include a cover letter.
Complete the supplemental questionnaire: https://forms.office.com/g/Dh3S0377ZV.
Follow best‑practice guidelines at Apply for Jobs in the City and County of San Francisco Best Practices Guide.
Applicants may be contacted via email regarding this announcement; ensure your registered email address is accurate and not blocked by spam filters. Names of accepted senders: @sfgov.org, @sfdpw.org, @sfport.com, @flysfo.com, @sfwater.org, @sfdph.org, @asianart.org, @sfmta.com, @sfpl.org, @dcyf.org, @first5sf.org, @famsf.org, @ccsf.edu, @smartalerts.info, @smartrecruiters.com.
Confirmation email will be sent once your application is received. Retain this email; failure to receive one indicates the application was not submitted or received.
All information is kept confidential according to EEO guidelines.
For questions about recruitment or the application process, contact Debbie Yung at Debbie.Yung@SFGov.org.
Equality and Inclusion Statement The City and County of San Francisco encourages women, minorities, and persons with disabilities to apply. Applicants will be considered regardless of sex, race, age, religion, color, national origin, ancestry, physical or mental disability, medical condition (including cancer), HIV/AIDS status, genetic information, marital status, sexual orientation, gender identity, gender expression, military or veteran status, or any other protected category under the law.
#J-18808-Ljbffr
Position Summary Title:
Senior Data Engineer (Job class: 1043)
Type:
Permanent Exempt
Salary:
$153,686 – $193,388 per year
Hours:
Full‑time (Hybrid schedule)
Application Opening:
Wednesday, November 19, 2025
Application Closing:
Friday, January 9, 2026 at 11:59 pm (no earlier than this date)
Applicants must also complete a supplemental questionnaire by the filing deadline: https://forms.office.com/g/Dh3S0377ZV.
Job Description DataSF seeks a Senior Data Engineer with 3+ years of experience, reporting to the Principal Data Engineer. The role is responsible for designing, building, and maintaining the City’s data infrastructure, creating robust data pipelines and reliable data access for analytical and operational needs. The position applies advanced data engineering techniques to complex urban challenges, supporting the City’s commitment to efficient, equitable, and ethical service delivery.
Essential Duties
Platform Administration:
Manage the central Snowflake data warehouse – including access control, security policies, resource monitoring, performance tuning, and cost optimization – while promoting data democratization and protecting privacy.
Pipeline Development:
Build and maintain scalable, resilient pipelines for ingesting and structuring data from diverse sources, supporting both streaming and batch processes, and handling structured and unstructured data.
Infrastructure as Code (IaC):
Use Terraform to define, deploy, and manage data infrastructure, ensuring pipelines are reproducible, version‑controlled, and production‑ready.
Best Practices & Innovation:
Champion documentation, data modeling, warehouse architecture, SQL optimization, and testing. Drive creative improvements to platform capabilities and efficiency, and guide department partners on data engineering best practices.
Collaboration:
Work closely with data scientists, analysts, product managers, software engineers, and non‑technical stakeholders to understand data requirements and build solutions.
Monitoring & Support:
Proactively monitor platform and pipeline health, troubleshoot issues, and uphold high standards of data quality and availability.
Desirable Qualifications
Technical Knowledge:
Experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
Strong SQL skills, including performance tuning, and familiarity with dbt for data transformation.
Proficiency in Python (pandas, PySpark) for data processing and automation.
Experience with Terraform and orchestration tools like Azure Data Factory, Airflow, or Dagster.
Solid understanding of data warehousing, data modeling, and ELT principles.
Knowledge of data governance, security, and privacy.
Experience with real‑time streaming (Kafka, Kinesis, Snowpipe) and deploying pipelines for machine learning models.
Collaboration & Communication:
Strong problem‑solving skills and the ability to design practical data solutions.
Excellent verbal and written communication, including explaining technical concepts to non‑technical stakeholders.
Collaborative mindset with enthusiasm for cross‑functional work.
Mission Alignment:
Commitment to equity, transparency, and ethical data use.
Passion for public service and using data to improve government services.
Empathy for San Francisco’s diverse communities and a drive to make data and services more accessible to residents.
Interest or experience in public‑sector data or social impact work.
Qualifications Education:
Associate degree in computer science, engineering, information systems, or a closely related field, or equivalent in credit units (minimum 60 semester or 90 quarter credits, with at least 20 semester or 30 quarter credits in a related field).
Experience:
Three years of experience analyzing, installing, configuring, enhancing, and/or maintaining enterprise network components.
Substitution:
Up to two years of additional experience may replace the required degree. One year of experience equals 30 semester units or 45 quarter units, with a minimum of 10 semester or 15 quarter units in a related field.
Completion of the 1010 Information Systems Trainee Program may substitute for the required degree.
Verification Applicants must clearly demonstrate how they meet the minimum qualifications. Verification of education and experience will be required during the recruitment process. For more information on verifying foreign credentials or degree equivalency: Verification of Experience and/or Education.
Additional Information Selection Procedures:
The process includes evaluating applications against minimum requirements and assessing job‑related knowledge, skills, and abilities. Depending on the number of applicants, the department may use additional screening mechanisms, such as written or performance exercises, and oral interviews. Only those whose qualifications most closely meet the Department’s needs will be invited to continue.
Related to this classification: Position Counts by Job Codes and Department FY 2022‑23 (https://sfdhr.org/sites/default/files/documents/Forms-Documents/Position-Counts-by-Job-Codes-and-Department-FY-2022-23.pdf).
Benefits Overview, Equal Employment Opportunity, ADA Accommodation, Veteran’s Preference, Right to Work, etc.
– Visit the City’s job portal for details.
How to Apply All applications are accepted only through the online process at https://careers.sf.gov. Applicants must apply as soon as possible, but no earlier than Friday, January 9, 2026 (11:59 PM PST). The process includes:
Select the “Apply Now” button and follow screen instructions.
Include a cover letter.
Complete the supplemental questionnaire: https://forms.office.com/g/Dh3S0377ZV.
Follow best‑practice guidelines at Apply for Jobs in the City and County of San Francisco Best Practices Guide.
Applicants may be contacted via email regarding this announcement; ensure your registered email address is accurate and not blocked by spam filters. Names of accepted senders: @sfgov.org, @sfdpw.org, @sfport.com, @flysfo.com, @sfwater.org, @sfdph.org, @asianart.org, @sfmta.com, @sfpl.org, @dcyf.org, @first5sf.org, @famsf.org, @ccsf.edu, @smartalerts.info, @smartrecruiters.com.
Confirmation email will be sent once your application is received. Retain this email; failure to receive one indicates the application was not submitted or received.
All information is kept confidential according to EEO guidelines.
For questions about recruitment or the application process, contact Debbie Yung at Debbie.Yung@SFGov.org.
Equality and Inclusion Statement The City and County of San Francisco encourages women, minorities, and persons with disabilities to apply. Applicants will be considered regardless of sex, race, age, religion, color, national origin, ancestry, physical or mental disability, medical condition (including cancer), HIV/AIDS status, genetic information, marital status, sexual orientation, gender identity, gender expression, military or veteran status, or any other protected category under the law.
#J-18808-Ljbffr