Obsidian Works LLC.
Computer Systems Engineer
Obsidian Works LLC. - Virginia, Minnesota, United States, 55792
Work at Obsidian Works LLC.
Overview
- View job
Overview
We are seeking a highly skilled Subject Matter Expert in data engineering, with extensive experience in big data computational efforts and leveraging Amazon Web Services (AWS) technologies. The ideal candidate will have deep expertise in using EC2 compute instances, Graviton processors, and Photon vector processing, as well as proficiency in Spark Scala, PySpark, Python, and SQL.
This role involves developing advanced data pipelines, leading complex analytical initiatives, and mentoring team members and government partners in big data and advanced analytics. Active TS/SCI + CI Poly is required for access to classified environments and systems. Requirements
Subject Matter Expert in the use of Spark and the Hadoop distributed processing framework. Expert knowledge in utilizing different EC2 compute instance types for in-memory, Delta cache, and general-purpose use as it applies to big data computational efforts. Experience in the use of Graviton processors in a Spark environment with Photon vector processing, and applying Hilbert Curves/Z-Ordering for Delta Table join efficiencies is desired. Experience in working with Amazon Web Services, utilizing EC2, S3, IAM, and user roles that are compliant with ODS and ECDS requirements. Experience in the use of approved diodes for architecting data throughput. Experience in developing with Spark Scala, PySpark, Python, Spark SQL, and SQL. Experience in the development of tools, algorithms, and building big data pipelines for analysis and visualization. Experience in spearheading complex analytical initiatives and guiding teams in leveraging mathematical, statistical, and big data methods to solve critical agency challenges characterized by high ambiguity and complexity. Team-oriented, technically proficient intelligence professionals with a proven track record of leveraging commercial data to address key intelligence questions. Ability to mentor and guide analysts and other government partners on the use of big data and advanced analytics. Experience with/commercial application of location data types. Experience working with Databricks Runtime 13.3 LTS (or later) and Spark Livy 3.10 (or later). Experience in identifying phenomena indicative of spatiotemporal data. Experience in evaluating commercial datasets to assess coverage, accuracy, fill rates, and identify potentially fraudulent activity. Active TS/SCI + CI Poly required. Must live in or be willing to relocate to the Washington DC, Virginia, or Maryland areas. About the Company
Obsidian specializes in technology development and analytic support to global national security objectives. Our dedicated team - rooted in practical national security expertise - is committed to excellence, innovation, and partnershipqualities that make us not just service providers but mission-critical partners to our clients. #J-18808-Ljbffr
This role involves developing advanced data pipelines, leading complex analytical initiatives, and mentoring team members and government partners in big data and advanced analytics. Active TS/SCI + CI Poly is required for access to classified environments and systems. Requirements
Subject Matter Expert in the use of Spark and the Hadoop distributed processing framework. Expert knowledge in utilizing different EC2 compute instance types for in-memory, Delta cache, and general-purpose use as it applies to big data computational efforts. Experience in the use of Graviton processors in a Spark environment with Photon vector processing, and applying Hilbert Curves/Z-Ordering for Delta Table join efficiencies is desired. Experience in working with Amazon Web Services, utilizing EC2, S3, IAM, and user roles that are compliant with ODS and ECDS requirements. Experience in the use of approved diodes for architecting data throughput. Experience in developing with Spark Scala, PySpark, Python, Spark SQL, and SQL. Experience in the development of tools, algorithms, and building big data pipelines for analysis and visualization. Experience in spearheading complex analytical initiatives and guiding teams in leveraging mathematical, statistical, and big data methods to solve critical agency challenges characterized by high ambiguity and complexity. Team-oriented, technically proficient intelligence professionals with a proven track record of leveraging commercial data to address key intelligence questions. Ability to mentor and guide analysts and other government partners on the use of big data and advanced analytics. Experience with/commercial application of location data types. Experience working with Databricks Runtime 13.3 LTS (or later) and Spark Livy 3.10 (or later). Experience in identifying phenomena indicative of spatiotemporal data. Experience in evaluating commercial datasets to assess coverage, accuracy, fill rates, and identify potentially fraudulent activity. Active TS/SCI + CI Poly required. Must live in or be willing to relocate to the Washington DC, Virginia, or Maryland areas. About the Company
Obsidian specializes in technology development and analytic support to global national security objectives. Our dedicated team - rooted in practical national security expertise - is committed to excellence, innovation, and partnershipqualities that make us not just service providers but mission-critical partners to our clients. #J-18808-Ljbffr