Pyramid Technology Solutions, Inc.
Position: Azure Data Engineer
Location: Weehawken, NJ
Duration: 12+Months with possible extensions
Main Skill: Data Software Engineering
Skill Spec: DSE Python Azure Databricks
Must have skills: Apache Spark, Azure Data Factory, Azure SQL Microsoft, Azure Synapse Analytics, ETL/ELT Solutions
Nice to have skills: Apache Kafka, CI/CD and Python
Your Expertise:
- Expert level skills writing and optimizing complex SQL
- Experience with complex data modeling, ETL design, and using large databases in a business environment
- Experience with building data pipelines and applications to stream and process datasets at low latencies
- Fluent with Big Data technologies like Spark, Kafka and Hive
- Expert level understanding of Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service
- Designing and building data pipelines using API ingestion and Streaming ingestion methods
- Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code
- Experience in developing NoSQL solutions using Azure Cosmos DB
- Thorough understanding of Azure and AWS Cloud Infrastructure offerings
- Working knowledge of Python is desirable
Key Responsibilities:
- Design and implement scalable and secure data processing pipelines using Azure Data Factory, Azure Databricks, and other Azure services
- Manage and optimize data storage using Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB
- Monitor and troubleshoot data-related issues within the Azure environment to maintain high availability and performance
- Implement data security measures, including encryption, access controls, and auditing, to protect sensitive information
- Automate data pipelines and workflows to streamline data ingestion, processing, and distribution tasks
- Utilize Azure’s analytics services, such as Azure Synapse Analytics, to provide insights and support data-driven decision-making
- Document data procedures, systems, and architectures to maintain clarity and ensure compliance with regulatory standards
- Provide guidance and support for data governance, including metadata management, data lineage, and data cataloging