Systems Engineer/Senior Data Engineer - Splunk, ServiceNow & AppDynamics

Job DescriptionJob Descriptionn

***Security Clearance: Must possess an active TS/SCI with Full Scope Polygraph***

KDA is seeking a highly skilled and experienced Systems Engineer/Senior Data Engineer with a strong background in designing, implementing, and optimizing data pipelines and solutions for critical security and operational platforms, specifically Splunk, ServiceNow, and AppDynamics. The ideal candidate will be a proactive, problem-solving individual with a proven track record of managing complex data ingestion, normalization, and correlation processes within highly visible, operationally driven environments. This role requires an individual who thrives in a collaborative setting, possesses excellent communication skills, and is dedicated to continuous improvement and innovation.

Key Responsibilities:

n
• Splunk Data Engineering:
n
n
• Design, engineer, and maintain robust Splunk infrastructures, including clustered environments, for large-scale data ingestion, correlation, and reporting.
n
• Automate complex data ingestion methods (e.g., S3, syslog, JSON, APIs) from diverse sources across multiple enclaves.
n
• Develop and implement methods for data tagging and cataloging to ensure compliance with evolving security standards and facilitate efficient data discovery.
n
• Optimize data ingest performance and efficiency across various network environments. Familiar with Technical Add-ons.
n
• Parse and normalize non-standard data sets to enable comprehensive analysis and correlation within Splunk.
n
• Develop and refine Splunk queries, dashboards, and reports to visualize security events, infrastructure health, and operational metrics.
n
• Collaborate with IT operations and cyber security teams to enrich data sets, ascertain cyber threats, and bolster security posture.
n
• Maintain ITSI and SIEM-like tools and custom content within virtualized environments.
n
• Perform tuning and filtering of events and information, creating custom views and content.
n
• Familiar with UBA and Splunk.
n
• Collaborate with cross-functional teams to design and implement data integrations between various security and operational tools (including Splunk and AppDynamics) and ServiceNow.
n
n
n
• Develop and maintain data pipelines to ensure accurate and timely flow of security incidents, alerts, and operational metrics into ServiceNow for incident management, problem management, and reporting.
n
• Assist in defining and implementing data models within ServiceNow to support security operations and compliance initiatives.
n
• Work with third-party services for design review and optimal deployment configuration for enterprise cloud service utilization (relevant to integrations).
n
n
n
• Design and implement data collection strategies for AppDynamics, ensuring comprehensive monitoring of application performance and infrastructure.
n
• Integrate AppDynamics data with Splunk for centralized visibility and correlation with other security and operational logs.
n
• Troubleshoot problematic service deployments and data flows, utilizing forensic tools and audit log review (relevant to monitoring and analysis).
n
• Develop methods to leverage AppDynamics data for identifying potential risks and optimizing application performance.
n
n
• General Data Engineering & Systems Expertise:
n
n
• Collaborate with mission partners to develop long-term enterprise audit solutions and normalize non-standard data sets.
n
• Engineer and maintain secure virtualized and cloud environments for data platforms.
n
• Deploy and harden servers running Linux OS in accordance with CIS and other STIG guidelines.
n
• Develop runbooks, SOPs, and documentation for new processes and systems.
n
• Perform liaison duties between service providers and clients to bridge communication gaps and ensure adherence to SLAs.
n
• Review and evaluate data integrity and develop use cases for various data sets.
n
• Maintain system baselines and configuration management for data engineering tools.
n
• Contribute to the development of plans to safeguard data against unauthorized modification, destruction, or disclosure.
n
• Strong understanding of cyber security principles and experience with various security tools (e.g., Next-Gen Firewalls, IPS/IDS, Tenable Nessus, Rapid7 Nexpose, McAfee EPO, Symantec SEP).
n
n

Required Skills & Experience:

n
• 20+ years of progressive experience in Information Technology and Security, with a strong focus on data engineering and systems integration.
n
• Security Clearance: Candidate must possess an active TS/SCI with Full Scope Polygraph
n
• Demonstrated expertise in engineering and maintaining large-scale Splunk environments, including data ingestion, parsing, normalization, and content development.
n
• Experience with automating complex data ingestion methods (e.g., S3, syslog, JSON, APIs).
n
• Strong understanding of data tagging, cataloging, and data governance best practices.
n
• Proficiency with Linux OS administration and hardening.
n
• Familiarity with cloud security principles and deploying commercial services into protected/secured enclaves (e.g., AWS).
n
• Experience with SIEM solutions and their implementation, configuration, and maintenance.
n
• Strong scripting skills (e.g., BASH, Python, PowerShell).
n
• Excellent collaboration and communication skills, with the ability to work effectively in small teams and large collaborative efforts.
n
• Ability to troubleshoot complex technical issues and perform root cause analysis.
n
• Proven ability to develop and maintain documentation (runbooks, SOPs).
n

Desired Skills (Plus, but not required):

n
• Experience with ServiceNow platform administration, development, or integration.
n
• Experience with AppDynamics for application performance monitoring and data collection.
n
• Experience with configuration management tools - Git, Ansible
n