In this role, you will assist the threat detection team at Databricks in building threat detection pipelines (log analytics for security), using PySpark, SQL, and Databricks.
You will help get these pipelines deployed over the terraform.
For this specific contract, we are looking for someone to help us with platform migration involving migration of detection rules and content from a legacy platform and into databrick's native internal detection framework.
This will involve creating equivalent PySpark log pipelines, rule configuration files, writing unit tests, performing data validation checks, and then deploying into production.