Role: Data Engineer
Location: UK Remote
Duration: 6 Months
Day rate: £470 inside IR35
Active SC clearance required
Role Description:
- Strong understanding of data concepts - data types, data structures,schemas (bothJSON and Spark), schema management etc
- Strong understanding of complex JSON manipulation
- Experience working with Data Pipelines using a custom Python/PySpark frameworks
- Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
- Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc including experience of handling sensitive datasets
- Strong problem solving and analytical skills, particularly able to demonstrate these intuitively
Required skills:
Languages / Frameworks - JSON - YAML - Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.) - SQL - PySpark - Delta Lake - Bash (both CLI usage and scripting) - Git - Markdown - Scala (bonus, not compulsory) - Azure SQL Server as a HIVE Metastore (bonus) Technologies - Azure Databricks - Apache Spark - Delta Tables - Data processing with Python - PowerBI (Integration / Data Ingestion) - JIRA
