Data Security and Governance Guide for Data Engineers

Data security in data engineering is not optional. IBM’s 2024 Cost of a Data Breach report found that the average cost of a data breach reached $4.88 million, with healthcare breaches averaging over $10 million. Regulations like GDPR, HIPAA, SOC 2, and CCPA impose legal requirements for protecting sensitive data, with non-compliance penalties reaching 4% of global annual revenue for GDPR violations. Building secure data pipelines requires understanding encryption, access control, auditing, compliance frameworks, and incident response.

Encryption: Protecting Data at Rest and in Transit

Encryption at Rest

Data stored on disk must be encrypted to protect against unauthorized physical access, misconfigured permissions, and compromised storage media. Modern cloud storage services provide server-side encryption transparently, with minimal performance impact. The AWS Key Management Service documentation emphasizes that encryption at rest should be enabled by default for all storage services, with automatic key rotation managed through the KMS infrastructure.

Key management is critical to encryption. Use a dedicated key management service like AWS KMS, HashiCorp Vault, or Azure Key Vault. Never embed keys in code, configuration files, or environment variables. Implement key rotation schedules (annually or upon compromise), restrict access to key material to a minimal set of administrators, and enable key usage auditing to detect unauthorized key access.

Encryption in Transit

All data moving between systems must be encrypted with TLS 1.2 or higher. For service-to-service communication within a VPC, mutual TLS (mTLS) provides two-way authentication, ensuring both the client and server verify each other’s identity. For legacy systems that do not support TLS, use SSH tunneling or a VPN to establish an encrypted connection. The NIST SP 800-52 guidelines provide comprehensive recommendations for TLS configuration in government and enterprise environments.

Access Control

Principle of Least Privilege

Every user and service should have the minimum permissions needed to perform their function. AWS IAM policies, GCP IAM roles, and Azure RBAC provide fine-grained access control at the resource level. For data engineering, this means scoping permissions to specific S3 prefixes, database schemas, and tables rather than granting blanket access.

{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::data-bronze/*", "arn:aws:s3:::data-bronze"],
    "Condition": {
        "StringEquals": {"aws:SourceIp": "10.0.0.0/8"}
    }
---

Implement permission boundaries and service control policies (SCPs in AWS) to prevent privilege escalation even by administrators. Regularly review and revoke unused permissions through automated access review tools.

Role-Based Access Control

A typical RBAC model for a data platform defines roles based on job function and data access requirements:

Roles should be reviewed quarterly to ensure they remain appropriate as team members change responsibilities. Automated provisioning and de-provisioning through identity providers (Okta, Azure AD) reduces the risk of orphaned accounts with unnecessary access.

Column-Level Security

Sensitive columns containing PII, financial data, or health information require restricted access beyond table-level permissions. Most cloud warehouses support column-level security through views, row-level security, and column masking policies:

CREATE VIEW orders_public AS
SELECT
    order_id,
    order_date,
    amount,
    MASK(customer_email, '***@***') AS email,
    'REDACTED' AS ssn
FROM orders;

Column-level security policies should be defined centrally and applied consistently across all data products. Use data classification tags (public, internal, confidential, restricted) to automate policy application based on sensitivity level.

Data Masking

Non-production environments must never contain real sensitive data. Data masking transforms sensitive information while preserving data utility for development, testing, and analytics:

Dynamic data masking applies rules at query time based on the user’s role, eliminating the need to maintain multiple copies of the dataset. For example, a customer support agent might see the last four digits of a credit card number, while a data analyst sees only a masked version.

Audit Logging

Audit logs provide a record of who accessed what data, when, and from where. They are essential for security investigations, compliance audits, and detecting anomalous access patterns. Cloud providers offer managed audit logging through services like AWS CloudTrail, GCP Audit Logs, and Azure Monitor:

SELECT
    user_identity,
    event_time,
    action,
    resource,
    source_ip
FROM system.audit_log
WHERE event_time >= CURRENT_DATE - 7
  AND resource LIKE '%customer%'
ORDER BY event_time DESC;

Audit Categories

Audit logs must be stored in immutable, append-only storage to prevent tampering. AWS S3 Object Lock and similar features provide write-once-read-many (WORM) storage that satisfies regulatory requirements for log integrity.

Compliance Frameworks

GDPR (Europe)

The General Data Protection Regulation imposes strict requirements on organizations processing EU residents’ data. Key obligations include: right to be forgotten (erasure of personal data on request within 30 days), data portability (export in machine-readable format), consent tracking with granular opt-in, data protection impact assessments (DPIAs) for high-risk processing, and 72-hour breach notification to supervisory authorities.

For data engineering, GDPR compliance requires implementing data deletion pipelines that can identify and remove all traces of a user’s personal data across the data platform, including backups and archived data. Automated data classification tools help identify PII across large datasets.

HIPAA (US Healthcare)

The Health Insurance Portability and Accountability Act requires: encryption of all Protected Health Information (PHI) both at rest and in transit, 6+ years of access log retention, Business Associate Agreements (BAAs) with all vendors handling PHI, documented data disposal procedures, and annual security risk assessments.

Data engineers handling healthcare data must ensure that all pipelines processing PHI operate within HIPAA-compliant environments. Cloud providers offer HIPAA-eligible services with BAAs, but the responsibility for proper configuration and access controls rests with the organization.

SOC 2

SOC 2 certification demonstrates that a service organization meets five trust service criteria: security, availability, processing integrity, confidentiality, and privacy. The security principle requires protections against unauthorized access through firewalls, intrusion detection, and access controls. SOC 2 Type II reports validate that controls are operating effectively over a period of time.

Security Incident Response

A documented incident response plan ensures rapid, coordinated action when a breach occurs. The NIST Computer Security Incident Handling Guide (SP 800-61) provides a framework that most organizations adapt:

1. Detect: Alert on unusual access patterns, failed authentication attempts, data export anomalies, or security tool alarms
2. Contain: Revoke compromised credentials immediately, isolate affected systems from the network, preserve evidence
3. Investigate: Review audit logs to determine scope, affected data, root cause, and timeline of the incident
4. Notify: Inform legal, compliance, affected users, insurers, and regulators within required timeframes
5. Remediate: Fix the vulnerability, rotate all affected credentials, and improve monitoring to prevent recurrence
6. Review: Conduct a post-mortem within 30 days, update security policies, and document lessons learned

Security Checklist

Frequently Asked Questions

What is the difference between encryption at rest and in transit? Encryption at rest protects data stored on disk or in databases — if an attacker gains access to storage media, misconfigurations, or backups, encrypted data remains unreadable without the decryption key. Encryption in transit protects data moving across networks using TLS — if an attacker intercepts network traffic, they cannot read the encrypted contents. Both are required for compliance with most security frameworks including GDPR, HIPAA, and SOC 2.

How do I manage secrets in data pipelines? Use a dedicated secrets manager like HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Reference secrets by name in pipeline configuration; the secrets manager injects the actual values at runtime through API calls. Never store secrets in code, configuration files, environment variables, or plain text files. Implement secret rotation policies and audit access to secrets through the secrets manager’s logging capabilities.

What data needs to be encrypted at the field level? Personally Identifiable Information (PII) — Social Security numbers, credit card numbers, health records, passport numbers, and passwords — should be encrypted at the field level using application-layer encryption. This ensures that even if a storage system or database is compromised, the most sensitive data fields remain protected. Use envelope encryption with a key hierarchy to manage field-level encryption keys efficiently.

How often should I rotate encryption keys? Industry best practice recommends rotating encryption keys annually or whenever a key may have been compromised. Automated key rotation, supported by most KMS services, reduces the operational burden and ensures compliance with security policies. For high-value data, consider more frequent rotation (quarterly) and use separate keys for different data classifications.

What is the difference between RBAC and ABAC? Role-Based Access Control (RBAC) assigns permissions based on job roles — all data engineers get the same access to bronze and silver data. Attribute-Based Access Control (ABAC) uses attributes — user department, data classification, time of day, location, device type — to make fine-grained access decisions. ABAC provides more granular control but requires more complex policy management infrastructure. Many organizations use RBAC as a baseline and ABAC for specific high-sensitivity data.

Data Lakes Guide — Data Quality Guide — Pipeline Monitoring Guide