AWS Glue PySpark Job Cost Optimization

The client was experiencing unusually high AWS Glue costs for multiple PySpark jobs. Despite processing datasets of less than one million records, several jobs required G.8X workers to complete successfully, significantly increasing operational expenses.

• Sector: Banking
• Scale: < 1 million records per job
• Goal: Reduce AWS Glue execution costs without compromising job reliability or SLAs.

The data processing pipeline used AWS Glue for ETL workloads:

1. Ingestion: Source data loaded from S3 into Glue PySpark jobs.
2. Processing: Multiple transformations and validations implemented in PySpark.
3. Execution: Glue jobs configured with high-capacity G.8X workers to avoid out-of-memory failures.

• Driver Memory Pressure: Several transformations (e.g., frequent count() operations) triggered actions that executed on the driver node.
• Inefficient Patterns: Use of head() after full dataset scans and unnecessary broadcast() operations increased memory usage.
• Misleading Scaling: Increasing worker size masked code inefficiencies rather than addressing the root cause.

A code-level optimization approach was implemented instead of scaling infrastructure.

• Action Reduction: Removed frequent count() calls and replaced them with conditional checks and cached metadata where possible.
• Driver Optimization: Refactored logic to avoid driver-heavy operations and ensured transformations executed on executors.
• Broadcast Control: Eliminated unnecessary broadcast joins and optimized join strategies.
• Execution Tuning: Validated memory usage through Spark UI and Glue metrics.

Results:

• Worker Size: Jobs successfully migrated from G.8X to G.1X workers.
• Cost: Significant reduction in Glue job execution cost per run.
• Performance: Improved job stability with no driver OOM errors.
• Scalability: Jobs now scale predictably with data growth instead of requiring oversized workers.