dataflow/WARN/2026_001
Active Dataflow streaming jobs should have Streaming Engine enabled.
Product: Dataflow
Rule class: WARN - Something that is possibly wrong
Description
Streaming Engine is disabled for one or more active streaming Dataflow jobs.
Without Streaming Engine, worker VMs perform windowing, state management, and other resource-intensive operations locally. This legacy execution mode can lead to poor autoscaling performance, high VM resource utilization, and potential out-of-memory (OOM) issues.
Remediation
To enable Streaming Engine, configure the pipeline options based on the SDK language:
For Java SDK (version 2.10.0 or later)
Set the following pipeline option:
--enableStreamingEngine
For Python SDK (version 2.21.0 or later)
Streaming Engine is enabled by default in newer Python SDK versions when using Python 3 and Runner v2. For older SDKs or to explicitly configure it:
- Enable Dataflow Runner v2 by setting the following option:
--experiments=use_runner_v2(Dataflow Runner v2 requires Streaming Engine for streaming jobs). - Alternatively, explicitly enable it by setting:
--enable_streaming_engine