[Flow Control ] Tracking Production Readiness and Hardening

[LukeAVanDrie]

What is this?: This is a tracking issue for the tasks required to make the experimental Flow Control layer robust, observable, and ready for production use.

Action Items:

• Distributed Tracing: Ensure trace context is propagated correctly through the Flow Control layer so that the full lifecycle of a request (including time spent in queues) is visible.
• Logging Audit: Review and enhance logs to ensure they are structured, actionable, and provide clear insight into policy decisions, especially during saturation events.
• Context Propagation: Verify that all necessary request-scoped context (e.g., deadlines, tenant IDs) is correctly passed from the enqueue stage to the dispatch stage.
• Performance Profiling: Use pprof or similar mechanisms to allow for detailed performance profiling of the Flow Control logic under load. Make optimizations where necessary.
• Load Testing & Failure Injection: Develop a suite of tests to validate the system's stability and behavior under sustained high load and in various failure scenarios (e.g., pod churn).