Scheduler stalls: 3.2 exceeds_max_non_backfill write contends with DAG-processor SELECT … FOR UPDATE on the dag table

### Apache Airflow version 3.2.0+ (observed on 3.2.2) ### What happened The scheduler intermittently stalls: `scheduler.critical_section_duration` climbs while scheduler CPU drops and task scheduling starves. Inspecting the metadata DB during a stall shows scheduler statements of the form `UPDATE dag SET exceeds_max_non_backfill = … WHERE dag.dag_id = …` blocked with `wait_event_type = Lock` / `wait_event = transactionid`, waiting on the DAG processor's parse transaction. ### Root cause Before 3.2, the scheduler evaluated `max_active_runs` in memory and did not write the `dag` row during dagrun scheduling. **#60006 ("Separate 'next dag run' from 'max active runs'", merge commit `32bc0119b408c9a37841edd021b646324a02745e`, shipped in 3.2.0)** added the `dag.exceeds_max_non_backfill` column (migration `0097_3_2_0_add_exceeds_max_runs_flag_to_dag_model`) and made the scheduler **write it to the `dag` row** during dagrun scheduling: - Write: `airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs` (called from `_create_dag_runs` and `_start_queued_dagruns`). That write now competes with a pre-existing lock held by the DAG processor for the entire per-file parse transaction: - Lock: `airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags` → `with_row_locks(select(DagModel) …)` i.e. `SELECT … FOR UPDATE` on every `dag` row for a parsed file, held through `update_dags` + serialization + permission sync until the transaction commits. So the scheduler's `UPDATE dag SET exceeds_max_non_backfill = …` blocks on the DAG processor's `FOR UPDATE`. This is amplified when a single parsed file defines many DAGs (e.g. dynamically generated DAGs), because that parse transaction can run for minutes — holding the `dag`-row locks the whole time and stalling the scheduler loop. ### How to reproduce it (conceptually) 1. Use the FAB auth manager and generate many DAGs from a single DAG file. 2. While the DAG processor is parsing tha…