Standalone DAG Processor Causes DAGs to Appear and Disappear Frequently in Production

### Apache Airflow version 2.10.3 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? **Description:** I have Airflow 2.10.3 deployed in AKS using the Helm chart, and everything works fine. I tried to deploy the standalone DAG processor to run as a standalone process. Here is my configuration: ```yaml dagProcessor: enabled: true replicas: 2 revisionHistoryLimit: 5 resources: requests: cpu: 2500m ephemeral-storage: 200Mi memory: 2500Mi limits: ephemeral-storage: 200Mi memory: 2500Mi podAnnotations: *podAnnotations affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "airflow.workload" operator: In values: - dagprocessor podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: component: dagprocessor topologyKey: kubernetes.io/hostname weight: 100 tolerations: - key: "airflow.workload" value: "dagprocessor" operator: "Equal" effect: "NoSchedule" [core] standalone_dag_processor: "True" ``` I managed to separate the DAG processor pods and the worker node pool. However, I started encountering issues where DAGs appear and disappear frequently when the DAG bag size is large. In my QA environment with only 500 DAGs, I don't have this issue, but in production with more than 2000 DAGs, this happens frequently. In the cluster activity, I see the DAG processor state turning red (unhealthy) for a few seconds and then healthy again, in a non-ending cycle. The error in the logs is: ``` sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exceptio…