Airflow scheduler memory spikes when a dag is run with large run config

### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.2.0 ### What happened and how to reproduce it? When a dag with dynamic task is triggered with large run config, the scheduler memory increases by 5-6 times the regular usage when scheduling the dynamic tasks. Once the dagrun completes, the memory usage is back to normal. The run config is a json of size 512KB to 2MB To reproduce, create a dag with dynamic task that create 500+ tasks. Trigger the dag with a json of size 512KB to 1MB as run config ### What you think should happen instead? The scheduler memory should not spike when running dynamic task with large run config. ### Operating System Debian GNU/Linux 12 (bookworm) ### Deployment Other 3rd-party Helm chart ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-celery==3.17.1 apache-airflow-providers-cncf-kubernetes==10.14.0 apache-airflow-providers-common-compat==1.14.1 apache-airflow-providers-common-io==1.7.1 apache-airflow-providers-common-sql==1.33.0 apache-airflow-providers-fab==3.6.1 apache-airflow-providers-smtp==2.4.3 apache-airflow-providers-standard==1.12.1 ### Official Helm Chart version Not Applicable ### Kubernetes Version Not Applicable ### Helm Chart configuration Not Applicable ### Docker Image customizations Not Applicable ### Anything else? Issue happens in dynamic dags, everytime, the dag is triggered with big config. Older issue for reference - https://github.com/apache/airflow/issues/49076 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)