In the near future, Astronomer will have an option for KEDA autoscaling on Celery, combining a lot of the great features between Kubernetes executor and Celery executor. It’s also great for environments with long running tasks and users pushing code when jobs are running (since there is no grace period concept). The Kubernetes executor is great for dags that have really different requirements between tasks (e.g, the first task may be a sensor that only requires a few resources, but the downstream tasks have to run on your GPU node pool with a higher CPU request). In summary, the Celery executor is a great fit for any environment where the tasks are “similar” and you can find a configuration for the worker that fits all sizes, or for any tasks that need to run quickly (since the workers are “always on”).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |