v1.11.2

Important:
This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade to immediately to protect your systems and data.

Security Fixes

• [Cherry-pick 1.11] Add http response body size limit (#4252 @kevin-wangzefeng )
• [Cherry-pick 1.11] Add security context configuration (#4245 @JesseStutler)
• Remove the execute permission for some files, chmod to 644 (#4171 @JesseStutler)
• add a switch to control whether enable pprof in scheduler (#4173 @JesseStutler)
• Add warning msg when TLS verification disabled(#4211 @Monokaix)
• Add http server timeout(#4208 @Monokaix)

Other Improvements

• Bump image to v1.11.2 (#4232 @JesseStutler)
• Fix: remove controller-manager metrics that should not be introduced (#4202 @dongjiang1989)
• Filter useless logs in binpack (#4240 @XbaoWu)

Important Notes Before Upgrading

Change: Volcano Scheduler pprof Endpoint Disabled by Default
For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by:

• If you are using helm, specifying custom.scheduler_pprof_enable=true during Helm installation or upgrade.
• OR, manually setting the command-line argument --enable-pprof=true when starting the Volcano Scheduler.

Please be aware of the security implications before enabling this endpoint in production environments.

v1.11.0-network-topology-preview.3

Important
This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade immediately to protect your systems and data.

Security Fixes

• [Cherry-pick network-topology] Add http response body size limit (#4255 @JesseStutler)
• [Cherry-pick network-topology] Add security context configuration (#4250 @JesseStutler)
• Remove the execute permission for some files, chmod to 644 (#4171 @JesseStutler)
• add a switch to control whether enable pprof in scheduler (#4173 @JesseStutler)
• Add warning msg when TLS verification disabled(#4211 @Monokaix)
• Add http server timeout(#4208 @Monokaix)

Other Improvements

• Bump image to v1.11.0-network-topology-preview.3 (#4237 @JesseStutler)
• Add NetworkTopology plugin score doc (#4213 @ecosysbin)
• HyperNode supports select Nodes By labels (#4068 @ecosysbin)
• Update ubuntu base image (#4197 @Monokaix)

Important Notes Before Upgrading

Change: Volcano Scheduler pprof Endpoint Disabled by Default
For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by:

• If you are using helm, specifying custom.scheduler_pprof_enable=true during Helm installation or upgrade.
• OR, manually setting the command-line argument --enable-pprof=true when starting the Volcano Scheduler.

Please be aware of the security implications before enabling this endpoint in production environments.

v1.10.2

Important:
This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade immediately to protect your systems and data.

Security Fixes

• [Cherry-pick 1.10] Add http response body size limit (#4253 @kevin-wangzefeng)
• [Cherry-pick 1.10] Add security context configuration (#4246 @JesseStutler)
• Remove the execute permission for some files, chmod to 644 (#4171 @JesseStutler)
• add a switch to control whether enable pprof in scheduler (#4173 @JesseStutler)
• Add warning msg when TLS verification disabled(#4211 @Monokaix)
• Add http server timeout(#4208 @Monokaix)

Other Improvements

• Update ubuntu base image(#4194 @Monokaix)
• Bump image to v1.10.2 (#4231 @JesseStutler)

Important Notes Before Upgrading

Change: Volcano Scheduler pprof Endpoint Disabled by Default
For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by:

• If you are using helm, specifying custom.scheduler_pprof_enable=true during Helm installation or upgrade.
• OR, manually setting the command-line argument --enable-pprof=true when starting the Volcano Scheduler.

Please be aware of the security implications before enabling this endpoint in production environments.

v1.9.1

Important:
This release addresses multiple critical security vulnerabilities. We strongly advise all users to upgrade immediately to protect your systems and data.

Security Fixes

• [Cherry-pick 1.9] Add http response body size limit (#4254 @JesseStutler)
• [Cherry-pick 1.9] Add security context configuration (#4249 @Monokaix)
• Remove the execute permission for some files, chmod to 644 (#4171 @JesseStutler)
• add a switch to control whether enable pprof in scheduler (#4173 @JesseStutler)
• Add warning msg when TLS verification disabled(#4211 @Monokaix)
• Add http server timeout(#4208 @Monokaix)

Other Improvements

• Bump image to v1.9.1 (#4230 @JesseStutler)
• fix panic when get job's elastic resource (#4103 @lowang-bh)
• change to action cache v4 (#4075 @Monokaix)
• fix flaky test (#4121 @Monokaix)
• Supports rollback when allocate callback function fails (#3780 @wangyang0616)
• Supports rollback when allocate callback function fails (#3776 @wangyang0616)
• fix pg controller create redundancy podGroup when schedulerName isn't matched (#3675 @liuyuanchun11)
• Update Kubernetes compatibility (#3570 @Monokaix)
• Fix podgroup not created (#3572 @liuyuanchun11)
• update pod status when bind error (#3550 @bibibox)
• Update NominatedNodeName for pipelined task (#3501 @bibibox)

Important Notes Before Upgrading

Change: Volcano Scheduler pprof Endpoint Disabled by Default
For security enhancement, the pprof endpoint for the Volcano Scheduler is now disabled by default in this release. If you require this endpoint for debugging or monitoring, you will need to explicitly enable it post-upgrade. This can be achieved by:

• If you are using helm, specifying custom.scheduler_pprof_enable=true during Helm installation or upgrade.
• OR, manually setting the command-line argument --enable-pprof=true when starting the Volcano Scheduler.

Please be aware of the security implications before enabling this endpoint in production environments.

v1.11.0-network-topology-preview.2

What's Changed

• [cherry-pick]change to action cache v4 by @Monokaix in #4074
• [Cherry-pick network-topology] Replace queue status update by using ApplyStatus method & Bump image to v1.11.0-network-topology-preview.2 by @JesseStutler in #4153

Full Changelog: v1.11.0-network-topology-preview.0...v1.11.0-network-topology-preview.2

v1.11.1

What's Changed

• [cherry-pick]change to action cache v4 by @Monokaix in #4075
• [cherry-pick]fix creating a hierarchical sub-queue will be rejected by @zhutong196 in #4080
• [cherry-pick] Fix jobflow status confusion problem by @dongjiang1989 in #4094
• [cherry-pick] fix: the problem that PVC will be continuously created indefinitely by @ytcisme in #4144
• [Cherry-pick v1.11] Replace queue status update by using ApplyStatus method & Bump image to v1.11.1 by @JesseStutler in #4155
• [Cherry-pick v1.11] fix: remove lessPartly condition in reclaimable fn from capacity and proportion plugins by @JesseStutler in #4178

Full Changelog: v1.11.0...v1.11.1

v1.10.1

What's Changed

• [cherry-pick for release-1.10]fix job controller reports duplicate warnings by @liuyuanchun11 in #3755
• [cherry-pick for release 1.10] Fix predicate return unexpected result by @bibibox in #3859
• [cherry-pick for release-1.10]Supports rollback when allocate callback function fails by @wangyang0616 in #3864
• change to action cache v4 by @Monokaix in #4096
• [cherry-pick] Fix jobflow status confusion problem by @dongjiang1989 in #4095
• [cherry-pick] fix panic when get job's elastic resource by @lowang-bh in #4103
• Update release-1.10 api version to v1.10.1 and Bump image to v1.10.1 by @JesseStutler in #4154

Full Changelog: v1.10.0...v1.10.1

v1.11.0

What's New

Welcome to the v1.11.0 release of Volcano! 🚀 🎉 📣
In this release, we have brought a bunch of significant enhancements that have long-awaited by community users.

Feature Preview: Network Topology Aware Scheduling

In AI large model training scenarios, model parallelism splits the model across multiple nodes, requiring frequent data exchange between these nodes during training. At this point, network transmission performance between nodes often becomes a bottleneck, significantly impacting training efficiency. Data centers have diverse network types (e.g., IB, RoCE, NVSwitch) and complex network topologies, typically involving multiple layers of switches. The fewer switches between two nodes, the lower the communication latency and the higher the throughput. Therefore, users want to schedule workloads to the best performance domain with the highest throughput and lowest latency, minimizing cross-switch communication to accelerate data exchange and improve training efficiency.

To address this, Volcano has introduced the Network Topology Aware Scheduling strategy, solving the network communication performance issues in large-scale data center AI training tasks through a unified network topology API and intelligent scheduling policies. It provides the following capabilities:

• Unified Network Topology API

  Introduced the HyperNode CRD to accurately express the network topology of data centers.

• Network Topology-Aware Scheduling Policy

  Volcano Job and PodGroup can set topology constraints for jobs through the networkTopology field, supporting the following configurations:

  • mode: Supports hard and soft modes.
    • hard: Hard constraint, tasks within the job must be deployed within the same HyperNode.
    • soft: Soft constraint, attempts to deploy the job within the same HyperNode.
  • highestTierAllowed: Used with hard mode, indicating the highest tier of HyperNode the job is allowed to span.

Design doc: Topology Aware Scheduling.

User Guide: Topology Aware Scheduling | Volcano.

Related PRs: (#3850, #144, #3874, #3922, #3964, #3971, #3974, #3887, #3897, @ecosysbin, @weapons97, @Xu-Wentao,@penggu @JesseStutler, @Monokaix)

Supports Elastic Hierarchical Queue

In multi-tenant scenarios, fairness, isolation, and task priority control in resource allocation are core requirements. Different departments or teams often need to share cluster resources while ensuring their tasks can obtain resources on demand, avoiding resource contention or waste. To address this, Volcano has introduced the Elastic Hierarchical Queue feature, significantly enhancing queue resource management capabilities. Through hierarchical queues, users can achieve finer-grained resource quota management, cross-level resource sharing and reclamation, and flexible preemption strategies, building an efficient and fair unified scheduling platform. For users of YARN, they can seamlessly migrate big data workloads to Kubernetes clusters using Volcano.

Volcano's elastic hierarchical queues have the following key features to meet the complex demands of multi-tenant scenarios:

1. Supports Configuring Queue Hierarchies
   Users can create multi-level queues as needed, forming a tree structure. Each queue can set independent resource quotas and priorities, ensuring fair resource allocation.
2. Cross-Level Resource Sharing and Reclamation
   When a sub-queue is idle, its resources can be shared with other sub-queues. When jobs are submitted to the sub-queue, resources can be reclaimed from other sub-queues.
3. Fine-Grained Resource Quota Management
   Each queue can set the following resource parameters:
   • capability: The upper limit of the queue's resource capacity.
   • deserved: The amount of resources the queue deserves. If the queue's allocated resources exceed the deserved value, the excess can be reclaimed.
   • guarantee: The reserved resources for the queue, ensuring the minimum resource guarantee.
4. Flexible Preemption Strategies
   Supports priority-based resource preemption, ensuring high-priority tasks can obtain the required resources promptly.

For detailed design and usage guidance on elastic hierarchical queues, please refer to:

Design doc: hierarchical-queue-on-capacity-plugin.

User Guide: Hierarchical Queue | Volcano.

Related PRs: (#3591, #3743, @Rui-Gan)

Supports Multi-Cluster AI Job Scheduling

With the rapid growth of enterprise business, a single Kubernetes cluster often cannot meet the demands of large-scale AI training and inference tasks. Users typically need to manage multiple Kubernetes clusters to achieve unified workload distribution, deployment, and management. Currently there are already many users using Volcano in multiple clusters and using Karmada to managem them, in order to better support AI jobs in multi-cluster environment, support global queue management, job priority and fair scheduling, etc., the Volcano community has incubated the Volcano Global sub-project. This project extends Volcano's powerful scheduling capabilities in single clusters to provide a unified scheduling platform for multi-cluster AI jobs, supporting cross-cluster job distribution, resource management, and priority control.

Volcano Global provides the following enhancements on top of Karmada to meet the complex demands of multi-cluster AI job scheduling:

1. Supports Cross-Cluster Scheduling of Volcano Jobs
   Users can deploy and schedule Volcano Jobs across multiple clusters, fully utilizing the resources of multiple clusters to improve task execution efficiency.
2. Queue Priority Scheduling
   Supports cross-cluster queue priority management, ensuring high-priority queue tasks can obtain resources first.
3. Job Priority Scheduling and Queuing
   Supports job-level priority scheduling and queuing mechanisms in multi-cluster environments, ensuring critical tasks are executed promptly.
4. Multi-Tenant Fair Scheduling
   Provides cross-cluster multi-tenant fair scheduling capabilities, ensuring fair resource allocation among tenants and avoiding resource contention.

For detailed introduction and user guide, please refer to: Multi-cluster Scheduling | Volcano.

Related PRs: (see https://github.com/volcano-sh/volcano-global.git, @Vacant2333, @MondayCha, @lowang-bh, @Monokaix)

Supports Online and Offline Workloads Colocation

The core idea of online and offline colocation is to deploy online services (e.g., real-time services) and offline jobs (e.g., batch processing tasks) in the same cluster. When online services are in a trough, offline jobs can utilize idle resources; when online services peak, offline jobs are suppressed through priority control to ensure the resource needs of online services. This dynamic resource allocation mechanism not only improves resource utilization but also guarantees the quality of service for online services.

Volcano's cloud native colocation solution provides end-to-end resource isolation and sharing mechanisms from the application layer to the kernel, including the following core components:

Volcano Scheduler
Responsible for unified scheduling of online and offline jobs, providing abstractions such as queues, groups, job priorities, fair scheduling, and resource reservations to meet the scheduling needs of various business scenarios like microservices, big data, and AI.

Volcano SLO Agent
The SLO Agent deployed on each node monitors the node's resource usage in real-time, dynamically calculates overcommitted resources, and allocates these resources to offline jobs. Meanwhile, the SLO Agent detects CPU/memory pressure on the node and evicts offline jobs when necessary to ensure the priority of online services.

Enhanced OS
To further strengthen resource isolation, Volcano implements fine-grained QoS guarantees at the kernel level. Through cgroup interfaces, different resource limits are set for online and offline services, ensuring online services receive sufficient resources even under high load.

Volcano's cloud native colocation solution has the following key capabilities, helping users achieve a win-win situation in resource utilization and business stability:

• Unified Scheduling: Supports unified scheduling of various workloads, including microservices, batch jobs, and AI tasks.
• QoS-Based Resource Model: Provides quality of service (QoS)-based resource management for online and offline services, ensuring the stability of high-priority services.
• Dynamic Resource Overcommitment: Dynamically calculates overcommitted resources based on real-time CPU/memory utilization of nodes, maximizing resource utilization.
• CPU Burst: Allows containers to temporarily exceed CPU limits, avoiding throttling at critical moments and improving business responsiveness.
• **Ne...

v1.10.0

What's New

Support Queue Priority Scheduling Strategy

In traditional big data processing scenarios, users can directly set queue priorities to control the scheduling order of jobs. To ease the migration from Hadoop/Yarn to cloud-native platforms, Volcano supports setting priorities at the queue level, reducing migration costs for big data users while enhancing user experience and resource utilization efficiency.

Queues are a fundamental resource in Volcano, each with its own priority. By default, a queue's priority is determined by its share value, which is calculated by dividing the resources allocated to the queue by its total capacity. This is done automatically, with no manual configuration needed. The smaller the share value, the fewer resources the queue has, making it less saturated and more likely to receive resources first. Thus, queues with smaller share values have higher priority, ensuring fairness in resource allocation.

In production environments—especially in big data scenarios—users often prefer to manually set queue priorities to have a clearer understanding of the order in which queues are scheduled. Since the share value is dynamic and changes in real-time as resources are allocated, Volcano introduces a priority field to allow users to set queue priorities more intuitively. The higher the priority, the higher the queue's standing. High-priority queues receive resources first, while low-priority queues have their jobs reclaimed earlier when resources need to be recycled.

To ensure compatibility with the share mechanism, Volcano also considers the share value when calculating queue priorities. By default, if a user has not set a specific queue priority or if priorities are equal, Volcano will fall back to comparing share values. In this case, the queue with the smaller share has higher priority. Users have the flexibility to choose between different priority strategies based on their specific needs—either by using the priority or the share method.

Queue priority design doc: Queue priority

Related PRs: (#132, #3700, @TaiPark)

Enable Fine-Grained GPU Resource Sharing and Reclaim

Volcano introduced the elastic queue capacity scheduling feature in version v1.9, allowing users to directly set the capacity for each resource dimension within a queue. This feature also supports elastic scheduling based on the deserved value, enabling more fine-grained resource sharing and recycling across queues.

For detailed design information on elastic queue capacity scheduling, refer to the Capacity Scheduling Design Document.

For a step-by-step guide on using the capacity plugin, see the Capacity Plugin User Guide.

In version v1.10, Volcano extends its support to include reporting different types of GPU resources within elastic queue capacities. NVIDIA's default Device Plugin does not distinguish between GPU models, instead reporting all resources uniformly as nvidia.com/gpu. This limits AI training and inference tasks from selecting specific GPU models, such as A100 or T4, based on their particular needs. To address this, Volcano now supports reporting distinct GPU models at the Device Plugin level, working with the capacity plugin to enable more precise GPU resource sharing and recycling.

For instructions on using the Device Plugin to report various GPU models, refer to the GPU Resource Naming Guide.

Note:

In version v1.10.0, the capacity plugin is the default for queue management. Note that the capacity and proportion plugins are incompatible, so after upgrading to v1.10.0, you must set the deserved field for queues to ensure proper functionality. For detailed instructions, refer to the Capacity Plugin User Guide.

The capacity plugin allocates cluster resources based on the deserved value set by the user, while the proportion plugin dynamically allocates resources according to queue weight. Users can select either the capacity or proportion plugin for queue management based on their specific needs. For more details on the proportion plugin, visit: Proportion Plugin.

Related PR: (#68, @MondayCha)

Introduce Pod Scheduling Readiness Support

Once a Pod is created, it is considered ready for scheduling. In Kube-scheduler, it will try its best to find a suitable node to place all pending Pods. However, in reality, some Pods may be in a "lack of necessary resources" state for a long time. These Pods actually interfere with the decision-making and operation of the scheduler (and downstream components such as Cluster AutoScaler) in an unnecessary way, causing problems such as resource waste. Pod Scheduling Readiness is a new feature of Kube-sheduler. In Kubernetes v.1.30 GA, it has become a stable feature. It controls the scheduling timing of Pods by setting the schedulingGates field of the Pod.

Volcano supports unified scheduling of online and offline jobs. In order to better support the scheduling of microservices, we also support Pod Scheduling Readiness scheduling in Volcano v1.10 to meet the scheduling needs of users in multiple scenarios.

For the documentation of Pod Scheduling Readiness features, please refer to: Pod Scheduling Readiness | Kubernetes

Related PR: (#3581, @ykcai-daniel)

Add Sidecar Container Scheduling Capabilities

A Sidecar container is an auxiliary container designed to support the main business container by handling tasks such as logging, monitoring, and network initialization.

Prior to Kubernetes v1.28, the concept of Sidecar containers existed only informally, with no dedicated API to distinguish them from business containers. Both types of containers were treated equally, which meant that Sidecar containers could be started after the business container and might end before it. Ideally, Sidecar containers should start before and finish after the business container to ensure complete collection of logs and monitoring data.

Kubernetes v1.28 introduces formal support for Sidecar containers at the API level, implementing unified lifecycle management for init containers, Sidecar containers, and business containers. This update also adjusts how resource requests and limits are calculated for Pods, and the feature will enter Beta status in v1.29.

The development of this feature involved extensive discussions, mainly focusing on maintaining compatibility with existing APIs and minimizing disruptive changes. Rather than introducing a new container type, Kubernetes reuses the init container type and designates Sidecar containers by setting the init container’s restartPolicy to Always. This approach addresses both API compatibility and lifecycle management issues effectively.

With this update, the scheduling of Pods now considers the Sidecar container’s resource requests as part of the business container’s total requests. Consequently, the Volcano scheduler has been updated to support this new calculation method, allowing users to schedule Sidecar containers with Volcano.

For more information on Sidecar containers, visit Sidecar Containers | Kubernetes.

Related PR: (#3706, @Monokaix, @7h3-3mp7y-m4n)

Enhance Vcctl Command Line Tool

vcctl is a command line tool for operating Volcano's built-in CRD resources. It can be conveniently used to view/delete/pause/resume vcjob resources, and supports viewing/deleting/opening/closing/updating queue resources. Volcano has enhanced vcctl in the new version, adding the following features:

• Support creating/deleting/viewing/describing jobflow and jobtemplate resources

• Support querying vcjob in a specified queue

• Support querying Pods by queue and vcjob filtering

For detailed guidance documents on vcctl, please refer to: vcctl Command Line Enhancement.

Relared PRs: (#3584, #3543, #3530, #3524, #3508, @googs1025)

Ensure Compatibility with Kubernetes v1.30

Volcano closely follows the pace of Kubernetes community versions and supports every major version of Kubernetes. The latest supported version is v1.30, and runs complete UT and E2E use cases to ensure functionality and reliability.

If you want to participate in the development of Volcano adapting to the new version of Kubernetes, please refer to: adapt-k8s-todo for community contributions.

Related PR: (#3556, @guoqinwill, @wangyysde)

Strengthen Volcano Security Measures

Volcano has always attached great importance to the security of the open source software supply chain. It follows the specifications defined by OpenSSF in terms of license compliance, security vulnerability disclosure and repair, warehouse branch protection, CI inspection, etc. Volcano recently added a new workflow to Github Action, which will run OpenSSF security checks when the code is merged, and update the software security score in real time to continuously improve software security.

At the same time, Volcano has reduced the RBAC permissions of each component, retaining only the neces...

v1.9.0

What's New

Support elastic queue capacity scheduling

Volcano now uses the proportion plugin for queue management. Users can set the guarantee, capacity and other fields of the queue to set the reserved resources and capacity limit of the queue. And by setting the weight value of the queue to realize the resource sharing within the cluster, the queue is proportionally divided into cluster resources according to the weight value, but this queue management method has the following problems:

• The capacity of the resources divided by the queue is reflected by the weight, which is not intuitive enough.
• All resources in the queue are divided using the same ratio, and the capacity cannot be set separately for each dimension of the queue.

Based on the above considerations, Volcano implements a new queue elasticity capacity management capability, it supports:

• Allows users to directly set the capacity of each dimension of resources for the queue instead of setting a weight value.
• Elastic capacity scheduling based deserved resources, and queue's resources can be shared and reclaimed back.

For example, in AI large model training scenario, setting different resource capacities for different GPU models in the queue, such as A100 and V100, respectively. At the same time, when the cluster resources are idle, the queue can reuse the resources of other idle queues, and when needed, reclaim the resources set by the user for the queue, that is, the amount of resources deserved, so as to realize the elastic capacity scheduling.

To use this feature, you need to set the deserved field of the queue and set the amount of resources to be deserved for each dimension. At the same time, you need to turn on the capacity plugin and turn off the proportion plugin in the scheduling configuration.

Please refer to Capacity Scheduling Design for more detail.

Capacity scheduling example: How to use capacity plugin.

Related PR: (#3277, #121, #3283, @Monokaix)

Support affinity scheduling between queues and nodes

Queues are usually associated with departments within the company, and different departments usually need to use different heterogeneous resource types. For example, the large model training team needs to use NIVDIA’s Tesla GPU, and the recommendation team needs to use AMD’s GPU. When users submit jobs to the queue , the job needs to be automatically scheduled to the node of the corresponding resource type according to the attributes of the queue.

Volcano has implemented affinity scheduling capabilities for queues and nodes. Users only need to set the node label that require affinity in the affinity field of the queue. Volcano will automatically schedule jobs submitted to the current queue to the nodes associated with the queue. Users do not need to Set the affinity of the job separately, and only need to set the affinity of the queue uniformly. Jobs submitted to the queue will be scheduled to the corresponding node based on the affinity of the queue and the node.

This feature supports hard affinity, soft affinity, and anti-affinity scheduling at the same time. When using it, you need to set a label with the key volcano.sh/nodegroup-name for the node, and then set the affinity field of the queue to specify hard affinity, soft affinity label values.

The scheduling plugin for this feature is called nodegroup, for a complete example of its use see: How to use nodegroup plugin.

For detailed design documentation, see The nodegroup design.

Related PR: (#3132, @qiankunli, @wuyueandrew)

GPU sharing feature supports node scoring scheduling

GPU Sharing is a GPU sharing and isolation solution introduced in Volcano v1.8, which provides GPU sharing and device memory control capabilities to enhance the GPU resource utilization in AI training and inference scenarios. v1.9 adds a new scoring strategy for GPU nodes on top of this feature, so that the optimal node can be selected during job assignment to further enhance resource utilization. Users can set different scoring strategies. Currently, the following two strategies are supported:

• Binpack: Provides a binpack algorithm for GPU card granularity, prioritizing to fill up a node with GPU cards that have already been allocated resources to avoid resource fragmentation and waste.

• Spread: Prioritizes the use of idle GPU cards over shared cards that have already been allocated resources.

For detailed usage documentation, please refer to: How to use gpu sharing.

Related PR: (#3471, @archlitchi)

Volcano support Kubernetes v1.29

Volcano version follows the Kubernetes community version tempo and supports every base version of Kubernetes. The latest supported version is v1.29 and ran full UT, E2E use cases to ensure functionality and reliability. If you would like to participate in the development of Volcano adapting to new versions of Kubernetes, please refer to: #3459 to make community contributions.

Related PR: (#3295, @guoqinwill)

Enhance scheduler metrics

Volcano uses the client-go to talk with Kubernetes. Although the client can set the QPS to avoid requests from being flow-limited, it is difficult to observe how many QPS is actually used by the client, so in order to observe the frequency of requests from the client in real time, Volcano has added a new client-go metrics, which allows users to access the metrics to see the number of GET, POST and other requests per second, so as to get the actual QPS used per second, and thus decide whether or not the client needs to adjust the QPS. The client-go metrics also include client certificate rotation cycle statistics, response size per request statistics, etc.

Users can use curl http://$volcano_scheduler_pod_ip:8080/metrics to get all the detailed metrics of volcano scheduler.

Related PR: (#3274, @Monokaix)

Add license compliance check

In order to enhance the open source license compliance governance standards of the Volcano community, avoid the introduction of infectious open source protocols, and avoid potential risks, the Volcano community has introduced an open source license compliance checking tool. The so-called infectious protocol refers to software that uses this protocol as an open source license. Derivative works generated after modification, use, and copying must also be open sourced under this agreement. If the third-party library introduced by the PR submitted by the developer contains infectious open source protocols such as GPL, LGPL, etc., CI Access Control will intercept it. The developer needs to replace the third-party library with a loose free software license protocol such as MIT, Apache 2.0, BSD, etc. , to pass the open source license compliance check.

Related PR: (#3308, @Monokaix)

Improve scheduling stability

Volcano v1.9.0 has done more optimization in preemption, retry for scheduling failure, avoiding memory leaks, security enhancement, etc. The details include:

• Fix the problem of pods not being able to be scheduled due to frequent expansion and contraction of deployment in extreme cases, see PR for details: (#3376, @guoqinwill)
• Fix Pod preemption: see PR for details: (#3458, @LivingCcj)
• Optimize Pod scheduling failure retry mechanism: see PR for details: (#3435，@bibibox）
• Metrics metrics optimization: (#3463, @Monokaix)
• Security enhancements: (#3449, @lekaf974)

Changes

• cherry-pick bugfixs (#3464 @Monokaix)
• fix nil pointer panic when evict (#3443 @bibibox)
• fix errTask channel memory leak (#3434 @bibibox)
• register nodegroup plugin to factory (#3402 @wuyueandrew)
• fix panic when the futureIdle resources are calculated to be negative (#3393 @Lily922)
• fix jobflow CRD metadata.annotations: Too long error (#3356 @guoqinwill)
• fix PodGroup being incorrectly deleted due to frequent creation and deletion of pods (#3376 @guoqinwill)
• fix rollback unthoroughly when allocate error (#3360 @bibibox)
• fix panic when the gpu is faulty (#3355 @guoqinwill)
• Support preempting BestEffort pods when the pods number of nodes reaches the upper limit (#3338 @Lily922)
• change private function 'unmarshalSchedulerConf' to public function 'UnmarshalSchedulerConf' (#3333 @Lily922)
• add pdb support feature gate (#3328 @bibibox)
• add mock cache for ut and performance test (#3269 @lowang-bh)
• support dump scheduler cache snapshot to json file (#3162 @lowang-bh)
• Volcano adapts to the k8s v1.29 (#3295 @guoqinwill)
• add lowang-bh as reviewer ([#32...