WO2013095083A1

WO2013095083A1 - A method and system of extending computing grid resources

Info

Publication number: WO2013095083A1
Application number: PCT/MY2012/000152
Authority: WO
Inventors: Mohd Amril Nurman MOHD NAZIR; Mohd Sidek Salleh; Mohammad Fairus Khalid; Tamarai Selvi SOMASUNDARAM; Rajendar KANDAN
Original assignee: Mimos Berhad; Madras Institute Of Technology, Anna University
Priority date: 2011-12-19
Filing date: 2012-06-28
Publication date: 2013-06-27
Also published as: MY158553A

Abstract

A method of extending computing grid resources by deploying virtual machines (VM) is provided, the method includes the steps of configuring at least one grid computing image in a cloud system, requesting at least one virtual machine from the cloud system, deploying a plurality of grid computing nodes in a cloud environment and removing the plurality of grid computing nodes from the cloud environment.

Description

A METHOD AND SYSTEM OF EXTENDING COMPUTING GRID RESOURCES

FIELD OF INVENTION

The present invention relates to a method and a system of extending computing grid resources by deploying virtual machines (VM).

BACKGROUND OF INVENTION

Long wait times for queued jobs when physical server nodes are fully utilised are an issue. When grid resources are fully utilised, new jobs are queued until resources become available. Jobs criteria include batch, sequential, workflow and parallel jobs. However, this entails delays in processing said jobs.

US 7577722 B1 describes monitoring a status of a set of hosts platforms by obtaining a new identifier different from a model virtual machine identifier that identifies the staged VM. However, this solution relies on a threshold value before re-evaluating usage of resources which may inadvertently still result in delays.

US 20080059556 A1 describes a virtual machine hypervisor logic executable on nodes with computer processor and physical memory. However, there is no emphasis on interoperability of systems in this prior art.

Therefore, there is a need for a solution that resolves the waiting issue of queued jobs due to hardware capacity restriction while taking into account interoperability issues. SUMMARY OF INVENTION

Accordingly, there is provided a method of extending computing grid resources by deploying virtual machines (VM), the method includes the steps of configuring at least one grid computing image in a cloud system, requesting at least one virtual machine from the cloud system, deploying a plurality of grid computing nodes in a cloud environment and removing the plurality of grid computing nodes from the cloud environment. Further, there is provided a system of extending computing grid resources by deploying virtual machines (VM), the system includes at least one Resource Manager, at least one VM Manager network connectible to the Resource Manager and at least one Cloud Controller in communication with the Resource Manager wherein the system is used for cloud applications through life cycle.

The present invention consists of several novel features and a combination of parts hereinafter fully described and illustrated in the accompanying description and drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be fully understood from the detailed description given herein below and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, wherein:

Figure 1 shows a block diagram of interactions between hardware/network layer in Grid and the laaS layer in Cloud in the present invention;

Figure 2 shows a conceptual diagram of communications and interactions between Grid and Cloud; Figure 3 shows a flowchart that illustrates a method of extending computing grid resources by deploying virtual machines (VM) in the preferred embodiment of the invention;

Figure 4 shows a block diagram of a system comparison between Prior Art and the present invention; Figure 5 shows a flowchart of configuration Steps for Grid Compute Node VM image(s) in the preferred embodiment of the invention;

Figure 6 shows a flowchart of Request Virtual Machine(s) from Cloud Resources in the preferred embodiment of the invention;

Figure 7 shows a flowchart of Creation Steps for Deployment of Grid Compute Node in the preferred embodiment of the invention; and

Figure 8 shows a flowchart of Deletion Steps to remove Grid Compute Node(s) from Cloud in the preferred embodiment of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention relates to a method and a system of extending computing grid resources by deploying virtual machines (VM). Hereinafter, this specification will describe the present invention according to the preferred embodiment of the present invention. However, it is to be understood that limiting the description to the preferred embodiment of the invention is merely to facilitate discussion of the present invention and it is envisioned that those skilled in the art may devise various modifications and equivalents without departing from the scope of the appended claims.

The following detailed description of the preferred embodiment will now be described in accordance with the attached drawings, either individually or in combination. Figure 4 illustrates a diagram of interactions between a system and other components with which the system is designed to interface which shows a difference of the present invention and prior art.

The VM Manager is the core component of the present invention which consists of two modules namely, the VM Request & Creation, and the VM Removal. The Resource Manager selects and sends queued job information to the VM Manager. Based on the queued job information, the VM Request & Creation requests the additional number of virtual machines from the Cloud system. The Cloud system corresponds by returning the virtual machines' IP address and hostname to the VM Manager. Subsequently, the virtual machines are configured as Grid Compute Nodes and the VM Manager sends the newly configured Grid Compute Nodes information to the Resource Manager. The Resource Manager schedules the job with the newly configured Grid Compute Nodes. When the job has completed, the Resource Manager notifies the VM manager. The VM Removal sends a request for the removal of virtual machines from the Cloud.

A method of extending computing grid resources by deploying virtual machines (VM) is described herein as seen in Figure 3. The method includes the steps of configuring at least one grid computing image in a cloud system, requesting at least one virtual machine from the cloud system, deploying a plurality of grid computing nodes in a cloud environment and removing the plurality of grid computing nodes from the cloud environment.

The step of configuring further includes the steps of creating a new disk image, installing a resource manager and operating system, mapping a plurality of user accounts to a client, configuring security and network settings, registering kernel and RAM disk image to a controlling means and uploading the disk image to the controlling means. Figure 5 illustrates the configuration steps for Grid Compute Node VM image(s). First, a new disk image is created for grid compute node. The Operating System is then installed on the new Disk Image. Subsequently, the Resource Manager client is also installed. Thereafter, user accounts from the Resource Manager Server are mapped to the Resource Manager Client. Next, Security settings and network communications between Resource Manager server and Resource Manager client are configured. The kernel and RAM disk is extracted from the disk image. Once extracted, the kernel and RAM disk is uploaded to the Cloud storage to be registered. Finally, the disk image is uploaded to the Cloud storage controller. The step of requesting at least one virtual machine further includes the steps of determining a queued job, extracting numbers of nodes, processors, memory capacity and storage of job, requesting number of virtual machines (VM) based on extracted information, reserving jobs when there are sufficient number of VM instances, recording job information and reserved VM status, running VM from cloud, obtaining reserved VM hostname and IP address and configuring VM. Figure 6 illustrates these steps for requesting virtual machine(s) from Cloud. First, the waiting or queued job is obtained from the Resource Manager. Initially, the waiting or queue job to be served is retrieved from the Resource Manager. Once the job to be served is identified, the number of requested nodes, CPUs, memory capacity, and storage are extracted from the job requirement. Based on the extracted job requirement, the number of available virtual machines is determined. If there are enough virtual machines to meet job requirement, the job is reserved to virtual machines. The Job information and the Virtual machines' status are then marked as reserved.

Figure 7 illustrates the creation steps for the deployment of Grid Compute Node. According to the figure, the VM hostname and IP address which was reserved earlier from Cloud is obtained. Subsequently, the VM hostname and IP address information are added to the Resource Manager to enable virtual machine as Grid compute node. The virtual machine is then configured to support parallel jobs. Next, the virtual machine is further configured to recognize and to communicate with the Resource Manager. The steps are repeated until there is no other virtual machine to configure. Finally, the reserved job is scheduled to run the newly configured and deployed Grid Compute nodes.

The step of removing the plurality of grid computing nodes further includes the steps of retrieving a list of queued jobs, determining if a job matches that of those in the list of retrieved queued jobs, removing a corresponding node allocated to a job not found on the list of queued jobs and terminating corresponding VM instance allocated to the job not found on the list of queued jobs. Figure 8 illustrates steps for the removal of Grid Compute Nodes from Cloud. The Accounting initially records a list of mapping between job and virtual machines which have been reserved. The job queue is checked regularly and periodically from the Resource Manager for any completed jobs. Each job record from Accounting is checked against the job record from the Resource Manager. If the job record does not exist in the Resource Manager, all reserved virtual machines for that specific job record are marked. Further, all marked virtual machines are destroyed. Subsequently, the specific job record is removed from the Accounting. These same steps are repeated for all other job record in the Accounting.

The invention leverages cloud system to accommodate the demand of applications. Scientific batch, workflow and parallel applications are compute intensive, and usually require the co-located use of large amounts of resources for a shorter period of time. Jobs will tend to present tighter coupling, and parallel applications fall into this category. Virtual machines offered by cloud systems have the ability to be flexible in the amount of nodes requested and are not limited by hardware constraints in terms of amount of computer resources available. The positioning of hierarchy (respective layers) in a computer constituting Grid computing and Cloud computing is shown in Figure 1. Above a bottom layer for Grid computing comprises a hardware layer which includes motherboard or an input output (I/O) association such as a processor (Hardware) or network equipment (Network). Above this layer is a platform layer (Platform) which includes operating system (OS) and the like. For example, operating systems such as Windows, Linux, and Unix are equivalent to this layer. There is a layer that is called middleware above the layer. Software for controlling security or computer resource management in Grid computing such as, but not limited to, gLite, Globus, and Unicore etc. resides at the middleware layer. Finally, a normal application or portal layer (Application or Portal) exists in a top layer.

Figure 1 also illustrates hierarchical layers of Cloud Computing. Cloud computing is a facility of services, software, or infrastructure, delivered via the Internet in a pay-per-use and self-service way. The lower level of Cloud computing is infrastructure-as-a-service (laaS) which describes platforms that provide computer and server infrastructure typically provided as a virtualization environment. The platform would provide the ability for consumers to scale their infrastructure up or down by demand and pay for the resources consumed. This is where pre-configured hardware is provided via a virtualised interface or hypervisor. There is no high level infrastructure software provided such as an operating system at this layer, and this must be provided by the users embedded with their own virtual applications.

Above the laas layer, the Platform as a Service (PaaS) provides the ability for building and deploying custom applications on Cloud resources. On the other hand, the Software as a service (SaaS) layer resides on top of the PaaS layer. With SaaS, fully functional applications are provided. Saas offers fully functional applications on-demand to provide specific services such as email management, CRM, ERP, web conferencing and an increasingly wide range of other commercial applications.

From Figure 1 , we can also see that that there is a communication link between the hardware/network layer in Grid and the laaS layer in Cloud. The present invention aims to use laaS Cloud infrastructure to expand/extend grid resources for processing a wide range of queued jobs which include serial, workflow and parallel jobs. The queued jobs originally derive from the Application / Portal layer which are subsequently handled at the Middleware, Platform and Hardware/Network layer.

Figure 2 illustrates the flow of job execution based on Grid Broker of a prior art. The Grid Broker serves as a middleware responsible for the distribution and management of jobs across Grid resources, The Resource Manager serves as a scheduler of that schedules jobs across the computational resources (Grid compute nodes). In the prior art, the system starts by receiving job submission directly from the user from which the Grid Broker distributes the receiving jobs to the Resource Manager for execution. If job requirements can be met, the job is scheduled immediately when there are sufficient nodes. However, if job requirements cannot be met by any of physical resource nodes available, the job is queued. Furthermore, if the physical server nodes are not fully utilized, the prior art employs modern virtualisation to configure and deploy virtual machines as Grid Compute Nodes to accommodate and satisfy the requirements of queued jobs. Nevertheless, there is a limit on the number of virtual machines that can be deployed: virtual machines cannot be deployed when all physical server nodes are fully utilized. Therefore, the present invention resolves the limitation issue by enabling the system to deploy additional virtual machines using cloud resources.

The preferred embodiment of the invention includes the following main steps as shown in Figure 3:

1. Pre-configuration Steps for Grid Compute Node VM image(s). These include steps on how to register a Grid Compute Image to support automatic deployment of Grid Compute Nodes in Cloud System. This step is only performed once.

2. Request Virtual Machines from Cloud Resources. These include steps on how the Resource Manager requests Virtual Machines from the Cloud System.

3. Creation Steps for Deployment of Grid Compute Node(s) in Cloud Systems.

These include steps on how new grid compute node(s) are deployed in Cloud environment. 4. Deletion Steps to remove Grid Compute Node(s) from Cloud Systems. These include steps on how deployed Grid Compute Node(s) are removed from the Cloud environment.

This invention is adapted for resolving the waiting issue of queued jobs by using cloud resources to expand grid resources for processing queued batch, serial, workflow and parallel queued jobs, when the resource requirements of the job cannot be accommodated due to hardware capacity restriction, the queued jobs can still be served by deploying additional Grid Compute Nodes on Cloud The disclosed invention is suitable, but not restricted to, for use in cloud computing systems.

Claims

1. A method of extending computing grid resources by deploying virtual machines (VM), the method includes the steps of: i. configuring at least one grid computing image in a cloud system;

ii. requesting at least one virtual machine from the cloud system;

iii. deploying a plurality of grid computing nodes in a cloud environment; and iv. removing the plurality of grid computing nodes from the cloud environment.

2. The method as claimed in claim 1 , wherein the step of configuring further includes the steps of creating a new disk image, installing a resource manager and operating system, mapping a plurality of user accounts to a client, configuring security and network settings, registering kernel and RAM disk image to a controlling means and uploading the disk image to the controlling means.

3. The method as claimed in claim 1 , wherein the step of requesting at least one virtual machine further includes the steps of :

i. determining a queued job;

ii. extracting numbers of nodes, processors, memory capacity and storage of job; iii. requesting number of virtual machines (VM) based on extracted information; iv. reserving jobs when there are sufficient number of VM instances;

v. recording job information and reserved VM status;

vi. running VM from cloud;

vii. obtaining reserved VM hostname and IP address; and

viii. configuring VM.

4. The method as claimed in claim 1 , wherein the step of removing the plurality of grid computing nodes further includes the steps of: i. retrieving a list of queued jobs;

ii. determining if a job matches that of those in the list of retrieved queued jobs; iii. removing a corresponding node allocated to a job not found on the list of queued jobs; and

iv. terminating corresponding VM instance allocated to the job not found on the list of queued jobs.

5. A system of extending computing grid resources by deploying virtual machines (VM), the system includes:

at least one Resource Manager; at least one VM Manager network connectible to the Resource Manager; and at least one Cloud Controller in communication with the Resource Manager wherein the system is used for cloud applications through life cycle.

6. The system as claimed in claim 5, wherein the Resource Manager selects and sends queued job information to the VM Manager.

7. The system as claimed in claim 5, wherein the VM Manager further includes the VM Request & Creation and the VM Removal.