Hi  

VIDHYASAGAR

That's depend of the visualization you build. If you use direct DD access you get good performance, but your VM is not movable. If you use virtual disk you can move all the virtual machine relatively quickly. But you pay the virtual disk management and the COW if you use snapshot.

There are no dream IBM invent the virtual machine in 1970 and everyone know the cost.

So it's the balance between usability of management and performance cost. You can;t have both.

Sorry. !

Pierre Léonard

You see the clock speed of PS and VS. Due to the clock speed its performance is differ. Because same clock speed is shared between HOST and Guest OS in VM. Hence its performance is decreases.

There are different kinds of virtualization strategies implemented by different tools:  See link for an example of a performance study.

http://his.diva-portal.org/smash/get/diva2:747573/FULLTEXT01.pdf

Hi Sudhakar,

The différence is that for a virtual machine you have a piece of code named hyperviseur that provide the access to the physical machine and it take a part of the CPU.

Pierre Léonard

I'm puzzled: what are you expecting?

Virtualization shows things down - at best, virtualization can reduce performance by a small amount.  The code paths for IO (network, disk) is normally worse for a VM than without, since an operation has to traverse more layers of software.

In your case, the comparison is a bit silly.  Almost everything about the nodes being compared is different, especially the OS.  Why would you expect any similarity?  It is not by accident that most of both the web and HPC worlds use Linux, not Windows (for both physical and virtual OSs.)

Hi,

Yes Mark is right. And there are some other point to add ::

• each virtual machine has get memory for there kernel ;
• if you use a database in two virtual machine, you use two binary loaded into the memory ;

That means that you need more memory to get multiple VM and if you don't have enough memory, the host OS will swap an yuo get an important t loss of efficiency.

For example,; we have a spécofique computer for VM and that computer have 768To of mémory..

Hope that help you.

Pierre Léonard

performance will be less in virtual machine compare to physical machine. however  in virtualization, efficient resources re used.

Hi  

VIDHYASAGAR

That's depend of the visualization you build. If you use direct DD access you get good performance, but your VM is not movable. If you use virtual disk you can move all the virtual machine relatively quickly. But you pay the virtual disk management and the COW if you use snapshot.

There are no dream IBM invent the virtual machine in 1970 and everyone know the cost.

So it's the balance between usability of management and performance cost. You can;t have both.

Sorry. !

Pierre Léonard

It depends on the virtualization tool and on the application you are running. 

I am using VMware workstation. And running data mining algorithm on Hadoop cluster.

Software machines rely on virtualization software underneath therefore software machines are always less efficient than physical machines...the question is can you live with this or not?

I think running a cluster inside a workstation is less efficient than if you run the algorithm directly into it...you are not gaining any processing power through the VMWare software, on the contrary you are losing efficiency by adding more software layers to perform the same operation.

Recomend to look for a testing software that can give you metrics on how your software cluster is doing first. Knowing how many flops you can do in tests you can start looking for options... the sooner the better.

Does it mean we should not use virtulization technology and prefer physical machnie in context of performance gaining? Then whats the sole benefit of virtualization?

efficient utilization of resources, only if your application is not using all the computer resources you can handle them better using virtualization

How did you specify having one processor with 4 cores in your virtual machine?

Hi all,

The % of use of a server is around 20%. That's a large waste of money. That the reason virtualization provides a a solution for a better use of the server. But  As I said before, you use more memory for the different systems and take time to access to the physical machine through the hyper visor. There are different type of virtialization. simply wmvare player and virtualbox that are toys for the efficiency (you can take a part of the cores. but the memory is blocked all the time the VM will work. You can use a different system as the proxmox distribution which is more efficient because it provides all the element of the computer dynamically until the limit you have fixed. It's a very good solution in the free software.

Elsewhere you can purchase the vmware ESX solutions or the V virtualization of microsoft which are costly.

I have a big server with lot of VM. I build regularly VM with different kind of configuration for big memory, for lot of cores. Last VM I build the questions was : I want the most up to date system. He break his VM in ten days. and the other VM continue to work that also a good reason to use virtualization.

Hoping that help

Pierre Léonard.

@Ahmed El-Mahdy, I am using VMware workstation which provides to configure memory and processor, cores for the guest system.

Hi Sudhakar,

In all that discussion, excepted that you complain about poor performance, what is the real reason you wan't tu use a VM ?

Pierre Léonard

Hi Sudhakar,

One of the reasons to use VMs is the one others have mentioned, i.e., being able to use resources more efficiently. Another reason could be to have different OS in the same machine (like in your case), and yet another reason could be security/resilience, VMs act as sandboxes that isolate different services from one another.

Best,

Eduardo

Using virtualization enables efficient use of the resources of the computer system. In your case - in virtualization can effectively use expensive hardware. But virtualization available mechanisms for resource management, which are often the cause of reducing overall productivity. Pierre quite rightly gives examples of use. This does not mean  - not using virtualization. So you can realize the distributed system based on different operating systems. Furthermore, virtualization encapsulates your virtual machines and ensure security. I personally would not encapsulated services. In this case it slows too much interaction. But this is only my opinion. Whether you use virtualization or not is a matter of applications and objectives that you put on yourself.

For me to be more useful with my answers and views though I need more information.

Best regards!

It was really a nice and fruitful discussion. I really acquired a lot. Thanks to everyone. 

Hi Sudhakar,,

It was a pleasure to help. Me to I need help some time and a lot of people answer, thta the community impact and necessity.. I hope that you find all the answers you need.

You' will be welcome.

Pierre

I am taking part in this discussion late. But better late than never. I also had same question. I created Hadoop Namenode and 2/3 Data nodes on my VMs and later the same configuration on the physical nodes in our lab. Found that the CPU time taken to execute the MapReduce process for the same program took less time on physical machines than the VMs.

What can be the reason behind this?

I found some reasons behind it after spending some time on it in my case. Please let me know your cluster configuration in details, more importantly virtualization technology.

Sorry for the late reply.

On VMs the configuration is , single processor, 2 GB RAM and 20 GB HDD

Whereas in case of physical machines , Core 2 Duo PCs, 4GB RAM and 10GB HDD.

Please let me know what must be the reason behind CPU time taken to execute the MapReduce process for the same program took less time on physical machines than the VMs.

Actually , I feel  its like all the resources get the chance to be used fully on physical machines and which is not the case in VM.

Do let me know the reasons behind this, if you all feel something different.

Thanks in advance.

Actually I want to know the virtualization technology/tools used by you in your cluster.

I am using the VMware or Oracle  VM Virtual Box.

I have installed the various DFS and then on them Hadoop has been configured to execute mapreduce program using certain plugins.

@Sudhakar Singh, request you to mention the reasons here which you have found W.r.t Cluster Configuration Details. Might be helpful for me.

Thanks in advance.

VM Virtual Box or VMware are used to run on a Host Operating System. It creates a double layer communication. Better to use Virtualization tools that work as operating system i.e. directly interact with hardware rather than via a Host OS.

There might be other underlying factors like storage read and write speeds.