I have been playing around with Oracle Ravello Cloud and my colleagues asked me about this Cloud platform. I thought of writing this post to give an overview of the Oracle Cloud and hope you can get more detailed overview of the platform and it’s features. I had to read few whitepapers and thought of summarizing the key areas that I found.

If you are running your workloads in a hybrid cloud solution you need to keep your VMs running same as your on-premises datacenter. To achieve this Oracle Ravello Cloud built on a distributed hypervisor infrastructure called HVX. It enables enterprise to encapsulate a multi-VM application run it on any cloud platform including Oracle public cloud, Amazon Web Services and Google Cloud.

Key Features of Oracle Cloud:

• VMware/KVM as is: complex applications run on public clouds exactly as those applications run in the datacenter
• Up to 14x better performance on Oracle Cloud Infrastructure: boost the application performance with hardware assisted nested virtualization, and even more when running directly on OCI (Oracle Cloud Infrastructure)
• L2 networking access: Keep the same IPs, subnets, VLANs; access VMware native NICs, unlimited NICs per VM, multiple IPs per NIC and setup VxLANs – on leading public clouds
• Blueprinting – snapshot and share: Snapshot complete environment configurations, including the state of disks, memory, network of all application VMs to create a blueprint
• Automation: With Ravello’s rich REST API, users can automatically spin up complete environments application blueprints
• Permissions and ephemeral access: Ravello enables users to set permission rules, as well as provide ephemeral access tokens to users within and outside of the organization. Users can share environments and collaborate with others, while they monitoring the spread of usage of application environments

HVX: Virtual Infrastructure for the Cloud

The HVX Infrastructure consists with three main technology components and a management layer and offering a Software-as-a-Service (SaaS) to the customers. The Management layer handles the technology components provides the user interface and API, handles the image management and monitoring.

These are the three main technology components:

• High-performance nested virtualization engine—also called a nested hypervisor
• Software-defined network
• Storage overlay

HVX: Technology Architecture Diagram

High Performance Nested Virtualization

HVX nested hypervisor is capable of running any unmodified guests on top of already virtualized hardware with three main nested virtualization implementations

• Hardware assisted nested Virtualization
  • OCI allows multiple VMs to share the same underline hardware using the Virtualization Extensions. HVX leverages the hardware assist CPU instructions to improve the performance of the workloads.

Hardware Assisted Nested Virtualization

• Running directly on bare metal
  • HVX also runs in the bare metal Servers and ability to achive near native performance

Running on Bare Metal Servers

• Software assisted nested virtualization
  • HVX allows to run VMware VMs with great performance on public cloud infrastructure using a software based nested virtualization called binary translation

Software Based Nested Virtualization

Software Defined Networking (Overlay Network)

Software defined networking components resides within the OCI including multiple subnets, routers, and supplemental services such as DHCP, DNS servers and firewalls. If you need you can even create your own components by uploading the appliances to the cloud in this way you can create load balancing appliances and L2/L3 site-to-site or access VPN endpoints.

Dataplane consists with fully equiped distributed switches and routers and network packets intercepted and injected to the switch. For each and every connected device it creates a virtual port and it handles incomming and outgoing traffic. Macaddress learn from this and creates the forwarding table, for broadcasting frames virtual port flood the traffic to all the distributed ports withn the same broadcast domain. The packet transportation between the port done using the local transport of P2P tunnels over UDP.

Control Plane control plane includes distributed Routers, DNS and DHCP services

Overlay Network and P2P VPN Connectivity Overview

Oracle Ravello Cloud SDN

Overlay Storage

HVX abstract object storage and various types of block devices into local block devices which are exposed directly to the guest VMs. VMs feel like it is running in the same datacenter and VM’s device type and controller type to the location on the PCI bus remains the same.

OCI Storage Overlay Overview

For best performance VM image cashing stores the images locally and OCI provides the benefit of leveraging the Overlay Storage allowing to run any iSCSI, NFS, CIFS and SMD file system to the VMs. From the backend it provides the multi-VM snapshot features. HVX provide the various cloud storage primitives from various storage providers such as magnetic, provisioned IOPs and SSD based storage. Any VM can utilize it’s own CD-ROM to install its guest operating system from an ISO which was uploaded to the Image library.

Use the attached CD-ROM to install any OS from an uploaded ISO file

Resources : 

• Oracle Ravello Cloud
• https://cloud.oracle.com/en_US/home

To mitigate the risk in your VMware environment you need to have a properly configured Security Zones in your Network. Your end users should not have access to the Management interfaces and it should be separately operated and managed. Keep Virtual Machine workloads in the properly configured security zones can mitigate the security threats in your network. That’s where DMZ (Demilitarized Zone) configuration comes in to play. Let’s briefly discuss what are the features of DMZ architectures in VMware.

There are three main DMZ Architectures: 

• Partially collapsed DMZ with separate physical trust zones
• Partially collapsed DMZ with separate virtual trust zones
• Fully collapsed

Let’s discuss what are the features of these Architectures

Partially collapsed with separate physical zones

Advantages:

• Simpler, less complex configuration
• Less changes to the physical environment
• Less change to separation of duties, less change in staff knowledge requirements
• Less chance of misconfiguration due to the less complexity

Disadvantages:

• More Physical resources required (ESXi hosts/Clusters) for each and every zone – Less consolidation
• Higher costs for physical resources
• Complete physical separation of different application types and risks
• This method is not an optimal solution, there are separately allocated physical hosts and not shared or consolidated VMs – incomplete usage of the Virtualization concepts

Partially collapsed with separate virtual zones

Advantages:

• Different Zones in a single ESXi host – fully utilization of resources
• Full utilization of the advantages of Virtualization
• Lower costs
• Firewall Separation done only in the Network layer

Disadvantages:

• Greater chance of misconfiguration
• More Physical NIC cards will be required to connect to the each and every security zone
• This is a better use of Virtualization concepts when comparing to the Partially collapsed with separate physical zones
• This method is more complex and error prone configuration
• If there is any accidental connectivity to a different zone put your environment in a danger situation
• Regular audits might be required

Fully collapsed

Advantages: 

• Lower Cost Option
• There are no physical firewall in between the security zones, only a virtual firewall appliance is configured – full utilization of resources
• Virtual firewall handles the network segregation
• Management of entire DMZ and network from a single management workstation

Disadvantages:

• Greatest Complexity
• Requirement of explicit configuration of separation of duties to help mitigate risk of misconfiguration (Regular audits required)
• Proper configuration required otherwise loss of functionality of the system

In these three methods Virtual Machine network traffic is separated from the Management traffic. Also there should be a proper Change Management and Security Auditing methods for any sort of changes in these environments.

When you connected the NFS Datastores with NetApp filers you can be seen some connectivity and performance degradation in your Storage, one best practice is to set the appropriate Queue Depth Values in your ESXi hosts. If you search over the internet you might be able find lots of issues encountered in the ESXi and NFS environments. I would like to recommend to check this with the storage vendor and get the best fit for your NFS Queue Depth value.

These symptoms can be seen in your environment if you incorrectly configured the NFS Queue Depth Value: 

• The NFS datastores appear to be unavailable (grayed out) in vCenter Server, or when accessed through the vSphere Client
• The NFS shares reappear after few minutes
• Virtual machines located on the NFS datastore are in a hung/paused state when the NFS datastore is unavailable
• This issue is most often seen after a host upgrade to ESXi 5.x or the addition of an ESXi 5.x host to the environment

These are the simple steps to set the NFS Queue Depth Value in your ESXi hosts. It’s an ESXi host level advanced parameter.

Select the ESXi Host and Go to the Advanced settings under Configuration tab

Once you get the Advanced Settings window select NFS and scroll down for the NFS.Max QueueDepth parameter. You can see the minimum value of this Queue depth is 1 and maximum value as 4294967295. Set the Value you want. In my case it was a NetApp filer and VMware and NetApp both recommended to set the Queue Depth to 64

Once you change the value you have to reboot the ESXi host to apply the changes.

You can also use the esxcli command to adjust the value

Use esxcfg-advcfg -g /NFS/MaxQueueDepth to see the current value and esxcfg-advcfg -s /NFS/MaxQueueDepth to set the value

References:

• VMware KB Article (2016122)
• NetApp KB Article

VMware supports to reclaim the deleted thin provisioned Storage blocks from UNMAP commands also known as SCSI UNMAP command. This VAAI storage reclamation primitive feature has been introduced in vSphere 5.0 for the first time. This feature was introduced to reclaim deleted blocks effectively from the storage. Initially it was designed to reclaim the space from the storage soon after deleted or migrated a Virtual Machine from the Storage, but there were some issues in this approach and VMware advised to disable this feature in 5.0.

This automatic reclamation process was disabled in ESXi500-201112001 (ESXi 5.0 Patch 02) and ESXi 5.0 Update 1 which means storage vMotion, deletion, snapshot consolidation or any other process stopped the automatic storage reclamation after this patch version and 5.0 U1. It was a manual process in vSphere 5.5 and now it has been automated in vSphere 6.0 with improvements.

What is the meaning of Storage Reclamation in a simple word?

Once you perform an operation like Virtual Machine Deletion, Storage vMotion or Snapshot Consolidation you free up the space of the VMFS datastore from the thin provisioned virtual disks. These operations leave blocks in the storage array allocated and shows as allocated in the capacity. This is due to the array does not know these blocks have been deleted and cleared from the space, storage has to release these blocks to show the actual deleted space in the storage. These UNMAP commands manually inform to the storage array that there are some deleted blocks in the storage array and you are free to clear up the space.

As I mentioned, this was automated in vSphere 5.0 and VMware stopped after experiencing some serious issues. Now again, this has been automated in vSphere 6.5. In vSphere 5.5 it was a manual process and introduced a new command.

esxcli storage vmfs unmap <Volume_Label or UUID>

Previously, in 5.1 it was using vmkfstools -y <percentage_to_reclaim> (however, you won’t be able to see the options in vmkfstools –help option), I have been using these two commands in my environments as I had to work with both 5.1 and 5.5 ESXi hosts.

Let’s see how we can execute these commands in vSphere 5.5 and 5.1 ESXi hosts.

Executing SCSI commands in ESXi 5.5 Host to reclaim the deleted blocks

Open a putty session to an ESXi host and execute the esxcli storage vmfs unmap -l <datastore_name>

I had to cut off the datastore name

This command will take sometime to complete the operation and it depend on the size of the datastore. Once you execute the command .asyncUnmapFile file will be created in the datastore. This file will delete after completion of the operation. In case if the operation is interrupted before the completion, this file left in the datastore and you have to run the command again against the datastore, file will delete once it complete the operation. This is the only option to check the status of the operation.

Executing SCSI UNMAP commands in ESXi 5.1 ESXi hosts

As I mentioned this esxcli commands introduced in vSphere 5.5 and vmkfstools is the command for this UNMAP operation. Change the directory in to the datastore which you need to reclaim and execute vmkfstools -y <percentage_of_the_deleted_blocks_to_reclam> (vmkfstools -y 60)

esxcli command will not work

VMFS 6 new features with automatic space reclamation

In vSphere 6.5 you have the option to set the reclamation priority in the datastore. However you can disable and set the priority only to Low for the reclamation

If you need to turn this automatic reclamation feature off set the priority to None from Low.

You can easily deploy your Lab with this Oracle Ravello Cloud and I deployed my VMware VSAN datastore on top of this nested virtualization platform. I thought of writing this article after completion of the successful VSAN deployment on Oracle Ravello Cloud.

Initially I built a cluster VSAN cluster with six ESXi hosts for the deployment and I placed my Management components in a management cluster. Installed the ESXi on the hosts and made them ready for the deployment.

Specification of my ESXi hosts:

• 4 CPUs AMD Athlon Processors
• 16 GB of Memory
• 400 GB HDDs (3×100 GB for VSAN and 1×100 GB for OS)

I have created a separate switch for the VSAN network and this is how it looks like

As we all aware VSAN needs added disks to the ESXi hosts so I have added disks from the Ravello Console. It is a quite easy and you can add disks from few clicks.

After that updated the published application, I didn’t want to start my Virtual machines so I deselect the Start all new VMs automatically option at this time

So Now my Oracle Ravello Cloud environment is ready for the deployment. Once I powered on the Virtual machines this is how it looks like. I’m not going to show you all the complete steps of configuring the VSAN just wanted to touch the base of the VSAN configuration in this post.

I created a Distributed Switch and created the relevant portgroups and connected the ESXi hosts to the Distributed Switch

So I Turned on the VSAN on the cluster

I moved to the Old flash client as VSAN feature is not available yet in my HTML 5 client

Validated the Network and Claimed the disks

Finally, My VSAN datastore appeared in my vCenter server.

Previous posts about Oracle Ravello Cloud

• Build your Lab with Ravello Cloud – Experience the true benefit as a vExpert
• Nested Virtualization: VCSA 6.5 deployment on Oracle Ravello Cloud
• Nested Virtualization: Oracle Ravello Cloud Limitations and VCSA 6.5
• Nested Virtualization: Oracle Ravello Cloud – Infrastructure Technical Overview

In vSphere 5.1 and earlier version of VMware vSphere there was only one single TCP/IP stack for all the traffic types such as management, vMotion and virtual machine traffic. Because of this common TCP/IP stack method all the configured vmkernels had to use some common parameters like, same default gateway, memory heap, ARP and routing tables. This was a limitation of this edition and it ended up some common issues.

As a result of finding solutions for those issues VMware allowed to create multiple TCP/IP stacks with vSphere 5.5. After releasing the vSphere 6.5 VMware created a separate TCP/IP stack for vMotion and provisioning traffic other than the default stack. But still there is no option in the GUI to create a custom TCP/IP stack and we need to use the esxcli commands in the command line interface to create a custom stack. But still there are some limitations.

Let’s see how we can create a custom TCP/IP stack in VMware

Use esxcli network ip netstack list to view the available TCP/IP stacks and use esxcli network ip netstack add -N “<tcp_ip_stack_name>” to add a new TCP/IP stack

What are the Limitations?

Unfortunately, custom TCP/IP stack aren’t supported for management traffic, fault tolerance traffic, vSphere Replication and VSAN traffic. When you select a created custom TCP/IP stack traffic, those options automatically disable themselves (see below screen capture when adding a vmkernel) at this point you can use this custom TCP/IP stack only for IP based storage like iSCSI and NFS.

That’s it for now. Hope you enjoyed my post!

VMware latest releases ship with the vPostgres database as their embedded database. If you are planning to migrate your current VMware environment you will get this as an option. If you are not sure with the next upgrade and the recommendation try vSphere Topology and upgrade planning tool. You can read my previous article from here.

So then you will have to work with this VMware embedded database and you need to use these commands. I believe this article will help you to interact with this embedded database. First of all you need to open a ssh session and connect to the shell

View the VCSA 6.5 embedded database configuration information

Use:

less /etc/vmware-vpx/embedded_db.cfg

command to view the configuration

You can see the details such as database type, database server, database port, database instance, database user, etc

To connect to the vCenter database :

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

you can see the release version after that

Execute:

SELECT * FROM VPX_VERSION;

to check the VCSA Product Version

Execute:

\du

to view the defined user roles in the database server, in the below output we have four users created. “postgres” is a super user.

To view all (describe all) the tables use :

\dt

To describe a specific table use:

\dt <table_name>

To turn on\off Expanded display use (see the described table command below as an expanded command) :

\x

To view the list of available functions use:

\df

To display the overall size of the vPostgres database in MB execute the below query:

SELECT pg_size_pretty(pg_database_size('VCDB'));

To display the Table size of the VPX_EVENT in KB use:

SELECT pg_size_pretty(pg_total_relation_size('VPX_EVENT'));

To list all the databases use:

\l+

I hope you might need to get a good understanding about these commands when you are working with vCenter vPostgres embedded database and my post will help you to get familiar with the vPostgres database.

A word of advice…

Appreciate your input. Not gonna touch in production unless we have a special reason, also without VMware premium support advice

— Aruna Lakmal (@techcrumble_net) November 16, 2017

VMware has launched the VMware on AWS at the VMworld 2017 and it leverages the VMware’s powerful software capabilities along with the AWS’s elastic bare metal infrastructure as a Cloud service. VMware has launched a new pricing packages as a on-demand, 1 year and 3 year reservation subscriptions.

Basically two pricing options are available:

On-Demand Pricing

• Pay for physical hosts per hour
• No upfront cost
• Hosts can be scale up and down and you have to pay for the up time hours

Reservation Pricing 

• Longer term reservations for 1 year and 3 year commitments
• 50% cost savings compared to the On-demand package for the same time period
• Pay upfront for the reservation

What features are covered in these Pricing packages?

VMware SDDC vSphere software packages (vSphere, VSAN, NSX), AWS infrastructure and associated support costs.

What features are not covered in these Pricing packages?

Data transfer and elastic IP address charges are not included

Data Transfer charges:

• Data transfer IN to VMware Cloud on AWS from internet: $0.00/GB
• Data transfer IN to VMware Cloud on AWS from another region: $0.00/GB
• Data transfer IN/OUT/BETWEEN same Availability Zone: $0.00/GB
• Data transfer IN/OUT/BETWEEN different Availability Zones using elastic IP or ELB: $0.01/GB
• Data transfer OUT from VMware Cloud on AWS to internet: $0.05/GB Data transfer OUT from VMware Cloud on AWS US West (Oregon) to another region: $0.02/GB

IP address charges:

• Elastic IP address associated with a running instance: $0.005/IP/hour
• Elastic IP address not associated with a running instance: $0.005/IP/hour
• Elastic IP address remap: $0.1/IP/hour

Note: You can redeem your HPP or SPP credits for these subscriptions

What is Hybrid Loyalty Program?

VMware offers up to 25%  discount for your active licences for number of eligible SDDC products. These discounts calculated automatically and there is no requirement of convert or trade in your licences. This is only applicable for reservation subscriptions not for the On-demand subscriptions. On-prem 2 CPU licences provides discounts for 2CPUs on VMware on AWS.

Eligible On-prem products and discounts 

Reference:

• https://cloud.vmware.com/vmc-aws/pricing

You might aware that the book Released by @FrankDenneman and @NHagoort in this year. One of the great book that you can learn about vSphere 6.5 resources. Now you can download this as a free e-book.

Click on below Download button and get your free e-book from Rubrik page.

The VMware vSphere 6.5 Host Resources Deep Dive is a guide to building consistent high-performing ESXi hosts. Written for administrators, architects, consultants, aspiring VCDX-es and people eager to learn more about the elements that control the behavior of CPU, memory, storage and network resources.

Today I just received the VMware vExpert Swag bag from FedEx Delivery. VMware just sent this as a token of appreciation of the work that has been done by the vExperts. That was really awesome…! Thank You VMware for the gifts.

vExperts were able to collect the gift at the VMworld and I was not that fortunate to participate the VMworld in this year. But anyway Thanks to VMware and the vExpert team for sending me these gifts.

Just got my VMware vExpert Swag bag. Thank you #VMware #vExpert #techcrumble pic.twitter.com/9Gz92PdMwm

— Aruna Lakmal (@techcrumble_net) December 2, 2017

VMware vCenter Appliance is the next generation vCenter server and this VCSA 6.5 Appliance builds have been replaced due to a deployment-impacting issue as of 14th of November 2017. This is only for the VCSA and not intend to the Windows based vCenter Server server.

To resolve this issue your VCSA should be upgraded to the 6.5f or 6.5 U1c (or later). This This does not affect VCSA/PSC 6.5GA or 6.5.0a deployments prior to October 22, 2017 and appears only restoring to 6.5GA or 6.5.0a versions. You can read more about on this with VMware KB51124.

Also you can see this warning when you are going to download the VCSA 6.5 appliance.

References:

• Full Patch for VMware vCenter Server Appliance 6.5 Update 1 (2149879)
• Deploying, upgrading, migrating or restoring to vCenter Server Appliance 6.5 GA or U1 release fails (51124)

I’ve been busy with the vSphere 5.5 to 6.5 upgrade work with last month and didn’t a chance to upgrade my blog. I thought to write an article to show the step by step guide to migrate the Windows based distributed vCenter server to a vCenter 6.5 Appliances.

I wrote the similar article to show the steps to migrate to a Windows vCenter Server and you can read it from here.

First of all, If we are migrating a distributed vCenter server 5.5 installation to a vCenter 6.5 Appliances you need to have at least two appliances to your PSC and vCenter server. Here is what I’m going to guide you in this post I have separate Windows based servers for SSO, Webclient and vCenter server.

I was involved with this upgrade plan and had to do lots of KB article readings and Compatibility checks before we start the upgrade. Since, this has a separate SSO server in vSphere 5.5 we need to migrate the SSO server in to a PSC Appliance first. Then we need to proceed with the vCenter Server migration.

Migrating the Single Sign-On Server to a Platform Service Controller

As you may aware vCenter Appliance deployment is a two steps process. Firstly, deploy the appliance and secondly, migrate the services. Those steps are applied to this upgrade as well.

Download your VCSA 6.5 Appliance from the VMware website and mount it to the Windows based SSO server.

Quick Tip….

If you are downloading or upgrading to the VCSA 6.5 make sure to download the 6.5f or 6.5 U1c as VMware has replaced the builds as of 14th of November 2017. Read my post here.

Browse the ISO file and go to the migration-assistant folder and run the VMware-Migration-Assistant.

It will run the application and prompt you to enter the Single Sign-On password

Type the SSO password and hit enter to continue

It will prompt you to start the Migration and do not close this window until you complete the migration, don’t worry once it completes the migration your SSO server will be powered down automatically

Now go back to the vcsa-ui-installer/win32 folder double click on the installer 

Once you get the vCenter Server Appliance Installer window click on Migrate option

Click on Next 

Accept the EULA and click Next

Provide the sso server FQDN and the SSO password do not change the Migration Assistant port and click Next

Verify the thumbprint of the server

Provide the vCenter server or an ESXi host FQDN and relevant username and password (I use my sso username and password for the migration) click Next to continue

Accept the Certificate Warning and let it to connect to the server

Select the target VM folder in the inventory and click Next

Select the target compute resource (ESXi Server ) and Click Next

Provide the root password for the appliance and click Next

Select the target datastore, since this is a test environment I’m enabling the thin disk mode and click Next to continue

Configure the IP settings and you have to use a temporary IP address for the data migration, it will not be using this after the completion of the migration, click Next to continue

Review the configuration and click Finish to complete the Migration Stage 1 which is the appliance deployment

Installer will initialize, deploy and powering on the appliance, install the relevant RPM and complete the deployment after that.

Once it completes the Appliance deployment, click on Continue to start the Stage 2 of the Migration, you can close the window and continue this after login to the https://<temporary_IP_address>:5480 url

Installer will start the stage 2 of the Migration and click on Next in the Introduction window

It will perform the Pre-migration checks before start the data processing and Migration

In the next step we need to have an AD account which has the permissions to add this machine to the domain. Provide credentials and click Next and it will validate the account and this new PSC will be added to the domain

Configure the Customer Experience Improvement Program (CEIP), I encourage you to select this in the production for the future improvement and click Next to continue

In the next step you have to accept the the backup availability of the source SSO server and the required data of the database, click Finish to complete the Migration

You will get the warning message to power down the source SSO server after initiating the network configuration in the Appliance, click Yes to accept the warning

It will start the migration and power down the source SSO server and complete the Migration

Now you will be able to login to the PSC with the old SSO FQDN

Now we have partially complete the 6.5 Migration and we need to start the vCenter server Migration

Migrating Windows based vCenter 5.5 to a 6.5 appliance

Basic Migration procedures are similar to the above steps, I’m not posting all the screen captures as this is going to a lengthier post (it’s already). Follow the below steps to proceed with the migration.

Mount the VCSA 6.5 ISO

Run the Migration assistant as mentioned in the above

Provide the SSO Password

Run the installer in the vcsa-ui-installer/win32 folder

Select the Migration Option

Click Next in the introduction window

Accept the EULA and click Next

Provide the source vCenter server and the SSO password, again do not change the Migration assistant port

Verify the thumbprint by Clicking Yes

Provide the deployment target and the credentials, here I’m using the source vCenter server and the SSO admin credentials. Click Next to continue

Accept the certificate Warning

Provide the target folder, target compute resource in the next steps

Provide a root password for the appliance and click Next 

Select the Appliance deployment size according to your environment in the next step and click Next to continue

Provide the target datastore and you can enable thin disk mode for the testing purposes

Provide the temporary IP address details and click Next

After reviewing the configuration click Finish to complete the Stage 1 of the migration which is the appliance deployment

Let the installer to complete the deployment and click Continue to move to the stage 2 of the migration, similar to the SSO deployment we can close the installer and can continue with the temporary IP and the 5480 port url for the rest of the migration

Click Next in the Introduction window

Provide an AD account credential to add this appliance to the domain

In the next step select the migration data that you want to copy from the source vCenter server and click Next to move forward

We have to accept the backup availability of the source vCenter server and the required data from the database and click Next

Warning message will display to power down the source vCenter server after the completion of the network configuration, click OK to start the migration

Let it to complete the data transfer and the appliance setup

if this window closes before it complete the data migration open the appliance management url and you can continue with that, I had a similar issues several times

Now we can access the vCenter server appliance with the same old vCenter FQDN. Note that we are now not able to use the vSphere C# client and we can use only the web client to access the vCenter server

That’s it now we have migrated the Windows based vCenter server to the vCenter Appliances. We have two appliances for PSC and vCenter server.

We have experienced this in few environments and I thought to write this in my blog. Firstly, I’d like to share the symptoms that we experienced in this case before we discuss the fix.

We have been informed from one of our teams that Virtual Machines were not responding and unable to check the OS status from the VMware console. It was a bad situation and we had to check the status of couple of Virtual Machines and we were able to identify this error message in all of the Virtual Machine consoles “Unable to connect to the MKS: Virtual machine config file does not exist.” Also we have noticed that those Virtual Machines were residing in the same ESXi host and the LUN. Here is the message we could see in the Virtual Machine.

We tried to browse the datastore and it was not showing the content of the datastore properly.

After that checked the vmkernel log and identified there are IO reservations errors for few LUNs of the ESXi host

Here is the complete output of the log

This LUNs were locked by one of other ESXi hosts and lock was not released properly and it leads this to this unresponsive state. So we had to reset the LUN to find a fix for this.

One of the great tool that we use for lots of reasons is “vmkfstools” command line interface commands. So it had to pop up to fix this.

“vmkfstools -L lunreset /vmfs/devices/disks/<naa_id>” was issued to fix the issue lun naa ids were displaying the vmkernel log and also you can easily find those. Here is the command example

Unfortunately, there were no identified improvement with the situation and we used vml ids instead of the naa ids. To find the vml ids used “esxcfg-scsidevs -l” command, here is the similar output and you need to identify the affected LUN in the output.

Executed the LUN reset with the vml id

Finally, LUN came online and was able to browse the datastore. Virtual Machines started to response and some Virtual Machine rebooted to fix some issues.

Reference KB: 1000044

It was announced that the CPU data cache timing can be abused by the software layer and can lead to an information security vulnerability. This is a direct impact to a shared resource utilization and below variants have been identified by Google Project Zero and other associated researchers.

These are the variants :

• bounds check bypass – a.k.a. Spectre
• branch target injection – a.k.a. Spectre
• rogue data cache load – a.k.a. Meltdown

VMware hypervisor affected only with first two variants (only with Spectre) and at the time of writing this post we do not need to worry about the  third variant.

There are three type of mitigation categories in relation to VMware hypervisors

• Hypervisor-Specific Mitigation – Mitigate the information leakage from hypervisor or guest Virtual Machine to a malicious guest Virtual Machine which is running in the same host.
  • Affected Products: 
    • VMware vSphere ESXi (5.5,6.0 and 6.5)
    • VMware Workstation (12.X and 14.x)
    • VMware Fusion (8.x and 10.X)
  • VMware Security Advisory ID: VMSA-2018-0002
  • Patches for ESXi: 
    • 6.5 – ESXi650-201712101-SG
    • 6.0 – ESXi600-201711101-SG
    • 5.5 – ESXi550-201709101-SG (Only address to the branch target injection)
  • Patches for Workstation: 
    • 14.x – Not affected
    • 12.x – 12.5.8
  • Patches for Fusion: 
    • 10.x – Not affected
    • 8.x – 8.5.9

• Hypervisor-Assisted Guest Mitigation – It virtualize a speculative-execution control mechanism to a guest VM. So mitigation requires a specific microcode patch from OS or the processor firmware/BIOS vendor
  • Affected Products : 
    • VMware vCenter Server (VC) – (5.5, 6.0, 6.5)
    • VMware vSphere ESXi (5.5,6.0 and 6.5)
    • VMware Workstation (12.X and 14.x)
    • VMware Fusion (8.x and 10.X)
  • VMware Security Advisory ID: VMSA-2018-0004
  • Patches for vCenter:
    • VC6.5 – 6.5 U1e
    • VC6.0 – 6.0 U3d
    • VC5.5 – 5.5 U3g
  • Patches for the ESXi: “All the ESXi patches associated with VMSA-2018-0004 have been pulled back from the online and offline portal”. These patches issued(ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG)along with the Microcodes and issues appeared after patching the OS, so ”
    For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead”.
  • Patches for Workstation:
    • 14.x – 14.1.1
    • 12.x – 12.5.9
  • Patches for Fusion:
    • 10.x – 10.1.1 (OS X)
    • 8.x – 8.5.10 (OS X)

• Operating System-Specific Mitigations – This mitigation should be done with the OS vendors.

References: 

• VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245)
• VMSA-2018-0002.2
• VMSA-2018-0004.2
• Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004 (52345)
• https://meltdownattack.com/

I was stuck with my office work and didn’t get a chance to write couple of posts in my blog, so after a while I thought to write this useful article for all my it folks. I’m going to show the complete steps to reset Dell ESXi iDRAC passwords. Normally, anyone has a password vault but there might be some scenarios which you might need to break the credentials and login to the iDRAC.

Here I’m going to use the Dell OpenManage Server Administrator (OMSA) to get access to the iDRAC which is not accessible at the moment.  You have to download and install this in one of your Windows Server and please note that this article intended only for VMware ESXi hosts.

First of all we need to download and install the OpenManage Server Administrator vSphere bundle (VIB) and install it to your ESXi. You can use your Update Manager to install this as an extension to the ESXi or multiple ESXi hosts. These are the VIB versions at the time of writing this article and follow this link to download them.

I have downloaded and uploaded it to my VMware Update Manager, created an extension patch baseline (I hope you already know to create a patch baseline, not a magic work). Here is a sample screen capture

I remediated, pushed and rebooted the ESXi host.

Download and installed the OMSA in one of the Windows servers, select the appropriate version (download them from the above link)

Run the installation and follow the instructions, not a big deal

You can see this icon on your desktop, double click and open the OMSA

It will open your web browser and lead to the OMSA login page

Provide the root credentials of the ESXi and ignore the certificate Warning!

Credentials will be verified and installed VIB will get you through to the OMSA console with the root credentials

Goto Remote Access -> Users and select one of the available number to create a new user for the access, I don’t want to reset the password for the existing user in my case

Provide a name for a new user, password and set the role for the user and privileges and save the configuration (Save button is not in my screen capture)

That’s it and check the new login for the iDRAC access.

With the beginning of the year 2018 Oracle Ravello has decided to start the monthly Community call and I’m so excited with this. Here I’m sharing the YouTube video call for all my followers and readers.

The goal behind this is to share the real-life experiences, use cases or exciting projects out to the community at large.

This is a good start and keep up the good work Oracle Ravello…!!!

You can read some blog articles and videos from vExperts all over the world from the below link, just spend sometime and get familiar with those.

Follow below link to read more about Oracle Ravello bare metal cloud infrastructure

I was about to execute couple of PowerCLI scripts and I encountered some strange issue while trying to connect to my vCenter server. I was using PowerCLI 6.5.1 which releases as a PowerShell module and I stopped using the old installed PowerCLI prompt.

There were no changes to the computer which I was using and suddenly it stopped the functionality and was throwing this error message when connecting to the vCenter using Connect-VIServer command. Here is the error message:

I searched the reason behind this and I was able to find that one of my colleague has installed the old version of PowerCLI command prompt in the same computer.

I uninstalled the older version, closed the PowerShell window and tried to connect again. Fantastic! It was connected.

Actually, this was not quite a troubleshooting but spent some time before I started the actual planned work. So I hope this will help someone to get things done easily without wasting time.

If you are running a business critical workload in a VMware Virtualized datacenter proactive analytics, fast and environment specific remediation and recommendations are one of the key factors come in the first place. VMware is trying to achieve these measures with the innovative support technology and trying to deliver the maximum visibility of the datacenters to their customers.

Recently, VMware released VMware Skyline as one of the best proactive measures to address the above requirements and deliver the break-fixes to these issues before they actually appear in the customer datacenters.

Key benefits of VMware Skyline:

This VMware Skyline solution includes components in customer site and the VMware cloud.

Customer Site : VMware Skyline virtual appliance collector collects details such as configuration, features and performance data. Appliance must be registered with the VMware account and limited customer interaction required.

VMware Site: VMware cloud platform receives the data from the collector and performs the analysis such as determine alignments with the VMware best practices. VMware compare the licences and determine whether there are any known issues and bugs in the system and provide the proactive measures before it appear in the customer’s infrastructure.

VMware Technical Support Engineers (TSEs): VMware engineers use these collected data to provide proactive support to the customers along with the best practises.

VMware Skyline Deployment

VMware Skyline Collector deployment is not a big deal and it’s a simple appliance deployment in your in house VMware vCenter and ESXi environment.

Download the VMware Skyline appliance from My VMware download page. Click the Download button to download the appliance.

Once downloaded the OVF appliance deploy it as a generic appliance deployment. I’m not showing all the steps of the appliance deployment in this post, only the important steps are there.

Accept the Licence agreement and click Next

Provide the root password and the IP address configuration

Review the summary and complete the deployment

Open the browser and login to the appliance using http://<IP_ADDRESS>/login to login to the Skyline collector and continue the configuration

Click on wizard to continue the configuration in the dashboard

In the first step of the Registration you can change the password of the appliance

Set the proxy if you have one

Enter My VMware account details and validate the account

Successful validation will give you the collector id , click continue to continue the configuration

You can see the registered validated entitlement in the next window, click continue 

You can review the documentation if you want

In the next step you can configure the vCenter server, provide the details such as IP address (host address), vCenter Read-only account (if you don’t have one setup an account with the read-only access), password, SSO admin and STS url, click ADD to add the vCenter server

You can see the details in the dashboard as below and you can follow the same step to add a NSX Manager to the Skyline collector in the same way

This is the sample VMware Skyline report (image courtesy VMware)

I haven’t got the actual report yet. I will post it soon

References:

• VMware Skyline Datasheet
• VMware Skyline Solution Overview
• VMware Skyline FAQ

I was doing some SSL certificate updates in one of my VMware environments and I had to face couple of issues while I’m updating the SSL Certs. Issue appeared in Windows based vCenter 6.0 U2 server. We had some webclient login issues which I described in this article. In terms of finding a solution we were regenerating the SSL solution user certificates in the vCenter server. To find out how to regenerate and reset all the SSL certificate read the second part of the article.

SSL regeneration process was failing and Rolling back at the end of the process as you can see in the below screen capture.

So I re-run the process and window closed without throwing any error and I was confused with this status, I checked the certificate manager logs which is located in “C:\ProgramData\VMware\vCenterServer\logs\vmca\certificate-manager.log” in a Windows based vCenter server. You can find this error log in VCSA in “/var/log/vmware/vmcad/certificate-manager.log” location.

I was able to see the below error message in the log and I located the .cfg (SSL Configuration file location) in the log

So I browsed the configuration file location and I could see below .cfg files

I moved all the files to a new folder, I name it as “old-cfg”

I re-run the certificate regeneration process described in the second part of this article, and this time it was succeeded

It was able to fix the error and re-generation process succeeded.

This is another error appeared in one of my SSL certificate regeneration tasks. I was regenerating the solution user certificates and certificate regeneration was failed and performed the automatic Roll-back operation. If you need to know the steps to regenerate the solution user certificates please read the second part (first part explains the time sync issue) of this post.

It was bit confusing and I had to spent lot of time to find the proper solution for this error. Again, once I encountered with the error I checked the certificate error log located in “C:\ProgramData\VMware\vCenterServer\logs\vmca\certificate-manager.log“. I was able to see the below error message and the output.

I generated the solution user certificates providing the same name to all the certificates and that caused the issue. These certificates should use with different names and I started the process with the different names (provide different names for all the certificates).

Different names should be used for the “Name” field in all the certs

This time Solution user certificate generation was successful without an issue. Once it completed the process generated webclient certificate will look like this.