I have been working with Windows to VCSA 6.5 Migrations and wanted to share some steps of the VCSA configuration. Most common task was to add the VCSA to the Active Directory Domain and I’d like to show the steps to connect the VCSA from GUI and command line to the Active Directory Domain.

This is a pretty easy configuration, but we had some issues with the domain permissions when Windows to VCSA migration. So we had to connect the VCSA several times to the domain and here are the steps that anyone can follow.

Login to the VCSA webclient, hope you know that there is no more C# fat client for VCSA 6.5 access. Click on the Home menu and select the Administration

Click on System Configuration

Click on Nodes and you can see the vCenter server name unde Nodes, Select Active Directory and hit Join button to add to the Domain

Provide the details such as domain name, Organizational Unit, Username and Password to connect to the Active Directory

You need to reboot the VCSA to display the domain, without a reboot this change will not be applied. Once the reboot is completed. Domain will display under Active Directory section

Connect to the Active Directory using the Command Line

You can use the Command Line interface to connect the VCSA to the Active Directory here are the steps

Login to the shell, type “shell”

Use below command to connect to the Active Directory

/opt/likewise/bin/domainjoin-cli join [domain] [username]

It will prompt you to type the password, type the AD password for the user account that’s it

Proactive analysis is playing a key role in nowadays modern datacenters. Everybody is looking for options and tools which they can manage and identify the coming issues and outages to their datacenters. Runecast and VMware Skyline are two similar products which you can use to do a proactive analysis and take proactive measures to your mission critical workloads.

Recently, I joined to a webinar about the introduction of Runecast and also I was able to work with VMware Skyline as well. So I thought to try and do a review about these two products, hope my readers can take a great insight about these products after reading my post.

I already wrote an article about VMware Skyline and I thought to give a quick introduction about the features of Runecast Analyzer in this post.

So What is Runecast Analyzer?

It’s a tool which brings issue-prevention solutions to the business and minimize the impact to the complex systems can have by giving you the power of prevention measures along with the proactive analysis.

You can start it with a small OVF deployment and configuration can be varied with Small, Medium and Large scale deployments. You can select it in the deployment and it is obvious that virtual hardware resources can be different according to the configuration.

Once it deployed you can connect one or more vCenters to the Runecast Analyzer and after scanning your environment it will give you the status of your datacenter, it has a great interactive dashboard which customer can see the status of their datacenter against the VMware knowledge base.

I will write another article for the Runecast Analyzer review and I’d like to compare Runecast analyzer and VMware Skyline each other in this post.

Runecast Analyzer:

• Is a software product for end-users
• It has analytical engine with runs on-prem on the user side and evaluates VMware setup conditions. It scans vSphere and vSAN, immediately (1-2 minutes max.) shows detected problems. This can be repeated manually anytime or scheduled
• In addition to configuration and hardware details, Runecast Analyzer performs real-time log analysis and identifies issues in the logs immediately and links them to Knowledge Base articles
• Runecast software comes with an interactive dashboard with real-time issues, alerts and reports
• Works fully offline with no need for internet connectivity
• The software runs as a server on a single VM instance, no agents needed

VMware Skyline: 

• Comes as a service. Only for VMware customers under “VMware Premium Support”
• The end-user doesn’t have any visual dashboard/UI showing detected issues, alerts and risk management of connected vCenters
• VMware Skyline uses a “Collector” appliance which needs to be installed on-prem and it scans users’ systems and send findings to the Skyline cloud for further analysis. VMware personnel analyzes the data and generates a report on what is recommended to fix. The report is then delivered to the end-user for further communication. The end-user can’t control the speed of analysis or a time when the report is returned

Runecast Analyzer:

• Runecast Analyzer is frequently updated with VMware Knowledge Base articles for vSphere and vSAN, and support for NSX-V is under the development. It also covers VMware Best Practices for vSphere and vSAN
• The big advantage of Runecast product is the log analysis. It collects syslogs from vCenters, ESXi hosts and VMs and uses smart analysis to discover Knowledge Base problems and also speeds up troubleshooting by point to the trending log events, which are usually related to VMware problems
• In addition to vendor knowledge sources, Runecast Analyzer includes scanning against known issues in validated social media sources (blog articles, forums). In some cases critical issues appear first in social media before being officially documented by the vendor

VMware Skyline:

• VMware Skyline conducts analysis of a user data collected from the collector in alignment with VMware best practices
• Since VMware Skyline processes the users’ data through their support tier, the extra recommendation can come from that

Runecast Analyzer:

• Runecast Analyzer contains a database of VMware security hardening checks and federal DISA-STIG 6 compliance. The Analyzer scans periodically attached vCenters and reports all violations and passed criteria for both security profiles
• Aside from regular Knowledge Base updates, Runecast delivers special and immediate updates (daily basis) in any kind of severe security risk, such as CPU flaws Spectre and Meltdown, which allows admins to very quickly detect vulnerable assets and patch them properly

VMware Skyline:

• VMware Skyline doesn’t claim any specific security feature or compliancy standards

I have been able to log a support request and get the support from Runecast support, so at this point I’m not able to provide a comment on Runecast support.

My Thoughts

Download, installation and configuration was really easy. Connecting to the vCenter server was not a big deal either. Soon after I connected the vCenter I was able to scan my environment and interactive dashboard displayed all my issues and KB articles on the dashboard. I didn’t want to do a google search and find the relevant KBs it directly gave the KB articles, just a simple mouse click away. Also, this tool works without the internet and offline database can be included.

It provides the issue list as with the severity and I was easily identify the most critical and least critical issues in my environment.

Security hardening also one of the key point here and it gives you a greater visibility against the result with the Pass/Fail status, so it gives greater visibility of the environment security

I haven’t got a chance to log a support request and work with the Runecast engineers so I’m not able to comment on their support assistance.

According to my point of view Runecast is one of the greater tools for the Proactive and predictive analysis in your VMware based virtual environment. There is no this type of dashboard in VMware Skyline and also as a customer we are not able to see the real time status of our datacenter, we have to wait until we get the OSR from VMware engineer.

A side by side comparison of Runecast Analyzer and VMware Skyline from a person who tried both products:https://t.co/EppyYJcjq5

— Runecast (@RuneCastBiz) February 23, 2018

If you are planning to perform an ESXi host upgrade you need to find the compatible drivers for your ESXi host. This post is intend to show you the correct method to find the correct driver for your PCI device. If it is a Cisco UCS host you can find the correct enic and fnic drivers for the respective operating system.

Finding the correct driver is really important for the best performance, use of incorrect or the incompatible driver can lead to a performance impact and your production VM workload can be impacted. So, I think this post is quite important.

VMware Hardware Compatibility matrix also known as VMware HCL can be used to identify the drivers according to the hardware specification. We are going to use VID, DID, SVID and SSID to identify the driver

So what are these parameters?

Login to your ESXi host with the root user and execute vmkchdev -l and you will be able to see the similar output as below

So now go to the VMware Compatibility Guide and enter the values as below, select the search object as the IO Devices and hit Update and View Results to find the results. Select the required Hypervisor version and you will get the driver details page

Driver version and the download link will be there to download the driver make sure to check the Model and Driver firmware is correct for your PCI device

This is my Cisco UCS adapter details in my Cisco UCS Manager Console

So now you can find and download the compatible driver for your PCI device.

Also, you can use esxsfg-info | less command to get the same details for your PCI device, here is a sample output

I have been working with couple of vSphere 6.5 upgrades and I experienced this issue while I was upgrading Dell R630 Host in to ESXi 6.5. Host was not updating and getting failing throwing the below error message.

Actually, I have done host upgrades in many times but I did not get such issues with Cisco UCS blades and other ESXi hosts. But I got this error couple of times with these Dell hosts. I’m not sure whether there is a relation to the vendor but this is how I resolved this issue while I’m doing the ESXi upgrade.

I checked the disk space of the host and I noticed that one of the vfat scratch partition is full. I used “df -h” command to check the disk space to get only the vfat partitions use “df -h | grep vfat“. Here is the output.

I executed “du -h /<my_vfat_volume>” to get the breakdown of the utilization

I removed the “6.5.0” folder inside the packages folder. Executed “rm -rf <folder_name>” to remove the folder

Now located a “vpxa-worker-zdump.xxx” file in the /var/core/ folder removed the dump file. Executed “rm -rf vpxa-worker-zdump.xxx” command to permenetly delete the file [Step 7]. See the below detailed steps.

Again, checked the vfat partition space and now it is 0% utilized.

I started the upgrade and it was succeeded after that. Hope this post will help someone who experience the similar issue while doing the upgrade.

We experienced this issue with the template deployment with the custom specification wizard. The important thing here is we experienced this issue soon after the vSphere 6.5 upgrade. Our Virtual Machine deployment team reached us to find a solution after the upgrade.

Previously, they were using the old C# client to deploy the Virtual Machines and with the change of the vCenter this issue came in to the action. I thought to write this post to share and explain the findings and the steps to resolve the issue.

Error occurred while loading data. Check the log files for details

First thing first what are the permission levels required?

Below permission levels should be fulfilled to use the custom specifications for a Role.

Virtual Machine -> Provisioning -> Read Customization Specifications

Virtual Machine -> Provisioning -> Modify Customization Specifications

Also, the created or role that these provisioning users are sitting at should be applied to the vCenter level, it was added to the Datacenter level and I had to change it to the vCenter level as shown in the below screen capture

After that our provisioning engineers were able to see the custom spec scripts

Still not finished…..

“The public key in the specification does not match the vCenter public key. You have to reenter the password in order to proceed”

This error appeared after fixing the above issue

So we had to update the passwords of the templates, follow below steps to update the template passwords

Click on the “Home” icon and select the “Policies and Profiles”

Select the “Customization Specification Manager” and select the template and click on “Edit” icon

Select the “Administrator Password” and update it in the wizard

Click “OK” to complete the change

After that we were able to use the Customization Specification scripts without any issues

If you found this post as useful please rate the post and share it!

This is a bit of strange issue which came after the vSphere 6.5 VCSA migration from Windows based vCenter server 5.5 and 6.0 versions. First of all I’d like to mention that we have tested this migration under couple of scenarios in many times in our test environment and we did not encounter similar issue in one of the tests.  So, we were fully confident with the upgrade and we started this in our Production environment.

In this issue I’m not quite sure whether this is any environment specific issue or not because I was not able to find any online articles on the internet. Also, the opened support request with VMware BCS support is still open. The bad side of this issue is sometimes we lost entire permissions levels and we could not see any added permissions after the completion of the migration, but in some cases we could see the permission but those permissions were not working so we had to re-add them to get the access.

Also, VMware confirmed that there is no other way to restore the permission other than re-add them in the new vCenter server appliance. But, after doing couple of tests we were able to find a work around for this and I’m happy to confirm that this method is now working without issues. Anyone who knows any other method to restore the permissions or any other work around please let me know.

First step was configure existing Active Directory as a LDAP Server Identity Source while we keeping the Active Directory Integrated Identity Source before we start the Migration

Then remove the Active Directory Integrated Identity Source and test the user access. Start the Migration

Once the migration was completed identity sources was similar to this and set the created LDAP identity source as the default identity source

We had to remove the Local Identity source to avoid some connectivity issues and login latency for users.

Once we done these changes migration worked perfectly and we didn’t see any issues after that.

If you found this post as useful please rate the post and share it!

Runecast recently released the v1.7 with VSAN support and it has some greater improvements. The major improvement is VSAN support, and vCenter integrated Web plugin. In this post i’d like to discuss these new features and improvements of this release.

Runecast is a greater solution which can provide the best practises and security hardening measures based on the VMware knowledge base. Customers can have a greater visibility of their datacenters and proactive measures can be taken based on the guidelines and knowledge base articles provided in the dashboard. With the release of v1.7 it has extended its support for VMware VSAN as well.

It has three main added improvements to this release:

Greater visualization of your datacenter with it’s interactive dashboard and it’s widgets, now it has a improved dashboard called VMware Stack View Dashboard.

I’d like to highlight VMware Stack View dashboard as it show you the severity of the issue and where it exist in the layer of your VMware environment. It’s a simple pane of glass to all your vCenter resources.

Runecast Analyzer v1.7 Provides the VSAN best practises and recommendations

VSAN is one of the excellent software defined storage solution provided by VMware which can have greater benefits to its customers with highly performenced and scalable solution. Runecast provide the best practises and recommendations for VMware VSAN against the VMware’s larger knowledge base database. It is critical and should be monitored closely with extra care to avoid the possible future issues which can make a serious impact to your Virtual Machine workloads.

Runecast Analyzer is completely working in the offline mode and does not send any data over the internet, so your data stores in your on-prem datacenter.

Runecast Analyzer v1.7 initial configuration and vCenter Plugin configuration

Runecast Analyzer added another valuable configuration item to this release, vCenter plugin which can see the Runecast analysis status and the recommendations in the vCenter.

Once you deployed the Runecast Analyzer as a generic OVF deployment, you can add your vCenter server to Runecast Analyzer by Add vCenter option. Go to Settings->vCenter Connection and click on Add vCenter

Provide your vCenter details and credentials

You can see the Thank you note after the successful configuration to the vCenter

To install the vCenter web plugin click on the Edit button under Action and select Install plugin option

Click on Install in the next window to install the plugin

Let this installation to complete

You can see the plugin status under settings as Installed

After the successful installation of the plugin, you will be able to see the Runecast option in the Home dropdown menu as well as the Home tab of the vCenter. If you were logged in please logout and login to the vCenter to see the plugin

To setup the plugin click on the icon and you can see Runecast Main View, click on GO TO SETTINGS

Now Runecast API Access Token should be configured to start the connection.

To Setup the API Access Token, go back to Runecast Settings -> API Access tokens and click on Generate API access token

Provide a description , functionality mode and Generate the Access Token

Copy the generated token

Provide the FQDN, Generated Access Token and SAVE SETTINGS

You can see the empty Runecast main view and click on PERFORM SCAN to do a scan of your environment

Wait for couple of seconds and it will pull out the details for you and you can see all these details in your vCenter server view

Another good thing….

Runecast Analyzer has added the Spectra and Meltdown analysis Knowledge Base to its database and you can scan your environment against the latest security vulnerabilities and harden your VMware based environment

My Thoughts

Runecast Analyzer with the new release of the VSAN support is a greater product in terms of proactive analysis of your VMware based virtual environment. There is no doubt of the continuous improvement of the product and adding new features in the future.

From the update point of view if you have the active subscription you have the online auto update but it will work with more secure environment without the internet access using it’s offline database, so your data is completely secure.

If you found this post as useful please rate the post and share it!

Vembu has reached me and gave me an opportunity to test their flagship product Vembu BDR Suite, and I developed a Lab with a Backup server to test Vembu products. Before we start I’d like to give you the background of the company, company was founded in 2002 and currently serving more than 60,000 businesses all over the world.

Initially, Vembu’s product was around SQL Migrations and then they focused on Backup strategies, now it has been over 10 years and company developed with the more capable backup and disaster recovery solutions. Important thing of their products is the number of solutions provided for the different platforms including VMware, Hyper-V, Image backup, Network Backup, Cloud Disaster Recovery, Online Backup, SaaS and Management and Monitoring tools, also Vembu supports for Applications such as Microsoft Exchange, SQL and Sharepoint and Cloud products such as Google Apps and Office 365 and you can store them in offsite and onsite store or even to the cloud. I think below product download portal will give you a clear insight of the capabilities of these products.

You can access and download these products from Vembu Portal, If you don’t have an account you can sign up for the portal account and one of their agents will reach you shortly.

Once you got access to the portal you can see some greater features in the portal, You can see the licenses usage in the Dashboard, also Products, Licenses, Subscriptions, Billing Address, all invoices, BDR Server information, Purchase history and Profile can be update in the same page.

I don’t have License yet for my BDR Server

So Let’s see how we can configure the Vembu VM backup in the VMware Environment

Vembu VM Backup Installation

To Download the VM Backup Click on the Download under Vembu VM Backup

Click on the Download Installer and you can see the Supported Operating Systems which you can install the BDR Suite

Once You downloaded the Setup run the Installer

Installer will start and click on Next in the Welcome Window

Accept the License agreement and click Next

You can customize the configuration by selecting the checkbox and you can see the default configuration

If you wish to change the configuration you can change the configuration in the next step and click Next to continue

Select the Installation folder

Select the Drive letter for the Storage Repository in the next step

Note: C: Drive is not recommended by the vendor

If you select the C: drive as the Storage repository below warning message will appear on the screen

Also, adding a Shared folder (CIFS/NFS) is also supported and NAS and SAN devices connected using NFS or CIFS supported

You can change the Webserver port, admin username and password in the next window. Change the details according to your requirement and click Next to continue the installation

Click Install to begin the installation

Installation will install the necessary packages to your server and Click Finish to Start the services and open the Web Console (You can select these options from the check boxes)

Once it opens the Web Console type your username, password and click Sign in

Select the Time Zone for your BDR Server and click on Save

Provide a unique ID for your BDR Server Vendor recommends MACHINENAME.DOMAIN name for this because it is an unique id, click on Update the submit the changes

Once you submit the changes you can see the configuration option, I’m going to work with my VMware vSphere Lab so I select Add VMware vSphere Server to add my vCenter server

Provide the vCenter Server details,credentials and port to communicate with the vCenter server, click on Save

You can see the added vCenter server after that in the Console

So I’d like to run a backup and restoration procedure in a different post as this post is going to be a long post. So stay tuned, let’s see how we can backup and restore the Virtual Machines and what are the available option in this backup configuration

My Thoughts

This is was the first experience with Vembu backup and it was quite easy process to start work with the product. I was easily create an account in the Vembu portal and soon after I completed the Sign in process I got an email from one of their agents asking a convenient time for a walkthrough of their products and details under Vembu BDR suite. He was willing to help me to find the best choice for my requirement and the environment.

There were few links to their product configuration guides and procedures. Also, a summary of the best practises and and initial access procedures soon after the installation of the product. There were some useful email contacts for technical assistance and the sales. Also, you can request a support assistant during the evaluation when you click on the download, below widget will appear on your screen and you can easily request assistance by submitting the form

From the Installation point of view it was pretty easy task. Anyone who has the software installation experience can easily install the BDR Suite within couple of minutes, there was no separate database configuration Postgres database is installing automatically in the same server, so I did not want to worry about any Database configuration. Soon after the completion of the installation application was ready to use and Web console opened in my screen. It has a simple and user friendly dashboard for the rest of the configuration.

Adding my vCenter was just few clicks and it was added easily, so far according to my experience it was quite simple easy and straightforward to configure the BDR server. Let’s see what are the options of the Backup and Restoration procedures and how feasible these options in my next post. So, stay tuned!

If you found this post as useful please rate the post and share it!

In this article I’d like to show you the complete steps to configure SNMP v1 in the vCenter Server appliance. I have been closely working with vCenter server Migrations to 6.5 in my clients and had to re-configure the SNMP targets after the migrations, hope this article will help someone who try to configure the SNMP in a vCenter Server 6.5 Appliance.

What is SNMP ?

I assume you already know the meaning of the SNMP. These four letters stands for Simple Network Management Protocol which is collecting and organizing managed devices on IP networks. It can use to send device information from one device to another. Most of the network monitoring tools use SNMP traps to monitor the devices and capture the alarms and alerts.

In previous release of the vCenter server there was a GUI to configure the SNMP servers in the vCenter Server settings, but in VCSA you have to use command line interface to configure the SNMP target servers.

Firstly, enable ssh login to the VCSA

Open a shell console to the VCSA

Type:

snmp.get

To view the current snmp settings of the VCSA, initially there shouldn’t be any configuration in the VCSA

First thing, we need to configure the SNMP community to configure the SNMP in the VCSA, use below format to set the SNMP community to your VCSA

snmp.set --communities <COMMUNITY_NAME>

Type below commands to configure SNMP targets, for one SNMP target configure as below:

snmp.set --targets <IP_ADDRESS_OR_HOSTNAME>@<PORT>/<COMMUNITY_STRING>

You can see the example as below

To configure multiple targets use the command as below:

snmp.set --targets <IP_ADDRESS_OR_HOSTNAME>@<PORT>/<COMMUNITY_STRING>,<IP_ADDRESS_OR_HOSTNAME>@<PORT>/<COMMUNITY_STRING>

Once these targets are configured execute the snmp.get command to view the configuration

To set the SNMP log level use below command

snmp.set --loglevel <DEBUG|INFO|WARNING|ERROR>

Finally, enable the SNMP on the VCSA

snmp.enable

SNMP Test and Help

To send a SNMP Test command use the below command:

snmp.test

To View the SNMP Statistics, use below command:

snmp.stats

You can also view the available options and configurations, type “snmp.” and hit TAB to view the options

To view the available configuration options in “snmp.set” command use below command:

snmp.set --help

EX: This is the snmp.limits + TAB command output to see the Configuration Options

That’s it for now with the SNMPv1 Configuration.

If you found this post as useful please rate the post and share it!

We have been informed by VMware with regard to this issue and this is going viral all over the world after the patch updates comes under KB4088875 and KB4088878. This is basically affecting to the Windows 2008 R2 and Windows 7 Virtual Machines with VMXNET3 adapters. VMware is currently investigating and working in their full scope to find a solution for this serious issue. If you are planning to update your Patches please make possible effort to avoid these patches from your List.

You might be getting similar errors if you have DHCP configured

As per the feedback we are getting from social media some had to upgrade the Virtual Hardware levels and some had to add the IP information again to those affected Virtual Machines. If you come up with anything different to these steps please feel free to comment on this so we can share it with others.

Another workaround to fix this issue was remove the NIC and re-adding to the Virtual Machine, also reinstalling the driver was one of the workarounds, but at this point there is no guarantee of any of these workarounds to fix your issue. Just try and see.

Please keep your eyes on below external links to get the latest updates, I will update the post as I’m getting the information:

Official Update From VMware (as of 15/03/2018):

If you found this post as useful please rate the post and share it!

We had this issue while we were upgrading the ESXi hosts to vSphere 6.0 from 5.5. vCenter server was successfully migrated with the VMware Update Manager and it’s database. We tried to upgrade the host after importing the base ESXi iso and crating the baselines from the beginning. I can confirm that we did not get any error while we are upgrading the VMware Upgrade Manager.

However, we had to follow the below steps and had to reinitialize the database to fix this issue. Make sure to take a backup of the Update Manager Database, your ISO images, offline patches and any host extensions before we start the process.

This was the appeared error while we were trying to upgrade the host.

Before we start the re-initialization of the database, stopped the Update Manager service, opened a command prompt as an administrator and changed the directory to C:\Program Files (x86)\VMware\Infrastructure\Update Manager\ by executing below command.

cd "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\"

Executed the below command to start the re-initialization process ( dot is required after -C and -L switches)

vciInstallUtils.exe -O dbcreate -C . -L .

See my screen capture for the command that I executed

It will run database queries and will take couple of time to complete the task similar output can be seen in the command prompt

After the completion of this command all the baselines, Images and extensions wiped out from the VUM database. Had to manually add them and created the necessary baselines. That’s the reason behind I asked you to save all your VUM data before start the process.

If you found this post as useful please rate the post and share it!

I thought it is great to write this article as this was quite useful tip for VMware admins to find out what are the default values of the vCenter server appliance such as SSO domain, DB user, DB DSN, domain DN and I wanted to know which account used to add the VCSA to my active directory domain while proceeding with the 6.5 upgrade.

So, I used below steps to find these details in my VCSA and I think this might be helpful for all my VMware colleagues.

Another practical situation was, I built couple of test vCenters with 6.5 and used a custom vSphere SSO domain, I couldn’t touch my vCenters for weeks and I started my test again. Bad Situation !!!, I completely forgot my custom test domain and I didn’t take a note of that, what should I do with this at this point?

Use of “Install Defaults Directory”

In vCenter Server Appliance, it has a directory called “Install Defaults” and it contains lots of files with these details and we can easily read those files and find out the required information.

Open a SSH session and login as root. Yes, you need to have the root credentials anyway to check this. Then go to Shell prompt

change directory to /etc/vmware/install-defaults, list the contents of the directory by executing “ls” command you can see a similar output like this and use below commands

cd /etc/vmware/install-defaults
ls

What is my SSO domain?

Read the content of the “vmdir.domain-name” to view the SSO domain using cat command and in the output you can see the SSO domain of your vCenter server

cat vmdir.domain-name

Which AD account used to connect to the Active Directory?

Read the content of the “ad.domain.username” file

cat ad.domain.username

Reading the content of these files provide more information about the vCenter appliance these are just few examples, these is just the output of reading the database related information

Here, I have created a short video to show “How to get the forgotten Single Sign-On domain from VCSA”

I hope you enjoyed this post!

If you found this post as useful please rate the post and share it!

We have been noticing some Virtual Machine unavailability alerts due to a running Virtual Machine level backups. Customers were complaining about the application unavailability and they were getting some alerts saying Virtual Machine is not available or down state at the same time of everyday. This issue appeared due to the FREEZE operation on the file system.

We have tried several troubleshooting steps to find a solution for this and had to spent considerable amount of time on the same issue. Finally, we were able to find a solution after performing the below steps separately for Windows and Linux Virtual Machines.

Note : VMware Tools must be installed and should be up to date.

This was the workaround for Linux Virtual Machines

Connected to the ESXi host which was running the affected Linux Virtual Machine and changed the directory to the Virtual Machine folder.

#cd /vmfs/volumes/<datastore_name>/<Virtual_Machine_folder>

Checked disk.EnableUUID setting is in the .vmx file

/vmfs/volumes/<datastore>/<virtual_machine_folder>/#cat virtualmachinename.vmx | grep disk.EnableUUID

In my case this parameter did not exist, so I had to edited the .vmx file using a text editor, added the disk.EnableUUID parameter and set it as “FALSE”, if the parameter is exist set the value to “FALSE”. (I used “vi” editor and “:wq” used to save the parameter).

Confirmed the parameter addition using the same command

After the completion of the parameter addition logged in to the Operating system of the Virtual Machine and changed the directory to “/etc/vmware-tools/” folder and check whether there is “tools.conf” file is there, if it id not there then create a file. In my case we had to create the file.

#cd /etc/vmware-tools/
#/etc/vmware-tools/ls

To create the file used vi editor

#/etc/vmware-tools/vi tools.conf

Added the below content to the file, save it and exit

[vmbackup]
enableSyncDriver = false
[logging]
vmtoolsd.level = debug
vmsvc.level = debug
vmusr.level = error
vmvss.level = debug
guestinfo.level = error
vix.level = error
vmsvc.handler=vmx
vmbackup.level=debug
vmbackup.handler=vmx

Workaround for the Windows Virtual Machines

If it is a Windows Virtual Machine we had to check the Windows Application logs to identify whether there are errors related to the VSS Writers. In some cases VSS Writers may having some issues and had to exclude the VSS Writers from the snapshot operation. A sample issue can be seen in this Microsoft article, you might be seen something similar to these symptoms.

To find the VSS Writers run a Command Prompt with the Administrative privileges and run the below command

C:\Windows\system32>vssadmin list writers

Read more about VSSAdmin Utility from Microsoft Documentation.

Then opened the “%ALLUSERSPROFILE%\Application Data\VMware\VMware Tools\” location in the explorer

Created a “vmbackup.conf” file, to do that opened a Notepad added the VSS Writer and saved it as a configuration file. See below screen captures for more information

Created the “tools.conf” file in the same location and added the below content

[logging]
log = true
vmtoolsd.level = debug
vmtoolsd.handler = file
vmtoolsd.data = C:\ProgramData\VMware\VMware Tools\vmtoolsd.log
vmsvc.level=debug
vmsvc.handler=vmx
vmbackup.level=debug
vmbackup.handler=vmx
vmusr.level = error
vmvss.level = error
guestinfo.level = error
vix.level = error

You should be able to see something similar to this after the creation of the file

Restarted the VMware Tools Service

After doing these configuration changes to the .vmx file and the .conf file of the affected Virtual Machines this issue was resolved and Virtual Machines are not freezing during the backup operation.

If you found this post as useful please rate the post and share it!

This is just a small tip for the VMware tools installation on a Linux Operating System. If you are installing the VMware tools on a Linux Machine you might be familiar with this error message when you perform the “./vmware-install.pl” command. So, I thought to write this as Tip! for the VMware administrators and I think this might be really helpful for them.

This is the error that can appear on your terminal and I just listed the files in the “vmware-tools-distrib” folder for your information.

For example this is the command for the CentOS:

[root@server vmware-tools-distrib]#yum -y install perl

For more information See the steps to install the tools on a linux system, I wrote this article using a CentOS Virtual Machine

How to Install VMware Tools on Linux Virtual Machine

Mount the VMware Tools installer to the Virtual Machine, Right-Click on the Virtual Machine -> Guest -> Install/Upgrade VMware Tools

Login to the Operating System and follow the below Steps in the terminal:

[root@server]#mkdir /mnt/cdrom
[root@server]#mount /dev/cdrom /mnt/cdrom
[root@server]#cp /mnt/cdrom/VMwareTools-version.tar.gz /tmp/
[root@server]#cd /tmp
[root@server tmp]#ls | grep VMwareTools
#Note the VMware Tools Version, ex: VMwareTools-5.0.0-12124.tar.gz
[root@server tmp]#tar -zxvf VMwareTools-version.tar.gz
[root@server tmp]#cd vmware-tools-distrib
[root@server vmware-tools-distrib]#./vmware-install.pl

Check the VMware Tools status in the Summary Tab

Once You confirmed the status unmount and remove the installation packages from the Operating System for the clean up

[root@server vmware-tools-distrib]#cd #
[root@server]#umount /mnt/cdrom
[root@server]#rm /tmp/VMwareTools-version.tar.gz
[root@server]#rm -rf /tmp/vmware-tools-distrib

If you found this post as useful please rate the post and share it!

I recently attended a webinar with Runecast and it was about how to Mitigate Spectre and Meltdown in VMware Environments.  In this post, you can see the steps to Mitigate Spectre and Meltdown in VMware Environment using Runecast Analyzer. Previously, I wrote an article about Spectre and Meltdown and the VMware recommendations to mitigate this Spectre and Meltdown vulnerabilities, you can read it here.

In a VMware Virtual environment protection should be start from all layers starting from Management to core components.

Mitigation should be staring from vCenter server to the ESXi hosts, Runecast analyzer will scan the environment against the VMware knowledge base and provide the recommendations.

Runecast Analyzer Appliance 1.7.3.0 – Update Appliance

If you are planning to mitigate Spectre and Meltdown vulnerabilities using Runecast Analyzer make sure to have the latest version of the Analyzer. Sometimes you may already seen the update available notification in the top panel.

To update the  Runecast Analyzer login to the appliance VAMI interface type “https://<IP_ADDRESS_OR_FQDN>:5480” on the browser and use the credentials provided in the Runecast Analyzer user guide.

Go to update Tab and check whether you can see the “Available Updates” notification and the appliance version, otherwise click on “Check Updates” button. To install the updates click on “Install Updates” button

It will prompt to install the updates click on OK to start the update.

Check the version again in the VAMI interface after the update

After login to the Runecast Analyzer you can see the updated information in the notification panel

Run an analysis against the VMware environment and you can see the Spectra and Meltdown issues under issue list

Perform the Analysis

Click on the issue it’ll provide the VMware response for the Spectra and Meltdown, KBs and detailed mitigation steps under details tab

Affected components can be seen in the “Findings” tab

One important thing is it will show the complete updates and communications regarding the Spectre and Meltdown updates. Follow the instructions and the updates in order to mitigate the vulnerabilities in the environment. Runecast will give you the best instructions and updates for your environment.

Use vCenter Web Client

Runecast Analyzer has a vCenter Web Client plugin and you can find my previous article about installing Web Client plugin to the vCenter can be found here.

Plugin can be used to scan the environment and issue can be seen in the “Details” tab

Under “Affected objects” displays the ESXi hosts subjected to the vulnerability

Change Runecast Analyzer Update Settings to Automatic Updates

To enable the automatic Updates, login to the VAMI interface. Go to “Update” tab and Select “Settings“. It has three main options for the updates. Select according to the requirement.

Runecast Analyzer updates may include KB/Best Practice/Security hardening updates, application component updates or appliance OS updates.

If Runecast Analyzer has the internet connectivity it can use the default repository for the updates. If there is a proxy requirement add the proxy configuration under network settings.

If you found this post as useful please rate the post and share it!

The post How to Mitigate Spectre and Meltdown in VMware Environment with Runecast Analyzer appeared first on TECHCRUMBLE.

This OVF Deployment Error occurred after the vCenter server 6.5 upgrade. It was not happening in the previous versions and I can confirm that we were able to deploy the same OVF template in the 6.0 and 5.5 vCenter servers. This OVF Deployment error was not a common error and I thought to post this in my blog and I hope this post will help others as well.

Deploying OVF template is just a common task as a VMware admin and I’m not going to show you the steps of deploying an OVF template. The error message was appearing as below in the OVF Deployment window.

How to fix this OVF Deployment Error ?

To fix this OVF Deployment error, open the OVF file using a text editor. I use Notepad ++ which is one of my favourite software. Right click on the OVF file and open with the text editor.

We were able to identify the below block of codes with the “IDE Controller”

We have changed the Controller type as “SCSI Controller” and saved the file

After saving the file, we restarted the OVF deployment and this time it was successful. It was a quick fix and hope this post will help others to resolve the similar OVF deployment errors.

If you found this post as useful please rate the post and share it!

The post vCenter Server 6.5 : OVF Deployment Error – :VALUE_ILLEGAL: Value ‘‘3’’ of Parent element does not refer to a ref of type DiskControllerReference. appeared first on TECHCRUMBLE.

In this article, let’s see how to enable ESXi Host Encryption in the vSphere 6.5 environment. Some Virtual Machine Encryption tasks enables ESXi Host Encryption automatically if the account has the relevant privileges. One important thing, Virtual Machine Encryption can be enabled only if you enable the ESXi Host level encryption. Encryption tasks are possibly only in environments that include vCenter Server.

After host encryption mode is enabled, all core dumps are encrypted to avoid the release of sensitive information from your secure Virtualized environment. If you no longer use the Virtual Machine encryption you can disable the Host Encryption. Disabling Host Encryption is not that easy and let’s see how we can work with these settings.

vCenter Cryptography Privileges and Roles

By default, vCenter Server administrator has all the relevant privileges. Also, there is a No cryptography administrator role which does not have the Cryptography Privileges.

Cryptographic Operations Privileges:

This is a sample output for the No Cryptography administrator Role privileges

How to Enable ESXi Host Encryption

To enable the Host encryption “Cryptographic operations.Register host” privilege is required.

Login to the vSphere client, click on Host->Configure->Security Profile. Click on “Edit” under “Host Encryption Mode”

Set the Encryption Mode to Enable and click “OK”

How to Disable the Host Encryption Mode

If you are no longer use the Virtual Machine Encryption you can disable the Host Encryption. Enabling the Host Encryption mode is just a easy task if you have the right privileges. But disabling Host Encryption is bit of a pain. Unfortunately, I don’t have the screencaptures to show you the steps and if I get a chance in the future I will update the post with the screen captures.

To disable the Host Encryption follow the below Steps:

If you found this post as useful please rate the post and share it!

The post vSphere 6.5 : Enable ESXi Host Encryption appeared first on TECHCRUMBLE.

VMware partnered with the pioneer of the Global Cloud platform AWS and released athe VMware cloud on AWS platform for its users and now you can experience the best of the both worlds in a single Cloud Service. Previously, I wrote an article about the Technical Overview of the VMware Cloud on AWS environment after announcing the initial release.

VMware Cloud on AWS hypervisors running on top of the i3 bare metal instances and integrated with the Amazon Cloud Formation to build the Cloud based Software Defined Datacenter. These bare metal i3 instances give you the direct access to it’s compute resources and have the below hardware specifications:

How VMware Cloud on AWS use the AWS Services for VMware SDDC ?

VMware Cloud on AWS leverage the AWS services such as Cloud Formation, IAM and Lambda to provide and deploy the i3 Bare metal instances and complete vSphere Software stack (vSphere, NSX and VSAN) in the cloud.

First of all customer will login to the VMware on AWS Console and start the SDDC deployment. It will generate the Cloud Formation Template which customer run in the AWS account, Cloud Formation will leverage the AWS Identity Access Management (IAM) policies to grant the cross account access. These cross account access describes the AWS VPC, VPC Subnets and Route tables to create and manage the ENIs (Elastic Network Interfaces). These policies allows the full visibility to the operations which are performing the VMware SDDC.

Once the access is granted, AWS Lambda communicates to the Cloud console to provision the VMware SDDC in the customer VMware account. It will deploy the hosts according to the number of hosts selected during the provisioning stage along with the VMware software stack.

Each ESXi host provide the connectivity to the AWS ENIs, one additional ENI connectivity in each host will use the Gateway to migrate across the hosts (additional ENI connectivity shows in the first EC2 instance only). NSX will create the logical networks and these ENI connectivity allows to utilize the AWS services such as S3, Redshift, Lambda, etc.

Try VMware Cloud on AWS on VMware Hands-on Lab

VMware Hands-on Lab allows you to try this cloud platform and it’s worth of spending some time and playing around with this. Use your myVMware account to access the Hands-on Labs.

Fill the required details and accept the End User License Agreement to start the free trail

Wait for sometime to create the Hands-on Lab

Once you get the Desktop access open Google Chrome

Click on Student Check-In, type the same myVMware account email and search, once it listed the email click on Click to Validate to validate the email address

Check your inbox and you will get the confirmation email and you will be able to access the VMware on AWS Console from your browser

Now you can follow the steps to configure the VMware Cloud on AWS platform in the instruction.

I will post another complete post with the configuration steps. I hope this post will give you the high level overview of the VMware Cloud on AWS and how it leverages the AWS services and vSphere software stack to provide the best cloud experience.

If you found this post as useful please rate the post and share it!

The post VMware Cloud on AWS : Experience the Best of Both Worlds! appeared first on TECHCRUMBLE.

It’s pleasure to write this article as there is a new release of VMware vSphere and vCenter Server 6.7. VMware vSphere 6.7 is officially released and it comes with lots of enhancements. In this post I’d like you to walk you through the vCenter 6.7 features, deployment and the enhanced options.

VMware has officially announced that vCenter Server appliance is the default deployment of the vCenter server and Windows based vCenter Server will be discontinued in the next numbered release. But still, You can download the Windows based vCenter Server in this 6.7 release.

From the appearance point of view it introduced with Clarity Theme and it supports the Embedded PSC with Enhanced Linked Mode.

Upgrades and Migrations to the vCenter Server 6.7

Only vCenter Server 6.0 and 6.5 versions can be upgraded directly to vCenter Server 6.7 version and vCenter Server 5.5 is not possible to upgrade directly. vCenter Server 5.5 should me upgraded to 6.0 or 6.5 before upgrade to vCenter Server 6.7.

vCenter Server (6.0 or 6.5) which are managing ESXi 5.5 hosts cannot be upgraded or migrated to vCenter Server 6.7 directly and should be upgraded these hosts to 6.0 first.

These are the benefits of this deployment model:

vCenter Server 6.7 Appliance Deployment

Let’s try to deploy the appliance and discuss the differences while we enjoying the deployment. Download the ISO and mount it to a one of the Server available to start the deployment. Actually it doesn’t have any differences with the vCenter Server 6.5. Go to “vcsa-ui-installer/win32” folder and run the installer

Similar to vCenter 6.5 this has two Stages to deployment and the Configuration of the vCenter Server

Stage 1 – vCenter Deployment

Installer GUI has completely changed and icons explain the procedures. To start the deployment click on the “Install” Option

Similar to 6.5 vCenter Server deployment it has two stages in the deployment wizard, This is the introduction screen anc Click on “Next” to proceed

Accept the End User License Agreement and Click “Next”

It has both the Options to install as an Embedded Deployment or Install with a Separate PSC, In this deployment I’m going to use the Embedded deployment.

Provide the target vCenter or the ESXi host, Port and the Root Credentials to the Appliance

Accept the Certificate Warning

Provide Virtual Machine name and the root Password

Select the Deployment and Storage Size

Provide the Network details

Review the Configuration and Click “FINISH” to start the Appliance Deployment

Wait until it Completes the appliance deployment

Once it completes the Stage 1 – Appliance deployment , Continue to the Stage 2 – vCenter Configuration

Stage 2 – vCenter Configuration

This is the Introduction of the Stage 2 , no configuration is there

NTP and SSH access configuration can be seen in this window

I got this Warning! , but I didn’t change the IP address of the vCenter Server, click OK as we do not change the IP address

Provide the Single Sign-On domain and administrator Credentials

If you wish to join the Customer Experience Improvement Program select the check box

Review the Configuration and hit “Finish” to start the vCenter Configuration

Wait for the completion of the Configuration

Once it completes the configuration click on the Getting Started page link to open the vCenter server on your browser (Google Chrome is recommended)

Getting Started page is different than the Previous version of the vCenter Servers and HTML5 option is highlighted

Allow the “vmware-cip-launcher.exe” and open the launcher

vCenter Server Management Interface Improvements

Appliance management can be accessed from the same url (https://<fqdn_of_vcenter>:5480), GUI is completely different

VAMI interface has given a completely different look, here the Summary tab of the appliance

Three Syslog Servers can be configured

File level Backups of the vCenter Server can be Configured and scheduled, this is one of the newly added great features. Previously, there was no scheduling option and we had to use typical cron jobs to automate the backing up the files of the vCenter Appliance. This is a great improvement

Scheduling option

vCenter Server Interface and Features

vCenter Server Interface has got some major improvements and it is much similar to a cloud console, complete summary can be seen in the vCenter Home. A great overview of the resource utilization

Platform Service Controller configuration can be seen under “Administration”

vRealize Operation integration also added to this and can be seen in the vCenter “Home“.

Update Manager has integrated to the vCenter similar to the vCenter server 6.5 and has a different appearance in this release and added with a different icon

There are couple of posts on the way and stay tuned!

If you found this post as useful please rate the post and share it!

The post Introducing vSphere 6.7 : What’s New in vCenter Server 6.7 appeared first on TECHCRUMBLE.

This issue has been noted while and I thought to share this in my post as this was a common situation in Windows 8 and Windows 2012 Server in all our vSphere 5.5 environments, these virtual machines hang at splash screen without booting up the Operating System.

VMware KB article (2092807) clearly mentioned that, this issue is causing with the virtual machines configured with the virtual hardware version 10 (vmx-10), but I have seen some virtual machines lower than virtual hardware version 10 have the same symptoms when booting up the OS. I’m not quite sure this issue is limited to the virtual hardware version 10 (vmx-10) or that’s something wrong with our virtual machines. But we were able to fix this after following the same workaround.

Virtual Machines Hang at Splash Screen

This is how the the console looks like when the Operating system Loading, it is sitting in the same splash screen until we perform a manual reset of the virtual machine.

The cause of this issue explained as below:

This issue is fixed in below VMware ESXi versions:

Virtual Machines Hang at Splash Screen workaround Fix!

You can follow the below workaround to fix this splash screen and we were able to fix the similar issues in all our Windows 8 and Windows 2012 servers.

Power down the virtual machine, right-click and select “Edit Settings…”

Go to “VM Options” tab, expand “Advanced” menu click on “Edit Configuration…”

Add the “monitor_control.enable_softResetClearTSC = TRUE” parameter to the virtual machine (parameter and value is in separate columns, follow the screencapture to the parameter update)

Save the configuration and Power On the virtual machine, Windows Operating system will load without an issue.

If you found this post as useful please rate the post and share it!

The post Windows 8 and Windows 2012 Server or later virtual machines hang at splash screen after reboot appeared first on TECHCRUMBLE.