Adsense Code

Thursday, 3 September 2015

Snapshot still being created on a VM for hours but never ever completes? - Here's what to do.

If you are creating a snapshot or your backup product has tried to create a snapshot and many hours later, it is STILL trying to create the snapshot then you don't have many options.

Once you have tried the usual safe options like cancel, stop, stopping the backup job etc etc - then you are left with the KILL option.  And like the word kill, it is kind of final.  You will have downtime.  Only as long as it is for the VM to power backup etc but we all know those workloads are variable these days - so the downtime thing is a pain.

You will need


  • Putty client
  • The name of the host that the VM is on
  • The display name of the VM
  • Downtime
  • Console open on the VM that you are going to KILL


Method

  • Go to the security profile on that host and enable SSH
  • Connect via SSH (putty) to that host.
  • Type in

    ps | grep vmx | grep <name of VM>
  • This will then display the VM in question like so

The first column shows the different processes that are tied to the parent process.  In this case, the parent process is 21592627 and it is that process that we need to kill.  Remember - this has downtime.

Just type in

kill <parent process number>


Now go to the console of the VM and you can monitor what is going on.  You should get an error message about the vmx file and the VM should restart.  You should then within a few seconds get the snapshot job cancel!  Yay!


Tuesday, 1 September 2015

vSphere Server appliance using up lots of disk space and how to tidy it up

I had a problem where the vCenter Server appliance that we use ended up with very full disks.  First of all, and as you may have gleamed from other previous blogs, I got around this by extending disks and migrating data around.

But this just keeps the wolf from the door!  I logged a call with VMWare and they were referred me to this article which is great in that it stops the issue from getting any worse but it doesn't tidy up the woe that has been left behind.

So what do you do?

Maybe do a clone of your vCenter appliance for an immediate backup.

Find out which host is running your vCenter appliance and then connect directly to the host with a vSphere client as you are about to bring your vcenter environment down.

Login to the console your direct vcenter appliance now.

Type in

service vmware-vpxd stop

Then  

service vmware-vpostgres stop


Type the following command in make a copy of the original postgresql.conf file (all one entry)

cp /storage/db/vpostgres/postgresql.conf /storage/db/vpostgres/postgresql.conf.orig

So now we need to edit postgresql.conf - the VMware article says with a text editor, I used vi which is on the appliance but I have also included instructions for vi as I come a from a Windows background and whilst I have come to enjoy vi's quirks - when there are big issues going on, you don't want to faff about!

vi /storage/db/vpostgres/postgresql.conf

Page down and using the cursor keys scroll down to line 312 where you want to edit the log_min_messages = error line

Press the insert key and now you are editing the config file!

Delete the first hash/number so that the config file will be read and then delete the word warning and replace with the word error.



Now press the escape key and now you are not editing the file.

Press colon and then w and then press enter - you'll see in the bottom left of the screen a

:w

Pressing enter saves the file because you have just written the file (hence the w) 

Press colon again and then q and then press enter

:q

This will allow you to quit (hence the q....)

But now we need to delete the old excess data - change to the directory where all the excess data

cd /storage/db/vpostgres/pg_log

Then this gem of a command that I found which deletes EVERYTHING in that directory except for the 5 most recent files.

rm `ls -t | awk 'NR>5'`

On my British keyboard within the VM console, to get a pipe, | , I need to press Shift and ~ over by the main enter key.

If you have run out of space drastically, you will then have a lot of data to delete.  So this command can actually take a while to run.  Monitor it on your graphs as you do have a direct connection to the host that is running your vCenter appliance after all.

Now you just need to get vCenter back up and running.

service vmware-vpostgres start

and then

service vmware-vpxd start 



Monday, 13 April 2015

.Net2.0 framework - how do you install it on Windows 2012R2?

Are you having a problem just getting .Net2 framework to install properly on your 2012R2 server?


This was a niggley problem for us where .Net2 framework Handler Mappings would not appear on 2012R2.  When you go to the roles and features, .Net3 was installed (required to make .Net2 work on 2012)  We could even go into the registry as see on


HKLM\Software\Microsoft\NET Framework Setup\NDP\v2.0.50727


and install had a number 1 against it (installed, 0 would be not installed)




In the end, typing in this command an admin elevated command prompt got things working.




dism /online /enable-feature /featurename:NetFx3 /All /Source:<path to your SXS source> /LimitAccess


Hopefully this will get you up and running sooner!

Wednesday, 18 March 2015

Upgrade VDP 5.8 to 6.0 - and do you actually need to upgrade from vSphere 5.5 to 6.0 to take advantage?

I did just follow the documentation on the Vmware website here
 http://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vmware-data-protection-administration-guide-60.pdf

Importantly, I upgraded a VDP on 5.8 upto 6.0 still within a vSphere 5.5 environment as I wanted to take advantage of a VDP in 6.0 (free - or included with your licensing) having the larger capacity compared to the VDP in 5.8.  The capacity that was previously only on Advanced is now standard in 6.0 - 8TB of deduping storage for backups!

Do this when you are not going to run backups and you have plenty of time before backups start running.

The summary of the upgrade was as follows:-


  1. Download ISO.
  2. Shutdown the VDP appliance (nicely - use vmware tools, guest os shutdown)
  3. Edit settings - got to each virtual disk except for disk 1, and make them all dependent disks and snapshottable.
  4. Snapshot the VDP appliance.
  5. Power up the VM.
  6. Wait for the blue instruction screen (web management advice)
  7. Attach the ISO
  8. Wait (apparently according to the instructions the appliance can take a while - I happened to received a phone call at this point so I don't know)
  9. Go to https://<IP ADDRESS OF VDP APPLIANCE:8543/vdp-configure/
  10. On the config tab, make sure all services are running
  11. Go to the upgrade tab
  12. Upgrade - this can take a while - I mean, an hour or so.
  13. After it finishes, it will power down the VDP appliance.
  14. Edit settings, make all the disks independent (except disk 1) and not susceptible to snapshots
  15. Power on.
  16. This bit can take sometime too - you have just performed a large upgrade you know!


That's it!

The backups continued working and we have been able to restore data.

Java - Tomcat SSL certificates and Vmware View 5

This is to go through the process of making a website SSL compliant on Tomcat.  Which is just very VERY particular.  On the server that needs the certificate, I suggest creating a directory on the root of the C drive as a temporary working area.  Something like c:\newkeys  In this example, I will be using the server, cm-testserver
You need to add a java tool to your environment path first, go to Properties of My Computer, Click the Advanced Tab, Click Environment Variables.
In the System Variables group, select path and click Edit.
Type the path the JRE directory in the Variable Value Text box, on cm-testserver this is
D:\Program Files\VMware\VMware View\Server\jre\bin
Note: use a semi colon to separate the path variables.
On the server in question (cm-testserver), open a cmd prompt and navigate to c:\newkeys.  Type in
keytool -genkey -keyalg "RSA" -keystore keys.p12 -storetype pkcs12 -validity 360 -keysize 2048
This is telling the server to create a keystore (genkey).  A keystore is analogous to a safe.  This safe is used to hold your certificates and use the RSA algorithm (RSA).  The safe is to be called keys.p12  The keystore will be created in the pkscs12 format, it's valid for 360 days and the safe will be generated with 2048-bit encryption.
When keytool prompts you for the first and last name, type the FQDN (Fully Qualified Domain Name) of the server.  So this example would be cm-testserver.<yourdomain>.co.uk
Enter all the rest of the information, remembering that the country code is GB for ISO compliance.
Now you need to type in the following to generate a certificate request.  You will then send out the request, they send back a certificate and you store it inside the keystore.
keytool -certreq -keyalg "RSA" -file certificate.csr -keystore keys.p12 -storetype pkcs12 -storepass ATopSecretPassword
At the time of writing, we are using Janet, which in turn uses Comodo to authorise certificates.  Janet / Comodo only send back *.cer certificates, and not PKCS7 standard certificates.  The ZIP file that contains all of the certificates needs to be extracted to a new empty temporary directory.  Highlight one of the extracted certificates, right mouse click, and click select Open on the menu.
A new window pops open.  Click the details tab at the top, and then click the Copy To File button on the lower right.  Click Next on the wizard.  We are now exporting your new certificate out into PKCS#7 format.  Select the third option down "Cryptographic Message Syntax Standard - PKCS #7 Certificates (.P7B) and ENSURE that the tick box "Include all certificates in the certification path if possible" IS ticked.  Unlike in other configurations where you might install the root certificates one by one, here you bundle them all together.
Export it out to a file name, your choice, but I called mine testserver.p7b, and remember where you save it.
Now go back to the server where you were creating the keystores and copy the file that you've created with the merged certificates into your c:\newkeys
Go back to your DOS prompt, making sure that you are at the path c:\newkeys type in the following command
keytool -import -keystore keys.p12 -storetype pkcs12 -storepass ATopSecretPassword -keyalg "RSA" -trustcacerts -file testserver.p7b
With the above command, you are telling the keytool to import into your keys.p12 file (the keystore) the merged certificates that are held in your testserver.p7b file.
Yay!  You now have a certificated keystore file!  Now we have to make your Tomcat environment work with it
You need to copy that keys.p12 file to d:\program files\vmware\vmware view\server\sslgateway\conf\
Within that same directory, you need to edit a file called locked.properties
If the file does not exist, you need to create it.
In that file, you need two lines.
keyfile=keys.p12
keypass=ATopSecretPassword
The password is the same one that you typed in earlier (remember?!!)

Then restart the View Connection Server service.  Job done!

Monday, 9 March 2015

Upgrade your VCP to 6 with 65% off!

http://blogs.vmware.com/education/2015/03/vcp-recertification-deadline-extended.html

The web link above has the info that if you upgraded your VCP to 5.5 by the March 9yh deadline, then as a reward, you will be able to upgrade to VCP6 with 65% off your exam fees!

Worth a look when they come out!

Tuesday, 3 March 2015

Convert your vCenter environment to the standalone VCSA appliance

A great bit of info from www.virtually ghetto.com about a more straightforward way to convert your current setup to the standalone appliance.

Take a look!

http://www.virtuallyghetto.com/2015/03/long-awaited-fling-windows-vcenter-server-to-vcsa-converter-appliance-is-finally-here.html

Monday, 2 March 2015

vCenter Appliance - you really have run out of disk space - what can you do?



Okay, so we have actually run out of space.  And to recoup space, we need to add space so that the processes can run through the database and delete old data.

I could just reset the database but we've got SRM and backup products hooked into this so I can't just rip out all those settings.  It would be so much better if we could do this just like we often do on the Windows servers and just extend the disk.  So I did the following.

As vCenter was off-line, I went to host where the appliance would be.  We use DRS rules to ensure it will only ever be on one of two hosts - which makes it easier to find in this type of circumstance.

Use a direct console window on the vCenter appliance and login.  Presuming you have not done any tinkering around with your vCenter disk configuration, your database data will be held under 

/storage/db 

and you will have a disk mounted under /storage/db

To check, type in

du -h /storage/db

You should see a result of a good few amount of gigabytes of under /storage/db/vpostgres

If so, type 

mount

This will then tell you which device /dev/<something> on /storage/db  As I have already altered my configuration in the past, mine was /dev/sdc1

So now that we know the device, type in 

cfdisk /dev/sdc1

This brings up a little utility called cfdisk and it will be telling you the size in MB allocated to the disk that you want to extend.


So on the screen grab above you can see that cfdisk utility is showing us that /dev/sdc1 is over 300MB in size (this screen grab is post disk extension sorry!).  I do all this disk checking because I am a Windows person and not a Unix person and I need to double check!

Select Quit and then shutdown the appliance (web browser on port 5480 to your vcenter box and login and select shutdown).

Use the downloaded qparted ISO utility, present this as a CD to your VM.  And whilst you are there extend the disk you want to extend (you identified it by the size before remember...)   And now snapshot it.  You need to do it that way as you cannot extend a disk that has been snapshotted.

Then start the VM ensuring that you boot off the CD.  Boot it up, select your country, tell you want it to boot, use StartX and eventually you get a GUI.  On the top right of the window, you choose your device you want to extend, so I select sdc1, and it kind of looks like a Windows Disk Manager.  It tells me that the disk is that size and I can extend it be this amount of size.  So I take it to the max - I don't want to be here again any time soon!

Then go!



Give it a bit of time - it took a bit longer than I expected (again, my own experiences are normally Windows) but it got there in the end.  You can see the image above where it was processing it.  You can click details to have a look.


And some more details after it finishes.


I just clicked close, exited, and then shutdown the box.  Ensuring that I removed the tick so that the CD doesn't boot!  :-)

I then just started the Server up and ta-da!  Logging in on port 5480, the extra space was there.  Backup the now working server and then delete the snapshot.

Job done!





vCenter service has gone down - vCenter not accessible on the appliance - What could have gone wrong?


We had a problem where the vCenter environment had gone down.  All the hosts were okay and we weren't getting any scary reports about VMs going down left right and centre so it sounded like just vCenter had gone awry.

Our vCenter is running on the appliance so we connected via a web client on port 5480.  And as you can see, on the screen grab below, the database was running at 100%.  Now, we have about 220 VMs and 7 hosts but we had allocated 100GB to the database volume on the appliance which according to VMware sizing was more than enough.


So what is causing the problem?  Well - I don't know how the setting crept in as the default is to keep tasks and event information for 180 days but the task cleanup and event cleanup boxes were unticked.  This suggests that the tasks and events would build indefinitely!

So on the web client, properties of your vcenter and you can find the tick boxes below.



And like wise, if still using the Windows client, just go to the vCenter properties and alter the Database Retention Policy on there



Friday, 27 February 2015

How convert Windows 2012R2 standard to Datacenter

There are few requirements for Datacenter as Standard now supports a lot of RAM out of the box.  The main thing that Datacenter does offer is HotPlug of RAM and CPU - useful for database servers or web servers where it is difficult to negotiate to downtime to bring down a system.
 
A system can be upgraded to Datacenter without losing the software configuration.  It takes about 10 - 15 minutes and it requires two reboots.
 
Logon to the Windows server you want to do the work on and start a command prompt with Admin privileges.
 
Type in:
 
dism /online /Get-TargetEditions
 
This will show you the OS you can upgrade to - it is useful just as a little check that the OS you are doing the work on will do what you are expecting it to do.
 
If all good, and you want to go to Datacenter then type in
 
dism /online /Set-Edition:ServerDatacenter /ProductKey:W3GGN-FT8W3-Y4M27-J84CP-Q3VJ9 /AcceptEula
 
The product key is just the KMS key that tells the server to look for a KMS server to authenticate it.  If you are working on a server in the DMZ, then you will need to use a MAK key and then use the telephone authentication system.
 
You can then add the extra RAM and CPU as required.  If you are doing this work on a live SQL server you will need to type in commands for the SQL process to 'see' the extra hardware resources - which will be another blog!

How to configure a DMZ server to use your corporate licensing server

Servers in the AD find the KMS server for operating system and Microsoft Office licensing because of DNS.  However, servers in the DMZ do not have access to that, so some manual configuration is required.

You will need to ensure that port 1688 TCP traffic is allowed to pass through from your DMZ environment into your normal production environment so that the licensing can actually take place.
 
On the DMZ server in question, edit the local hosts file and add the two following lines - you will need the IP address of your KMS licensing server and know the host name.
 
<your IP address>    <The fully qualified host name>
<your IP address>   <host name>
 
Save and exit.
 
Then from a command prompt with admin privileges, type in
 
slmgr.vbs /skms <The fully qualified host name>:1688
 
 
Check that the license has activated
 
slmgr.vbs /dli
 
 
If not, type in
 
 

slmgr.vbs /ato    then a   slmgr.vbs /dli  to confirm

How to monitor VMware SSD disk performance

At the time of writing, February 2014, there are no graphs to monitor to look at the cache ratio performance.
 
You need to use the esxcli command prompt against the host that the VM is currently situated on.

Enable SSH on the host and connect with your SSH client - probably Putty! :)
 
Log in and type
 
esxcli storage vflash cache list
 
 
That will provide you with a list of VMDKs that have got SSD caching enabled against them.  They are usually  vfc-<unique number references>-<servername><VMDK number>
 
So now type in
 

esxcli storage vflash cache stats get -c <That string of stuff>

Thursday, 19 February 2015

Virtual Flash Read Cache - Fantastic - but something odd today... massive disk latency

We have been adding Flash Read Cache against selected VMDKs gently and we have been getting excellent results.  In the main, we have allocated against some of our virtual Citrix servers and we have noticed a decrease in the latency reported on the Citrix VMs and a reduction in the CPU utilisation on our SAN.

Our Netbackup Master server is also a virtual server and we thought with all those data reads, adding vFRC against the VMDKs on the 20% ratios that VMware recommend would only improve things.

Not so!

Over the morning, more and more VMs over different datastores, over different IO architectures, reported disk latencies.  We were, on a couple of VMs, getting disk latencies of over 22500 milliseconds (yes, 22 and a half seconds!).

It took a bit of digging and thinking of the only major disk IO change that we had done.  We removed the FRC against the master netbackup server and over about 10 minutes, everything started behaving properly again!

So - we have solved the problem - but puzzling on why that would be the case.  If anyone has any thoughts - do let me know!

How to remove deduplication on a Windows 2012R2 server - hold onto your hats!

We ended up having to remove deduplication off our Windows 2012R2 server on a few volumes as Netbackup could not 'in a few circumstances' be able to restore data.  This was despite us checking the compatibility and, of course, completing some tests.

Fortunately, we had replicated the server over to our DR site using VMware's SRM product, so we were able to test this next step before we did it, gulp, on the live environment.


First things first - DO NOT UNINSTALL THE DEDUPLICATION SERVICE ON THE WINDOWS SERVER and then read through all of this before you do the work!

Then start a powershell and type in

start-dedupjob -volume VolumeLetter -Type Unoptimization



Note the spelling of unoptimisation - I want to type it with an S - I'm never sure which are the correct way round in this trans-Atlantic world that we live in today.

The unoptimization will start its work.  It will take a long time.  It will increase disk IO.  And despite you checking what your data size is and your unoptimised size - you will need to increase the disk size allocated to your Windows server.  The unoptimisation creates a large data footprint on the same volume but does tidy up after itself.  You can shrink the volumes afterwards.  We didn't change the disk size on our practice server and it just worked over many hours.  However, with the live environment, which we monitored during the working day, we could the remaining disk space just being eaten up and used up.

You can check how it is all going by typing in

get-dedupjob

It will stay at 0% for ages - hours even.  Then it will just go 50%, 60% and the remaining percentages over an hour or so.

This does work, we haven't lost data and we only did it because we couldn't guarantee the data protection for the backups - but that made it scary to do on the live environment (despite checking it in test which was a duplication of live) as if it all went wrong, we didn't have backup to go back to.

But it worked!

Hope it helps others.