disk space filling up on host where vmessentails console is running

[mikejdunphy]

Despite having a large NFS store to store images this directory on the host where the
vmessentials is filling up the local disk

/var/morpheus/kvm/images

I've lvextended to the max but probaby not enough, how do we divert this elsewhere ?
root@hp-vmessential1:/var/morpheus/kvm/images# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 152G 3.1M 152G 1% /run
efivarfs 496K 206K 286K 42% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv 216G 195G 12G 95% /
tmpfs 756G 45M 756G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda2 2.0G 195M 1.6G 11% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 756G 0 756G 0% /run/qemu
/dev/mapper/3624a9370678e444ef0404df200096272 10T 23G 10T 1% /mnt/97fd47ae-aa9a-4fd5-a16e-2af49d9e9fe7
tmpfs 152G 16K 152G 1% /run/user/0
root@hp-vmessential1:/var/morpheus/kvm/images# vgs
VG #PV #LV #SN Attr VSize VFree
ubuntu-vg 1 1 0 wz--n- <220.48g 488.00m
root@hp-vmessential1:/var/morpheus/kvm/images# ll -h
total 158G
drwxrwxr-x 2 morpheus-node root 4.0K Jul 10 16:24 ./
drwxrwxr-x 8 morpheus-node root 4.0K Jul 3 21:05 ../
-rw-r--r-- 1 root root 62G Jul 10 16:32 334b6f5b-0299-4e16-b611-4923a5b13eba
-rw-r--r-- 1 root root 29G Jul 10 16:23 334b6f5b-0299-4e16-b611-4923a5b13eba.vmdk
-rw-r--r-- 1 root root 14G Jul 10 14:20 33c0b4d3-e72b-4929-8810-6b5eff24bbe5
-rw-r--r-- 1 root root 14G Jul 10 13:09 40060f7f-dfad-45b0-9920-13c8f76c21df
-rw-r--r-- 1 root root 426K Jul 10 16:24 574e68b8-cca9-470b-9f7d-60fb7012c76b.vmdk
-rw-r--r-- 1 root root 5.3G Jul 10 16:24 6fc8667f-9cbf-4b4d-95d0-7aba9d49337a.vmdk
-rw-r--r-- 1 root root 22G Jul 9 16:28 9883fe6d-c7e0-46e3-90ad-2ce194c204b9
-rw-r--r-- 1 root root 14G Jul 10 13:19 b323deac-f9a0-45a3-bf5d-269f244278df

this on the guest frees some stuff 
but its not enough
root@hp-vmemanager:~# fstrim -av
/: 30.8 GiB (33070436352 bytes) trimmed on /dev/vda2

[mikejdunphy]

I think some user testing was using a wizard that I havent found yet that converts vmware ovf automatically
Anyways I found the 2 biggest newest files and nuked them and resumed the vm because
it paused when it 100% full
I am not sure if they have a option to select something else other then local
Apparently local means local on the host and not the vm-hpemanager guest where
the console is running on the host

[mikejdunphy]

another thing is I definitely was using the NFS datastore for this image work today
Why is it on the local disk and 3 copies of it 
root@hp-vmessential1:/var/morpheus/kvm/images# ll |grep 14431092736
-rw-r--r-- 1 root root 14431092736 Jul 10 14:20 33c0b4d3-e72b-4929-8810-6b5eff24bbe5
-rw-r--r-- 1 root root 14431092736 Jul 10 13:09 40060f7f-dfad-45b0-9920-13c8f76c21df
-rw-r--r-- 1 root root 14431092736 Jul 10 13:19 b323deac-f9a0-45a3-bf5d-269f244278df

I was deleting it in the gui 

I've nuked em all as well, hope it doesnt mess up whatever the console wants

This isnt good .. cant be having the node disk filling up all the guests pause

[Ranveer]

Hi,

Please make sure NFS data store is ticked for "IMAGE TARGET"

Infra >Clusters >Select Cluster name > Storage > Data Stores will be listed
Click on Pen icon to edit.
Tick "IMAGE TARGET" & Save Changes.

 Regards

Ranveer

[mikejdunphy]

@Ranveer,

it is indeed

[mikejdunphy]

looks like it happens when when it is provisioning the vm
I caught the nasty bug in the act
the first step Provision - Upload Image downloads the image
from the default nfs datastore and writes to /var/morpheus/kvm/images on whatever node
it is provisioning too
i.e. the one I just did you can see the 14GB one I am doing with the test
and a 3GB one from a while back.

root@hp-vmessential3:/var/morpheus/kvm/images# ll -h
total 17G
drwxrwxr-x 2 morpheus-node root 4.0K Jul 11 12:01 ./
drwxrwxr-x 8 morpheus-node root 4.0K Jul 3 21:05 ../
-rw-r--r-- 1 root root 14G Jul 11 12:04 388e6277-8025-4954-866c-ac722c03bd77
-rw-r--r-- 1 root root 3.0G Jul 4 00:20 adf2d315-7526-4674-b828-f2cdb58df556

after the provisioning is complete it just leaves the file there
and if I delete the vm and the instance and everything it is still there

this cannot possibly work, and needs to be fixed , no need to copy the
file down from the nfs and what if the image is more then the space
available on the host.

The image that killed my host was 34GB ovf vmware export and with
all the files from previous provisionings in there it filled up the
disk. I lvextended the filesystem to the max using all the free
extents in the vg but it still was not enough.

root@hp-vmessential1:/var/morpheus/kvm/images# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 216G 45G 162G 22% /

Am I missing something here ? Is there a setting I missed ? Will this
stuff get fixed ... this is a trial killer if I have ever seen one.

I should change the subject of this discussion, its not really disk space
where the console is, it can be any node, it just made it worse on node
1 because I have not migrated the console off the local space yet so
there is less available. Nobody has responded to my other thread yet.

[Arnout_Verbeken]

You need to understand how images work in VME.

There’s 2 layers of images

- The virtual images is like your “content library” (NFS share, CIFS share, or local storage in the VME Manager VM)

- Then there’s a datastore on the cluster for localized image caching

When you provision a virtual images, It is first copied to your image target datastore (/var/morpheus/kvm/images on the host where the VM lands)

Then cloned from there to the volume of the vm

Once it’s copied to image target it’s reused on subsequent provisions. 

This makes it superfast  to do multiple deployments of the same image since it does not need to be copied every time from the "content library".

You can select your "big datastores" as "Image Target" and it should not use the local image target datastore.
I will test later if that functionality is broken since apparently you have that mark already set.

If those "cache images" are eating up your local disk, you can delete them via "Cluster - Virtual Image tab". 

If the cache store (var/morpheus/kvm/images) is >80% full, VME will start auto-purging old images. (in your case, the latest big cache image immediately made your cache/local disk full even before VME had a chance to cleanup)

It is advised to use a real datastore as "Image Target" so you don't have to worry about local host disk space.

Hope this helps.

[mikejdunphy]

@Arnout_Verbeken,

so I have 2 large nfs shares one for images and one for backups
your saying that I need another datastore for just images ? Why cant it use the NFS ?

So how do you mark a datastore for image target  I cant find it. I went to datastores and clicked edit .. no options
When I go to cluster -> virtual image tab I am seeing the images that are on the nfs datastore, I dont
have a issue with that only local disk space. 

Here are my datastores, for this trial I would like everything to just go in the 10tb gfs2 and I guess
delete all the others so I dont have this problem.
I need to know how to mark that 10tb datastore as image target

[kurotan]

Re: Temporary file issues when migrating VMware VM... - Hewlett Packard Enterprise Community

Looks like the same issue as mine.
Are you migrating a VM?

My issue seems to be that temporary saving to local storage is gone in v8.0.6.
However, there is still the issue of pre-conversion files remaining instead of being deleted.

[Arnout_Verbeken · Accepted solution]

Hello!

I think it is important to first go over all the different types of storage possible wihtin VME so you understand what is used where, and what the exact flow of data is. I fear that you miss some design concepts of VME, which confuses you.

File shares (configured via Storage - File Shares - Add):

• This is used to configure the storage of Virtual Images (cfr Vmware Templates) and/or Backups
• Depending on the options you chose, the share you configure will be the default share for your Virtual Images and/or backups.
• You need to configure at least 1 share for Virtual Images.  If not configured, it default to /var/opt/morpheus/morpheus-ui/vms.  Be aware of disk space issues inside the Manager VM!
• You can chose between Azure, CIFS, NFS and local.
• Those file shares live INSIDE YOUR MANAGER VM.
• If you chose local, you will need to configure a directory inside the manager VM.  Note that all your virtual images will be stored here so you need to take care about free space for the manager to function.  It is recommended to attach a 2nd disk to the VM, partition, format and mount this disk, and use this mount for Virtual images.  You will keep your root filesystem free and all virtual images will land on the new disk/mount

In short, your virtual images and/or templates live on a simple share.  This share is probably not the fastest storage you have and can even be outside your DC (Azure).  You mark your default Virtual Image Store with the checkbox "Default Virtual Image Store".

VM Datastores (configured via Clusters - your_Cluster - Datastores - Add):

Here you will configure the storage where your VM will land.  NOTE THAT YOU DO NOT CONFIGURE THIS VIA STORAGE - DATASTORES.

Different types of storage are possible.  I will highlight the most common.

• Directory Pool: Local storage on your Hypervisors.  Enabled by default.  If you do not configure any additional Datastore, all VM's will land on this local storage.  Your VM's can't be migrated to another host and you risk to run out of space on your hypervisor.  It is advised to uncheck the "Active"  checkbox so it is excluded for new VM's.
• NFS Pool:  An NFS share/volume that is used as datastore.  Not to be confused with the NFS File Share above that is used for Virtual Images. (I think) it can be the same share as the Virtual Image share, but I would not recommend it.  A minimum of 2 hypervisor hosts are needed.  The NFS Pool needs to be accessible by all hosts in the cluster.  Once created, it is advised to Edit the pool and enable "Hearbeat Target".
• GFS2 Pool:  An iSCSI or FC volume that is used as datastore.  A minimum of 3 hypervisor hosts are needed.  The volume needs to be accessible by all hosts.  Once created, it is advised to Edit the pool and enable "Heartbeat Target".
• These datastores live INSIDE YOUR HYPERVISORS (not directly in the manager VM)

Image Target (concept that does not exist in Vmware):

• Not to be confused with "Virtual Image Store" above.
• VME has the concept of caching your Virtual Images so deployments of your VM's go fast.  The first time you deploy a VM from a Virtual Image, the Virtual Image is copied to a "cache location", and from there copied again to the final datastore.
• This is a concept from the full Morpheus stack, where you can manage different clouds, including remote clouds.  Imagine you have your Virtual Images on a local NFS share and want to deploy VM's to e.g. Amazon or Google Cloud.  Without this caching, your Virtual Image would need to transfer the wire with every new deployment.  Hence the reason we "cache" the image as close to the target as possible.
• While not fully applicable to VME (since everything is on-prem), it still has a lot of benefits if configured correctly.  Once the image is "cached", deployments of new VM's go very fast since the Virtual image does not need to be copied every time again from your "slower" Virtual Image Store.
• By default, a local folder on your hypervisor hosts is used for this caching.  The default location is /var/morpheus/kvm/images
• If you want to use this local folder as cache, I would recommend to mount /var/ to a separate disk on your host during initial installation of Ubuntu.
• When you use a Virtual Image to deploy a VM for the 1st time, the Virtual Image is cached to every host in the above directory location (except for hosts in maintenance)
• You can view the location where a certain image is cached via Virtual Images - your_image - Location.
• You can also delete your "cached images"
• VME will automatically purge images if the location is used for more than 80%
• It will require some maintenance and follow up to make sure your hypervisors do not run out of disk space.
• You can always delete those "cached images".  It is recommended to do that from within VME (VIrtual Images - your_image - Location - Delete)
• BUT!
  
  
• You can also use your other datastores (NFS Pool, GFS2) as Image Target.
• You enable this by Editing the Datastore (Clusters - your_cluster - storage - Edit Datastore via pencil) and selecting the "Image Target" checkbox.
• I recommend to disable "Image Target" on local "Directory Pool" datastores (if it woul be enabled).
• Using your "big and fast" datastores as Image Target, will ensure your hypervisor does not run out of diskspace and your deployments will lightning fast.

So when your storage (on the hypervisor or on the manager VM) is running out of space, you probably have not configured one or more of the following:

• If backup is enabled, no Backup File Share is created.  Backups land on the disk in the Manager VM. 
• If no specific "Image Target" is configured, images will be cached on the local storage of your hypervisor hosts. 
• If no specific datastore is configured, the local "Directory Pool" will be used (equivalent to Vmware local datastores) as final destination for your VMS.
  
  Please let us know if you have any more questions related to this topic.

[mikejdunphy]

thanks for all of that, all I needed was You enable this by Editing the Datastore (Clusters - your_cluster - storage - Edit Datastore via pencil) and selecting the "Image Target" checkbox.


I couldnt find it on my own in the GUI. I have checked that box and will proceed today to provision a large image.