You might have faced this issue on HP G8 servers that running ESXi on it when you are connecting with vsphere client nor web client.
This issue due to excessive logging create by /var/log/hpHelper.log file. Also it can grow very larg and fill the ESXi ramdisk. This logs are generated by HP helper management agent (HP-ams).
Unable to Connect to MKS: Malformed response from server.
While you have to sure about port 902 open, You can resolve this issue by 2 way.
1 ) Workaround:
Simply stop the HP Helper Management agent service on ESXi host to temporarily fix it. Connect to your esxi with SSH and run this command :
Download the HP-ams offline bundle for your hardware model & ESXi version and install the updated hp-ams vib on your esxi host using the below command:
esxcli software vib install -d /datastore/directory/hp-ams-esxi5.5-bundle-9.5.0-15.zip
Reboot is required for the changes to take effect. This updated hp-ams VIB stops the excessive logging to the hpHelper.log file. Once it done, you will be able to access your virtual machines console without any issues.
Unable to connect to the MKS: Internal Error
Also you might see this error when you want use esxi console :
To resolve it do below steps:
connect to your ESXi Host via SSH and start the DCUI
press “F2” to log-in
select “Troubleshooting Options”
select “Restart Management Agents”
If your VMs showed as "Orphaned" in your inventory, Do not panic because it will take some minutes till they are connected back as normal machines.
Better you know that restarting the Management Agent, didn't impact to your machines and they will continue to run.